About this project
The Propaganda Detector reads a news article and highlights passages that use propaganda techniques, such as loaded language, name calling, appeals to fear and slogans, then labels which technique each one is.
It's built on SemEval-2020 Task 11, a shared research task where teams from around the world built systems for exactly this problem. The task defines 14 techniques, and this project follows that list and that way of measuring success, so that results can be compared with published work.
Highlights are suggestions, not verdicts. Even trained annotators often disagree about what counts as propaganda, and the best competition systems found only about half of the passages humans marked. Treat a highlight as a prompt to read that sentence more carefully, not as a judgement on the article or its author.
How it works
The detector works in two stages, the same way the task itself is split up.
- Finding passages. A DeBERTa-v3 model reads the article and marks each word as the start of, the inside of, or outside a propagandistic passage. A CRF layer on top keeps those marks consistent, so that each passage comes out as one clean, unbroken span.
- Naming the technique. Each passage goes to a second DeBERTa-v3 model, with marker tokens placed around it so the model sees both the passage and the text around it. It picks one of the 14 techniques, or "none" when the first stage raised a false alarm.
Every highlight carries two confidence scores: first, how sure the first stage is that the passage is propagandistic, and second, how sure the second stage is about the technique. The --min-confidencesetting above filters on the two multiplied together.
Results
Three numbers are measured, as each represents something different:
- Finding passages (span F1): 48.4 ± 1.2. For comparison, ApplicaAI, one of the top teams in the competition, reported 47.4 on the competition's development set using a single RoBERTa-large model with a CRF layer.
- Naming techniques, given the correct passages (micro-F1): 66.6 ± 0.6. ApplicaAI's single models reached about 72 on the development set.
- Both together, end to end: 25.6 to 27.9 across two seeds, with a seed-averaged figure still to come.
Scores are the mean and spread over 6 training runs with different random seeds, measured on 30 held-out development articles that the models never train on. The competition was ranked on a separate test set whose labels aren't public, so other teams' development scores are the closest comparison. Theirs cover the full 75-article development set rather than these 30, so treat the comparison as approximate. End to end is much lower than either stage alone because mistakes compound, and a passage the first stage misses can never be labelled correctly.
Full results, including scores for each technique, are inResults/121026.md(12 October 2026).
Limitations
- English only, and trained on news from 2017 to 2019. Newer topics and writing styles may trip it up.
- Propaganda is partly subjective, as the corpus's annotators agree only moderately with each other, which puts a ceiling on how well any model can score against them.
- The training articles come from a limited set of outlets, so the model may partly learn each outlet's house style rather than the techniques themselves. I've started testing this by matching development articles to their publishers using the proppy 1.0 corpus, which identified the outlet for half of them. Only two outlets had three or more articles, though, far too few to draw any conclusion. A broader comparison between propaganda-rated and mainstream outlets is still to come.
- Pasting a link is experimental. Many news sites block automated requests, so pasting the article text is more reliable.