Why Grading Community Notes Like a Fact-Checking Wire Misses the Point

Why Grading Community Notes Like a Fact-Checking Wire Misses the Point

A headline claiming a moderation tool failed to flag 93% of false claims sounds damning. It reads like a smoking gun. It feels like the platform is broken and the critics are vindicated.

Except it tests the wrong machine.

Every election cycle produces a familiar ritual. Watchdog groups tally false claims on X, run them against Community Notes, divide the noted posts by the total sample, and publish a percentage that looks like a failing grade. When the number comes back single-digit—93% unflagged—the lazy consensus forms in seconds. Crowdsourced moderation is a sham. The system cannot scale. X is letting misinformation run wild.

I have spent years watching trust-and-safety teams, policy analysts, and growth engineers pour millions of dollars into content moderation systems that look great in slide decks and fail under real traffic. I have seen compliance departments buy expensive automated classifiers that choke on sarcasm, context, and breaking news. And I have seen critics treat every moderation mechanism as if it were a wire service editor with an infinite newsroom budget and a mandate to slap a warning label on every bad take on the internet.

That is not what Community Notes is. Judging it by total coverage of every false claim on a platform is like grading a fire sprinkler system by counting how many candles it blows out across a city.

What Community Notes Actually Is

To understand why coverage ratios distort reality, you have to start with the mechanics. Community Notes is not a centralized fact-checking desk. It is a contributor-driven annotation system governed by a consensus algorithm designed to bridge ideological divides.

For a note to go public, contributors from different sides of a political or social divide—defined historically by how they rate past notes—must agree that the note is helpful. That bridging requirement is the whole point of the architecture. A note written by partisans and upvoted only by like-minded users gets suppressed. The goal is to surface context that persuades skeptics, not to win an argument with a badge.

Precision matters more than volume in a system built on cross-partisan agreement. A single bad note that reads like partisan spin destroys trust in the entire program. If a user sees a note dunking on a politician they support, they dismiss the label and sometimes the platform. By requiring broad consensus across divergent raters, the system trades speed and total coverage for lower perceived bias.

Criticizing that trade-off is fair game. Pretending the trade-off does not exist is lazy analysis.

The Denominator Trap in Election Coverage

Look closely at how election studies construct their percentages. Researchers typically gather a curated set of misleading or false posts—often pulled from partisan feeds, viral chains, or pre-identified narratives—and check whether each one received a published note.

When the denominator is a curated bucket of false claims, the resulting failure rate tells you something real about coverage on that specific set. It does not automatically tell you how often users encountered unflagged falsehoods in their actual feeds, nor does it account for how the system handles velocity, ambiguity, or low-reach posts.

Consider a thought experiment. Imagine a scenario where 10,000 false claims are posted during a tense week of midterm voting. Two thousand are posted by accounts with tiny audiences that never leave their echo chambers. Four thousand are phrased as leading questions, memes, or opinions that sit in a gray zone where raters struggle to reach consensus. Another three thousand are rapid-fire variations of the same core rumor. Only one thousand are clean, verifiable factual claims with enough visibility and contributor attention to generate a multi-party draft and clear the bridging threshold.

If an observer looks at all 10,000 claims and counts 700 noted posts, they report a 93% miss rate. That number is arithmetically accurate for their sample. Yet it lumps together posts that barely had eyeballs, claims too ambiguous for cross-partisan agreement, and high-reach rumors that actually moved through public timelines.

A raw miss rate flattens all of those differences into one dramatic number.

Why "People Also Ask" Gets the Question Wrong

When a story like this trends, search behavior follows a predictable pattern. People type variations of the same questions into search bars, looking for a simple verdict.

Does Community Notes work during elections?

It works within narrow boundaries, and that is why simple "yes" or "no" answers mislead people. If your definition of "working" is catching most false claims made during a midterm cycle, the answer is no. Contributors move at human speed. Writing sources, debating wording, and clearing consensus thresholds takes hours or days, while viral posts peak in minutes.

If your definition of "working" is attaching credible, high-trust context to high-visibility claims without triggering immediate partisan backlash, it works better than traditional top-down labels in specific cases—especially when claims rest on clear public records, voting dates, or straightforward data.

Asking whether it "works" without specifying the job you hired it to do guarantees a bad answer.

Why does Community Notes miss so many false claims?

Four bottlenecks explain most of the gap:

  1. Contributor attention is finite. Volunteers do not monitor every corner of the platform 24/7. Niche topics and regional races get far less attention than national flashpoints.
  2. Consensus is a high bar. Bridging partisan divides takes longer and fails more often than getting a quick majority vote. If users with opposing rating histories cannot agree a note is helpful, the note stays hidden.
  3. Ambiguity slows everything down. Election falsehoods rarely arrive as neat, textbook errors. They arrive wrapped in half-truths, selective framing, out-of-context clips, and loaded questions. Fact-checking nuance takes longer than inventing a distortion.
  4. Distribution outpaces verification. Algorithmic amplification and reposting move faster than human writing and peer rating. By the time a note clears review, a post may have already done most of its damage.

None of these bottlenecks are secrets. They are built into the incentives and rules of a crowd system.

Should platforms rely on crowdsourced moderation instead of professional fact-checkers?

Relying on either one alone is a mistake.

Professional fact-checkers bring rigor, editorial standards, and speed on complex claims. They also face structural limits: slow turnaround on fast-moving news, accusations of ideological bias from skeptical audiences, and an inability to keep pace with millions of daily posts across every topic.

Crowdsourced systems scale participation and can earn credibility across divides when they work. They struggle with edge cases, harassment dynamics, coordination, and low-attention corners of the platform.

Treating crowdsourced moderation and professional fact-checking as mortal enemies is a false choice. A serious trust-and-safety strategy treats them as imperfect tools with different failure modes.

What Research on Credibility Actually Shows

Researchers who study misinformation and platform interventions have spent years testing what happens when users see labels, corrections, or crowdsourced notes. Thinkers like Gordon Pennycook and David Rand have examined how ordinary users judge truth and accuracy online, showing that accuracy prompts and source cues can shift how people evaluate headlines, even when partisan motives pull in the opposite direction. Other researchers studying social media corrections—such as Brendan Nyhan and Jason Reifler—have long documented how difficult it is to dislodge entrenched beliefs once people tie them to political identity.

Across that body of work, one practical lesson stands out: a label is not a magic eraser. Even when a note appears, some readers ignore it, some argue with it, and some already made up their minds before reading the post.

That makes the obsession with hitting a near-100% label rate even stranger. Slapping a disclaimer onto every questionable post does not guarantee minds will change. In fact, overusing labels can breed label fatigue, where users tune out warnings the way drivers tune out constant chimes in a dashboard.

The Real Flaws of My Contrarian Take

Taking a contrarian stance on a headline like "93% unflagged" carries its own risks, and pretending otherwise would be dishonest.

First, defending the design logic of a system can easily slide into excusing poor execution. Even if Community Notes was never meant to catch 93% of false claims, a 7% hit rate on a curated set of election falsehoods still leaves a massive amount of unchallenged misinformation circulating during high-stakes moments like midterms. A system can have a smart theory of change and still underperform when put under real pressure.

Second, nuance does not stop real-world harm. If a voter sees a false claim about polling hours, mail-in rules, or a candidate's record—and there is no note on it—the distinction between "algorithmic bottleneck" and "system failure" feels academic. Voters make decisions in real time, not inside a methodology footnote.

Third, contrarianism can become a crutch if it dismisses legitimate accountability. Watchdog reports play a real role in pressing platforms to improve coverage, speed, and transparency. Ignoring a troubling metric just because the headline is punchy is just as lazy as accepting the headline without reading the methodology.

How to Actually Judge Moderation Systems

If you run a platform, advise policy makers, or simply want to read election coverage without getting played by cherry-picked percentages, stop asking, "What percentage of bad posts got caught?" Start asking sharper questions.

1. Measure reach, not just raw post counts

A false claim seen by 500 people is a different problem than a false claim seen by 5 million people. Track where notes land relative to impressions, not just relative to a static list of posts.

2. Separate unworkable claims from missed claims

Audit unflagged posts by type. How many were genuine factual errors that had enough visibility to warrant a note and still went untouched? How many were opinions, memes, or obscure posts? Grouping them together inflates outrage and hides where the real gaps sit.

3. Time the gap between spread and intervention

Speed is half the battle in election integrity. A note that appears four days after a rumor peaks solves a different problem than one that appears while the claim is actively spreading. Measure time-to-publication on high-velocity narratives.

4. Audit for consensus blind spots

Check whether cross-partisan agreement fails systematically on certain topics, regions, or languages. A system designed around bridging divides can still leave blind spots where contributor pools are thin or polarized.

5. Pair crowd context with clear enforcement lines

Do not ask a crowd-consensus system to do the work of clear rules against severe harms like explicit voter suppression, coordinated abuse, or illegal acts. Draw crisp lines between policy enforcement and crowd context.

Headline writers love a clean villain and a single shock number. Real systems are messier. Community Notes is neither the savior of online truth nor a total fraud—it is a slow, consensus-bound tool being asked to solve a fast, high-volume problem, and pretending a 93% miss rate tells the whole story only keeps everyone confused.

JH

James Henderson

James Henderson combines academic expertise with journalistic flair, crafting stories that resonate with both experts and general readers alike.