WikiPatrol
is an expensive AI worth paying for?
Anyone can edit Wikipedia, and some edits get undone. We tested whether AI can spot those edits as they happen, and whether a paid AI is worth its cost compared with a free one.
Most undone edits aren’t vandalism
We expected undone edits to be mostly vandalism. They aren’t — most are honest edits caught up in a disagreement or a clean-up. So this project predicts “will this edit be undone?”, not “is this vandalism?”.
Show the details
50 undone and 50 kept edits, sorted into vandalism or not by claude-opus-5-5 (AI model, blind to revert status; not hand-labelled). None of the kept edits were vandalism. "Undone" means reverted within 72 hours.
A free first step does a lot of the work
Before any AI, we skip edits from bots and from experienced editors (accounts at least 30 days old with 500+ edits). It costs nothing and removes most of the work — but it lets a few edits that later get undone slip past.
Show the details
| Hour of edits | What is skipped | Edits skipped | Undone edits lost |
|---|---|---|---|
| 2026-09-30, 12:00 UTC | Bots only | 1.2% | 0.0% 0.0%–1.4% |
| 2026-09-30, 12:00 UTC | Bots + accounts 4+ days old with 10+ edits | 84.1% | 36.3% 30.9%–42.1% |
| 2026-09-30, 12:00 UTC | Bots + accounts 30+ days old with 500+ edits (used) | 73.3% | 16.2% 12.3%–21.0% |
| 2026-09-30, 04:00 UTC | Bots only | 57.2% | 0.0% 0.0%–2.6% |
| 2026-09-30, 04:00 UTC | Bots + accounts 4+ days old with 10+ edits | 91.3% | 29.0% 22.2%–36.8% |
| 2026-09-30, 04:00 UTC | Bots + accounts 30+ days old with 500+ edits (used) | 87.1% | 24.1% 17.9%–31.7% |
| 2026-09-30, 10:00 UTC | Bots only | 2.8% | 0.4% 0.1%–2.5% |
| 2026-09-30, 10:00 UTC | Bots + accounts 4+ days old with 10+ edits | 78.5% | 22.4% 17.4%–28.3% |
| 2026-09-30, 10:00 UTC | Bots + accounts 30+ days old with 500+ edits (used) | 67.0% | 12.6% 8.8%–17.5% |
"Undone" means reverted within 72 hours, read from Wikipedia's own revert tags.
Which approach is the best judge?
Every approach looked at the same 1,000 edits. Of these, 17% were later undone — so flagging edits at random would be right about 17% of the time.
When it says “this will be undone”, how often is it right?
Higher is better. The dashed line is a random guess.
Out of all the edits that got undone, how many did it catch?
Higher is better — but an approach can catch more by flagging almost everything.
The paid AI (Gemini 3.8 Flash) is right 31% of the time when it flags an edit, against 21% for the free laptop AI; the free AI catches more undone edits only because it flags most of everything. Wikipedia’s older model is right even more often (41%) but flags so few edits that it catches only 29%. And a one-line rule — flag logged-out editors — is right 24% of the time, not far behind.
Show all approaches and prompts
| Approach (and prompt) | Right when it flags | Undone edits caught | Edits flagged | Ranking skill (AUROC) | Confidence gap (ECE) |
|---|---|---|---|---|---|
| Flag every edit that gets past the free filter baseline:filter-only | 16.7% 14.5%–19.1% | 100.0% 97.8%–100.0% | 100% | — | — |
| Simple rule: flag logged-out editors baseline:filter+temporary | 24.2% 20.5%–28.4% | 66.5% 59.0%–73.2% | 46% | — | — |
| Wikipedia's older model (ORES) baseline:liftwing-enwiki-damaging | 40.5% 32.2%–49.4% | 29.3% 23.0%–36.6% | 12% | 0.72 | 0.09 |
| Wikipedia's own model (revert risk) baseline:liftwing-revertrisk-language-agnostic | 22.2% 19.2%–25.5% | 88.6% 82.9%–92.6% | 67% | 0.74 | 0.46 |
| Gemini 3.5 Flash-Lite gemini:gemini-3.5-flash-lite__p1-plain | 27.6% 22.9%–32.8% | 50.9% 43.4%–58.4% | 31% | 0.70 | 0.19 |
| Gemini 3.5 Flash-Lite gemini:gemini-3.5-flash-lite__p2-guide | 30.0% 25.0%–35.5% | 51.5% 44.0%–59.0% | 29% | 0.69 | 0.18 |
| Gemini 3.5 Flash-Lite gemini:gemini-3.5-flash-lite__p3-reason | 27.7% 23.2%–32.8% | 55.1% 47.5%–62.4% | 33% | 0.69 | 0.20 |
| Gemini 3.8 Flash gemini:gemini-3.8-flash__p1-plain | 28.2% 23.7%–33.1% | 59.9% 52.3%–67.0% | 36% | 0.72 | 0.20 |
| Gemini 3.8 Flash gemini:gemini-3.8-flash__p2-guide | 30.6% 25.6%–36.1% | 53.9% 46.3%–61.3% | 29% | 0.72 | 0.16 |
| Gemini 3.8 Flash gemini:gemini-3.8-flash__p3-reason | 31.0% 26.0%–36.6% | 53.9% 46.3%–61.3% | 29% | 0.72 | 0.18 |
| groq:openai/gpt-oss-120b__p1-plain | unfinished — 979 of 1000 edits answered; not counted | ||||
| groq:openai/gpt-oss-120b__p2-guide | unfinished — 58 of 1000 edits answered; not counted | ||||
| groq:openai/gpt-oss-120b__p3-reason | unfinished — 8 of 1000 edits answered; not counted | ||||
| Small AI on a laptop (Gemma 3) ollama:gemma3:4b__p1-plain | 19.4% 16.8%–22.4% | 86.8% 80.9%–91.1% | 75% | 0.63 | 0.54 |
| Small AI on a laptop (Gemma 3) ollama:gemma3:4b__p2-guide | 21.4% 18.2%–24.9% | 74.9% 67.8%–80.8% | 59% | 0.63 | 0.40 |
| Small AI on a laptop (Gemma 3) ollama:gemma3:4b__p3-reason | 20.0% 17.0%–23.4% | 72.5% 65.2%–78.7% | 61% | 0.60 | 0.43 |
1,000 test edits that got past the free filter, frozen before any model saw them. Ranges are 95% confidence intervals. AUROC: how well the scores rank risky edits above safe ones (0.5 is a coin flip, 1 is perfect). ECE: the average gap between how sure a model says it is and how often it is right (lower is better).
Does the AI know when it’s guessing?
Each AI also says how sure it is. That only helps if “90% sure” really means right 9 times in 10.
Show the confidence charts
Each chart compares how sure a model said it was (across) with how often those edits were really undone (up). Points on the diagonal mean its confidence can be trusted.
The money question
The plan: let the free AI handle everything, and ask the paid one only when the free one is unsure. It didn’t work — the free AI never knows when it’s unsure (see above), so it almost never asked for help.
At English Wikipedia’s volume, asking the paid AI about every edit that passes the free step would cost roughly $17–28 a day at published prices.
Show the details
| Approach | Right when it flags | Undone edits caught | F1 | Sent to the paid AI | $ per 1,000 edits |
|---|---|---|---|---|---|
| heuristics only | 16.7% 14.5%–19.1% | 100.0% 97.8%–100.0% | 0.286 | 0.0% | $0.000 |
| local only | 20.0% 17.0%–23.4% | 72.5% 65.2%–78.7% | 0.314 | 0.0% | $0.000 |
| cloud only | 30.6% 25.6%–36.1% | 53.9% 46.3%–61.3% | 0.391 | 100.0% | $0.224 |
| ladder [0.6, 0.7) | 20.1% 17.0%–23.5% | 71.3% 64.0%–77.6% | 0.313 | 2.6% | $0.005 |
The ladder's rule (send an edit up when the free AI's score is between 0.6 and 0.7) was chosen on 200 separate practice edits and written down before the final test. Costs use published list prices retrieved 2026-10-04.
Adding help when it gets busy
The AI step runs as several copies of a worker. When edits pile up, more copies start; when it’s quiet, some stop. Over a full day of live Wikipedia edits, the number of copies followed the real workload.
Show the details
2026-10-07 04:58 to 2026-10-08 04:58 UTC, on a laptop Kubernetes cluster fed live from Wikipedia. Busiest hour: 25 edits a minute needed the AI, with 5.6 workers on average; quietest: 13, with 2.9. Correlation with the edits needing the AI, over 30-minute windows: r = 0.9193. Against all edits, including those the free filter skips: r = 0.1818 — that was the measure planned first; the second was chosen after seeing it, and both are reported. Each worker is limited to 5 AI calls a minute, standing in for a paid service's rate limit.
A waiting room for edits
A slow, rate-limited AI can’t keep up with the edit stream, so edits wait in line until it gets to them. Nothing is lost while they wait.
Noticing when an AI quietly changes
AI companies sometimes change a model without changing its name. We re-ask each AI the same 50 questions and compare its answers with last time.
Show the details
Each AI is re-asked 50 fixed test edits and compared with its own earlier answers; the checks repeat configurations already scored and never tune them. Alerts so far:
| When (UTC) | AI | Alert |
|---|---|---|
| 2026-10-04 06:41 | ollama:gemma3:4b__p1-plain (deliberately broken: 30% of answers flipped) | agreement 72.0% is below 90% (14 of 50 decisions changed) |
The check is written to run nightly as a scheduled job; so far it has been run by hand.
How it’s built
The tools behind the results, and what each one does — in a sentence.
Every edit as it happens, which edits were later undone, and details about each editor. Free, no login.
The language every part of the project is written in.
A queue that holds edits in order until the next step is ready, and keeps a record so tests can be replayed exactly.
Bundles the code and everything it needs into one box that runs the same anywhere.
Runs the workers, restarts any that get stuck, and adds more AI workers when edits pile up.
A small AI model running on a laptop, at no cost per question.
Cloud AI models that charge per question; the costs here use their published prices.
Wikipedia’s own prediction models, used as the bar to compare against.
Tests that run on every change, including spinning up the whole system on a throwaway cluster.
Hosts this results page.