The tool that grades the agents
Every build on this site claims a number. This is the thing that produces the number — a command-line tool that takes what a workflow did, compares it against answers written in advance, and reports one figure above all others: how often something that needed a person went out without anyone checking it. I built it for my own work with Claude Code, keep iterating on it, and use it regularly on new builds.
01 · The problem A workflow reported a 100% catch rate. It was wrong.
A compliance reviewer for law firm advertising checks marketing copy against the rules a bar association enforces. A rule layer catches banned words outright; a model catches the violations that are only implied. When the model is confident enough, the copy is held for a person. When it is not, the finding is written down as a note and the copy goes live.
The workflow’s own scorecard counted those notes as catches, and reported that it had caught 100% of violations.
The harness asked a different question: not “did the model notice?” but “did a human ever see it?” Of the twenty documents that went live with no one reviewing them, six needed a person — 30%, and between 22% and 30% across three runs of identical code.
Every one of the six was a real violation that the model had found, stated at a confidence just below the threshold, and so filed as a note that nobody would read. That detail decides the fix. The threshold was not too high; the model was hedging. A rule change would have papered over it, so the prompt was changed instead — anchoring what each confidence level is supposed to mean.
HEADLINE
rate count 95% range
Silent error rate 30.0% 6 of 20 14.5-51.9%
Straight-through 50.0% 20 of 40 35.2-64.8%
Silent omissions 0.0% 0 of 40 0.0-8.8%
Review-queue precision 100.0% 20 of 20 83.9-100.0%
At this size one record moves the silent error rate by 5.0 points.
SILENT ERRORS 6 record(s)
E15 medium SP-SHOULD-REVIEW, SP-MISSING-REJECTED [llm]
went: PASS_WITH_NOTES should: VIOLATION
violations proposed, not applied: IMPLIED_GUARANTEE
The result on those 40 documents, measured across four fresh runs and judged against the run-to-run noise of the old version (section 02 has what 500 documents showed):
The finding that mattered more: the answer key itself was wrong
The improved prompt appeared to get less precise — 10 to 20 points worse. It was naming real violations that the answer key had simply never listed, so the harness counted correct findings as mistakes.
That is a failure mode worth taking seriously, because it points the wrong way: it tells you to revert a change that actually worked. So the harness now lists every value a model states that the key does not contain, for a person to rule on, and flags the ones that recur in every run. Twenty-five were surfaced this way and nine were accepted into the key.
An evaluation you never question eventually measures your assumptions instead of your system.
02 · At scale Twenty-four articles said zero. Five hundred said 27%.
Small answer keys are where every workflow starts, and they flatter it. So both workflows were scored again on answer keys of 500 records: the originals plus hundreds more, each one built as a trap.
On its 24-article key, three runs of the news metadata tagger sent 7, 8 and 8 articles out with no editor, the same 8 each time, without a single silent error. On 500 articles the same pipeline’s silent error rate was 27% (54 of 200). That is outside the range the small set reported, because a range only covers the chance of which records you happened to draw. It cannot cover a test set that leaves out the hard cases.
Most of the 54 were one gap: 36 were missing a regulation or litigation subject beside the main one. The prompt told the model not to tag broader terms and never said what a broader term was. A revised prompt fixed that. Then a new guard rung: when the model proposes a code at 0.40 or more that the confidence floor sets aside, the article goes to an editor instead of publishing.
What the guard costs
Every safeguard is paid for in reviews, so the report shows the price beside the gain. About 41% of articles now publish with no editor, down from 47% with the revised prompt alone (40% originally), and more than half of the reviews are ones the answer key says an editor did not need.
The threshold question is measured rather than argued: lowering the industry gate from 0.70 to 0.60 would publish 25 more articles, 3 of them silent errors; raising it to 0.80 would send 48 more to an editor to catch 4.
The compliance reviewer’s 6.7% did not hold at scale either. On 500 documents — 243 violations, 257 clean, 112 of those written to look like violations — its silent error rate is 9.8% (25 of 256, likely 6.7–14.0%). The 40-document figure sat inside its own wide range; the larger set shows what was behind it.
It also shows where. 15 of the 25 are award claims (“Named to Super Lawyers”), which the model rates 0.75 almost every time and never above 0.80, under every client’s threshold. At 40 documents that looked like one stray case. At 500 it is a pattern, and a pattern can be fixed with one rule instead of case by case, though proving that fix still needs documents it was not designed on.
A fix measured before it is run. Every finding the model states is recorded, so the route each document would take under a different policy can be recomputed exactly, with no new run:
| Policy | Silent errors | Needless reviews |
|---|---|---|
| As run: client thresholds 0.80–0.90 | 9.8% · 25 of 256 | 26 |
| Cap every threshold at 0.85 | 7.2% · 18 of 249 | 26 |
| Award claims always go to a person | 4.1% · 10 of 241 | 26 |
| Both | 1.3% · 3 of 234 | 26 |
| Every note goes to a person | 0% · 0 of 211 | 46 |
Measured on the same documents it would be tuned on, so a fresh run has to confirm it before it counts.
Runs that were thrown out. Two 500-record runs started at the same time overloaded the model API: 309 of one run’s 500 calls failed, and 117 of the other’s. The failed calls went to a person, as designed, so the runs looked plausible — they measured the outage, not the prompt. Both were discarded, the workflows now send model calls two at a time, and the harness warns about failed calls before comparing runs.
03 · Why a harness An answer key is the exam. This is the grader.
A set of right answers says nothing until something marks the work against it, and the marking is where both workflows went wrong. Each reason below comes from one of them.
- The workflow was grading its own exam
The compliance reviewer had an answer key and its own scorecard, which reported a 100% catch rate because it counted “went out with a note” as caught. Counting what actually happened, 30% of the copy that went out unseen needed a person.
- An average hides the mistakes that matter
In the metadata tagger’s first 500-article run, 99.3% of the tags it applied were correct. Yet 27% of the articles it published with no editor had an error. Scored as tags, it looked finished; scored as what went out, it was not.
- It says how far to trust a number
Zero errors on the same 8 articles in three runs looked perfect, and on a 500-article stress test the rate was 27%. Three runs of identical code gave 6.3%, 6.5% and 8.6%. Ranges and repeated runs are what separate a real improvement from a lucky run.
- It says why, not just how often
The model had found the violations and understated its confidence, so the fix was the prompt, not the threshold. 15 of 25 misses were award claims. Regex rules, not the model, caused most needless reviews. A pass/fail count points at none of that.
- It answers “what if” without a new run
Every finding is recorded, so another routing policy can be scored from the answers already on file: capping thresholds at 0.85 and sending award claims to a person comes out at 1.3%.
- It checks the answer key too
It lists what the model states in every run that the key does not contain, for a person to rule on. That surfaced nine real violations the compliance key had missed.
When a spreadsheet is enough: a workflow that never decides what goes out unseen, rarely changes, and only needs a count of how many it got right. The harness earns its keep when a workflow decides what goes out without review, keeps changing, and has to be trusted.
04 · How a run works Six steps, start to finish
Each step opens for the detail underneath it.
Write down the right answers first
Someone who knows the work marks up a set of real records — this one is a violation, that one is clean, this one should have gone to a person. That file is the answer key, and it is written before the workflow runs.
Why before, and not after
An answer key written after the run agrees with the workflow by construction: you look at what it did, and it looks reasonable. Writing the key first is the only way the comparison can surprise you.
The set is built out of traps — records chosen because they are the ones likely to go wrong. A company name that is also an everyday word, a claim that is only implied, a near-duplicate from a weaker source. A set of easy records proves nothing.
The current keys hold 500 records per workflow. Each record’s fingerprint is taken when the key is frozen, and a run stops if any record has changed since, so the answers cannot be quietly edited to match the output.
Run the workflow and export what it did
The workflow runs over that same set of records and produces its usual output: what it decided for each record, and whether each one went straight through, went to a person, or was blocked.
It does not care what the workflow is built in
The harness reads an n8n export, a Zapier export, a log file, an API response or a spreadsheet. A short configuration file says where the records are and what the fields are called; nothing about the tool assumes a platform.
The repository keeps the same six records in six different file shapes, and the test suite requires all six to load to identical results. That is what stops “it works on my export” from being the actual rule.
Pair every record with its answer
Each record the workflow produced is matched to its entry in the key, one to one, by id. Nothing is compared in aggregate.
Why totals are not enough
Totals can agree while the records underneath disagree. Two records misrouted in opposite directions cancel out perfectly in a summary, and the summary says everything is fine.
If a record went into the workflow and never came out, the run stops and names it, rather than quietly scoring the ones that survived.
Decide, once, whether each record needed a person
For every record the harness makes a single judgment: did this need a human, and did it get one? Every number on the report is a count over those judgments, so no two figures can contradict each other.
What “needed a person” means exactly
It is set per workflow, because it differs. Where the output is the thing being published (tags on an article), a wrong value is itself the error. Where the output is a reason for a verdict (the codes a compliance reviewer cites), what matters is whether the verdict reached the right place.
Each failure is also given a type and a severity: something that should have been blocked and was not is critical; a correct value discarded by a threshold is medium. A single rate cannot tell you which kind of mistake is getting out. The types can.
Report the number that matters, with its margin
The headline is the silent error rate: of everything that went out with nobody looking at it, how much needed a person. Beside it sits the range the sample can actually support.
Why every rate carries a range
“Zero errors in seven records” reads like a perfect score. It is also consistent with a true error rate as high as 35%. A bare percentage hides whether it came from 7 records or 700.
So every rate is printed with a 95% range, and the report states how far a single record moves the number at that sample size. The interval is a Wilson score interval, chosen because the textbook formula claims a range of exactly zero to zero when the count is zero — precisely the case where the honest answer is “we do not know yet”.
Run it again before believing it
The same workflow, unchanged, gives slightly different numbers each run. The harness measures that spread, and then judges any change against it — reporting whether an improvement is real or just noise.
What this caught in practice
Two documents in the compliance set flip between runs of identical code: their top finding lands at 0.75 twice and 0.85 once, and at 0.85 it clears the threshold. Same code, different outcome. A single run would have called that a fix or a regression.
Three runs were thrown out entirely because they were started at the same time, overloaded the model API, and 20–47% of their calls failed. They measured the outage, not the prompt. The harness now counts failed calls per run and warns before comparing one.
It happened again at 500 records: two runs started together, and 309 of one run’s 500 model calls failed, and 117 of the other’s. Both were discarded and rerun, and the workflows now send their model calls two at a time.
05 · The numbers What each one is for
One rate can always be gamed. These are reported together so that improving one at the expense of another is visible rather than hidden.
The ranges keep a small result honest: on the news metadata tagger’s first 24 articles, “zero errors in seven” still allowed a true rate up to 35%, and three runs that sent out the same 8 articles with zero errors allowed up to 32%. They say how much a sample can support. They cannot say whether the sample was the right one — section 02 is what happened when it was not.
Reporting the ceiling alongside the result is the difference between evidence and a screenshot. At 500 records the ceilings are narrow enough to act on: 5.5–13.1% for the tagger’s current version, in the worst of three runs.
06 · Trusting the tool Built to refuse a guess
A measuring instrument that quietly miscounts is worse than none at all. If this is going to be the thing that says a workflow is safe to run unattended, it has to be harder to fool than the workflow is.
- It stops rather than guesses
Every way the tool could itself produce a wrong number is written down — more than eighty of them — with what it does about each. A confidence written as text instead of a number, a route name in the wrong case, a spreadsheet using a decimal comma: it halts and names the record instead of scoring it wrong.
- The arithmetic has rules
Zero divided by zero reports “nothing to measure”, not 0%. An F1 score is computed from counts, never from already-rounded percentages. A confidence of 0.7 lands in the 0.7–0.8 bucket even though 0.7 divided by 0.1 is 6.9999… in binary.
- The tests were checked for teeth
Over 200 tests, with the expected answers worked out by hand before the code existed. Then the code was broken on purpose — precision and recall swapped, a safety check weakened — to confirm a test actually failed each time. A test suite that passes on broken code is decoration.
- It runs on every push
Linux and Windows, on three versions of Node. The build also re-runs the commands in the documentation, rebuilds the sample reports and fails if they differ from the committed copies, so the write-up cannot drift away from the tool.
07 · What it is for Where this goes next
Most AI workflows are judged on a demo and then on complaints. This sits in between: a number you can put in front of someone before it runs unattended, and a way to tell whether last week’s change actually helped.
It is deliberately not tied to a platform, because the question outlives the tool the workflow happens to be built in. Anything specific to a given workflow lives in its own folder; the core knows nothing about any of them.
The practice behind it
The five safeguards every build gets, the answer keys they are scored against, and a tuning change I reverted because it traded errors for throughput.
See the method Live · real runWatch a checker run
A real run of one of these workflows replayed in your browser: what the model proposed, what the deterministic check threw out, and where each record ended up.
Run it08 · The code Open, and runnable in one command
MIT licensed, no dependencies, and it needs neither a network connection nor an API key — it reads two files and prints a report. The repository ships the fixtures, the worked-out expected answers, and the real runs behind every figure on this page.