How I make AI reliable
A demo proves an agent can do the thing once, on an example the person running the demo picked. This is the rest of it: the guards that stop a wrong answer reaching anyone, the answer keys I score against, and the numbers I report even when they are unflattering.
01 · Safeguards Five things every build has
The same five apply whether the build is a six-node workflow or a scheduled agent. Open any one for how it actually works.
Rules before reasoning
Regex, deduplication and entity matching run before the model, so it only ever chooses from grounded candidates.
Dive into the entity rule
The metadata agent matches companies with word-boundary regex against a 15-record authority file. The model is handed that candidate list and may not add to it — so a company it has never been shown cannot appear in the output at all.
The rule that earns its keep is small: an alias followed by an ordinary noun is dropped. shell companies, shell structures and shell accounts never resolve to Shell plc. The test set contains both a VAT-fraud article using the phrase and a genuine Shell plc article, because a rule that fixes one and breaks the other has fixed nothing.
Guards reject, they do not trust
Every claim the model makes is re-checked deterministically before anything moves.
See the rejection codes
A code that is not in the controlled vocabulary is rejected as HALLUCINATED_CODE. A real code returned under the wrong facet is FACET_MISMATCH. Quoted evidence is searched for verbatim in the source text, and a confident but fabricated justification fails as UNGROUNDED_EVIDENCE — deterministically, and for free.
Retired codes with a successor are remapped; retired codes without one are rejected. Confidence below the per-facet floor is BELOW_CONFIDENCE_FLOOR, and anything past the per-facet cap is FACET_CAP_EXCEEDED. Broader terms come from the taxonomy hierarchy, never from the model.
On the run you can replay on this site, the model proposed 97 codes and the guard rejected 13 of them.
Uncertainty goes to a person
Confidence floors are set per category, and a failed model call routes to review rather than being dropped.
Why the floors are per facet
Subject 0.85, industry 0.70, geography 0.85. One global threshold is the wrong shape: the facets are not equally hard, and a single number either over-blocks the easy one or under-blocks the hard one.
The floors are configuration, not code, so an editorial lead can move them without touching the pipeline. The threshold is also tested against the lead tag of a facet rather than the lowest one — gating on the minimum sent 87.5% of a clean queue to human review, because correctly-inherited broader terms sit at lower confidence by design.
Quiet by default
A brief raises an item only when it is clearly worth raising, and scoring prompts check their own calibration.
What that looks like in a prompt
Weekly briefs produce no filler sections on quiet weeks. If nothing moved, the brief says so in a line instead of manufacturing three paragraphs, because a report that always looks the same stops being read.
Scoring prompts carry an explicit calibration instruction — if most items score above 0.6 you are miscalibrated — because a model asked to rate things will drift upward until everything is important.
The agent grades itself, against a key written first
Decision-making builds run against a labeled set and report precision, recall, silent error rate, gate false positives and cost per 1,000 items.
The four numbers, and the one that decides
Straight-through rate is gameable on its own — a pipeline that auto-publishes everything scores 100%. Human touch rate is the operating bill. Unit cost is tokens and money per thousand, including work later thrown away.
Silent error rate is the one that decides whether the other three are safe to move: wrong output that went out with nobody reviewing it. Errors a guard or a person caught cost time. These cost trust.
These four numbers are not computed by hand. An open-source harness produces them from the workflow’s output and an answer key written in advance — and on one build it showed that 30% of the documents the workflow let through needed a person, though its own scorecard reported a 100% catch rate.
A record moving through those layers, end to end. The routes on the right are the only three places it can come out.
02 · Ground truth Write the answer key first
A key written after the run agrees with the model by construction, so the set gets labeled first — by the standard the work is actually judged against. Articles tagged the way an editor would tag them. Marketing copy marked against the advertising rules a bar association enforces.
Each build started with a small key: 24 articles for the metadata agent, 40 documents for the compliance reviewer. Sets that size show direction, not a production error rate, and the 24 proved it: zero silent errors in the same 8 articles three times there, 27% once the key grew.
Both keys now hold 500 records, each built as a trap and fingerprinted when the key is frozen, so a run stops if an answer changes underneath it. What the larger keys found.
See the ten traps planted in the set
A clean test set proves almost nothing, because the failures that matter are specific.
Half are paired: a trap and the genuine case it resembles. A rule that fixes shell companies and quietly breaks the real Shell plc article has not fixed anything, and only the pair shows it.
03 · Failure modes Wrong and missing are different problems
A conformance guard answers one question: is every tag on this record allowed? A record with three correct tags and one missing tag passes every check in it. That blind spot is structural — and in one earlier run, every single silent error was an omission rather than a wrong code.
So a second layer of deliberately dumb cues watches for under-tagging: place names mapped to geography codes, regex for regulators, courts and tax, and the registered industry of any company already matched. When a cue fires and its code is absent, the record cannot auto-publish.
Why the cues are allowed to block but never to add
Adding a tag would put an unreviewed guess into the index, which is the exact thing the pipeline exists to prevent. Because they only block, they cannot hurt precision — they convert silent errors into editor reviews.
That makes the layer measurable on its own terms: how many blocked records were genuinely wrong, against how many clean records it cost an editor time on. On the replayable run it blocked five records, all five genuinely had a missing tag, and no clean record was held up.
04 · Security Untrusted input, limited tools
Anything a workflow reads from outside — a news article, a guest application, a law firm’s marketing copy — is treated as untrusted, because text can carry instructions. The defenses are structural rather than a line in the prompt asking the model to behave:
- Outputs are constrained. The model can only return codes from the vocabulary and companies the rule layer found, and every claim must quote its evidence word for word, so hidden text cannot add a tag the source does not support.
- Tools are scoped. Each agent holds only the actions its job needs. The production agents reach the sheet and the mailboxes only through n8n workflows with a few named actions and fixed recipients, and every outbound step checks one emergency stop.
- Code decides the route. The model reports findings; thresholds in code decide whether something goes out, and rule layers run without the model at all.
- Nothing secret ships. The public workflow exports carry no credentials, webhook IDs, internal URLs or email addresses, and the build that produces them fails on a match.
The known gap is written down too: in the compliance reviewer, an instruction hidden in the copy could suppress the model’s own findings. Holding any copy that contains instruction-like text for a person is the next safeguard on that build.
Runs get the same suspicion as inputs. Two 500-record runs started together overloaded the model API and lost up to 309 of 500 calls; the failed calls went to a person, so the runs looked plausible. They measured the outage, not the workflow, and were discarded.
05 · Worked example Run 42, and why I put it back
One change: the industry confidence floor, dropped from 0.85 to 0.70.
Read only the first number and it is a clear win, which is exactly why the first number is not the one I manage to. The same floor went back in later, once the completeness cues existed to catch what it let through, and straight-through settled at 33.3%.
Two more entries from the same log
Merging two prompt rules into one cost 16 points of subject recall. Neither rule looked load-bearing on its own.
Gating on the lowest-confidence tag on a record instead of the lead tag sent 87.5% of a clean queue to human review — correctly-inherited broader terms sit at lower confidence by design, so the strictest-looking rule was the wrong one.
Neither was predictable from reading the change. Both were one run away from being found.
06 · Restraint Publish the null result
A take-home analysis examined 15 candidate drivers of summary quality across 200 recorded conversations. Tested with permutation tests and corrected for multiple comparisons, none survived. Several looked significant before the correction. The full analysis →
The dashboard reports that. Publishing the three that cleared an uncorrected threshold would have made a more impressive deliverable and a worse one, and the next person to act on it would have been acting on noise.
The same restraint applies to good news: a metadata run auto-published 7 records with zero tag errors, and seven is far too small a base to call a 0% error rate. The run’s own brief said so. When the answer key grew to a 500-article stress test, the rate was 27%.
How the reporting layer is itself checked
A scorecard is code, and code has bugs. A separate verification script re-derives every headline figure from the source rows, and the dashboards compute their numbers in the browser from raw data rather than from a summary someone typed.
Where a model writes the run brief it may only cite figures already on the scorecard — it interprets the evidence, it does not produce it — and if the brief call fails, the export still ships without it. An evaluation layer that can invent its own numbers is not an evaluation layer.
07 · Transfer The checklist
- Write the key before the run
A key written afterwards agrees with the model by construction.
- Plant the cases you are afraid of
Each trap came from asking what would embarrass this in front of the person who owns the process.
- Count the two failures separately
Wrong output and missing output need different defenses. A conformance check is blind to omission.
- Report the number that hurts
Lead with silent errors. If the safe number cannot be quoted, the throughput number is not a result.
- Change one thing and log it
Including the reversals. A tuning log is the only honest evidence a setting was chosen rather than inherited.
- Say what the sample cannot support
Sample size is a property of the evidence, not a caveat to bury.
- Re-derive the headline figures
A separate script recomputes every published number from the raw rows.
None of this is specific to news metadata. It is the same procedure for a support-triage agent, a document reviewer or a sourcing digest: decide what wrong means, label a set before you run it, plant the cases that scare you, and report the number you would rather not.