Metadata Enrichment Agent

The hard part of tagging news at ingest is not the tagging. It is deciding which articles can be trusted to publish with no editor looking — and knowing how often that call is wrong. An LLM can propose codes in seconds, but a wrong code that publishes silently does more damage than a slow correct one.

So the agent had three jobs: tag accurately, prove where every tag came from, and know which articles still need a person.

The deterministic half of this pipeline — the structural scan and the conformance guard — is plain JavaScript, so it can run anywhere. On 22 September 2026 the workflow processed a 24-article test batch, and what the model returned on each article was recorded.

Those two node bodies are shipped to a page on this site and executed in your browser over that recorded output, so you can pick an article and watch the guard adjudicate every label the model proposed. Replaying it reproduces the recorded run figure for figure — all 56 numbers on its scorecard.

Run 49 is later than the evaluation report, so its numbers are better: silent errors fell from 1 in 6 auto-published records to 0 in 7. Seven records is far too small a base to call a 0% error rate stable, and the run’s own brief says so. Subject recall, at 73.2%, is still the weakest facet, and the review queue is currently covering that gap instead of the model.

A 24-article answer key written the way editors would tag, with planted traps: "shell companies," a passing mention, retired-code bait and a renamed-company alias.

Across 5 test runs the error rate fell from 37.5% to 16.7%. Company matching scored 100% precision and recall.

Three fresh runs on those 24 articles published 7, 8 and 8 with no editor, the same 8 articles each time, and no silent errors, and I reported the ceiling with it: zero errors in 8 is still consistent with a true rate up to 32%. Then the answer key grew to 500 articles, 34 of them duplicates of another, each built as a trap. Scored by the evaluation harness, the same pipeline’s silent error rate was 27% (54 of 200). The small set was not representative, and a likely range covers chance, not a test set that leaves out the hard cases.

Most of those 54 were one gap: 36 were missing a regulation or litigation subject beside the main one. A prompt revision took subject recall from 82.8% to 94.7% and the rate to 11.6%. A new guard rung then took it to 6.5%, 6.3% and 8.6% across three runs of identical code, a drop bigger than the spread between them. The worst run, 8.6% (18 of 210, likely 5.5–13.1%), is the one to quote: the runs send out the same articles again, so adding them together (44 of 615) would claim a narrower range than the evidence holds.

The new rung’s threshold was chosen on half the articles and checked on the other half, where the three runs gave 5.4%, 6.6% and 7.6% (worst: 8 of 105, likely 3.9–14.3%). Duplicates held up at scale: 33 or 34 of the 34 suppressed in every run, and no real article ever suppressed.

What the build already does, and what I would add before it carried real volume. The second column is a plan, not a claim.