Metadata Enrichment Agent
Tags incoming news with controlled-vocabulary subject, industry, geography and company codes as it arrives, and decides which articles can publish without an editor.
For a businessClassifies incoming news automatically, so editors spend less time tagging and more time editing, while wrong tags are stopped before they ever reach readers.
01 · ContextThe problem
Editors tag every incoming article by hand against a fixed taxonomy. An LLM can do that in seconds, but a wrong code that publishes silently does more damage than a slow correct one.
So the agent had three jobs: tag accurately, prove where every tag came from, and know which articles still need a person.
02 · PipelineHow it works
- Load the vocabulary
The controlled vocabulary lives in a Data Table, so taxonomists can add or retire codes without touching the workflow.
- Remove near-duplicates
Articles are compared with 5-word shingles at
Jaccard ≥ 0.50. The canonical copy is picked by source tier, then body length, then date, and duplicates are suppressed before any tokens are spent. - Layer 1: match companies
Word-boundary regex against a 15-record authority file, with a common-noun guard so
"shell companies"never matches Shell plc. - Layer 2: LLM tagging
Claude proposes subject, industry and geography codes. It can only choose companies from the Layer 1 candidates, and labels each one primary or passing mention.
- Layer 3: taxonomy guard
Rejects
HALLUCINATED_CODE,FACET_MISMATCH,UNGROUNDED_EVIDENCE(the quoted span must appear word for word), retired codes with no successor,BELOW_CONFIDENCE_FLOORandFACET_CAP_EXCEEDED. Retired codes with a successor are remapped, and broader terms come from the hierarchy, not the model. - Route every record
AUTO_PUBLISH,EDITOR_REVIEW,SPECIALIST_REVIEWorDUPLICATE_SUPPRESSED. If the LLM call fails, the record goes to a person. - Score and brief
Precision, recall, F1 and exact-match rate per facet, plus cost per 1,000 articles. An LLM-written run brief may only cite scorecard figures, and the Word export still ships if the brief call fails.
03 · ChoicesDesign decisions
Silent error rate is the headline
A tag an editor reviews costs minutes. A wrong tag that auto-publishes costs trust. The number that matters most is auto-published records carrying a wrong tag.
Cues can block, never add
Place names, regulators, industry words and the company's registered industry catch under-tagging. They can stop an article from auto-publishing, but they never add a tag, so they can't introduce errors.
Per-facet confidence floors
Subject 0.85, industry 0.70, geography 0.85. Below 0.60 is rejected, below 0.75 is marked provisional, and each facet is capped at 4 tags.
Secondary tags earn their place
A secondary subject has to clear the same confidence bar as the lead tag, so the model can't pad records with weak extras.
04 · EvidenceHow I tested it
A 24-article answer key written the way editors would tag, with planted traps: "shell companies," a passing mention, retired-code bait and a renamed-company alias.
Across 5 test runs the error rate fell from 37.5% to 16.7%. Company matching scored 100% precision and recall.
05 · IterationTuning log
- Run 42Dropped the industry floor to 0.70. Straight-through rose from 20.8% to 45.8%, but silent errors went from 1 of 5 to 3 of 11. Reverted.
- Later runRe-applied the 0.70 floor once completeness cues were in place. Straight-through settled at 33.3%.
- Prompt changeMerging two prompt rules into one cost 16 points of subject recall.
- Gate changeGating on the lowest-confidence tag instead of the lead tag sent 87.5% of clean records to review.
06 · Honest notesKnown limits
- The answer key is 24 articles, so gate precision will likely fall on real volume.
- The company authority file holds 15 records; production would need the full list.
- The scorecard includes a 90-day rollout plan for moving from test set to live ingest.
Want to see this one run?
I'll screenshare the workflow, the test set and the results, including what didn't work.