Prototype scorecard · Global News & Business Content Operations
A three-layer metadata pipeline for incoming news: deterministic entity resolution, model-proposed controlled-vocabulary tags, and a conformance guard that decides what is allowed to reach the index. Built and measured against an editor-assigned key.
Straight-through rate is the number everyone asks for. It is meaningless on its own — a pipeline that auto-publishes everything scores 100%. The number that decides whether the threshold is safe is the third one.
Straight-through rate went down from 33.3% and that is the point. The completeness layer moved five under-tagged records out of auto-publish and into the review queue, which is the trade this system exists to let you make deliberately: 8 points of throughput bought a 21-point reduction in errors reaching customers. Both numbers move from one config value, so an editorial director can set that dial rather than accept whatever the model happens to do.
Every silent error in the previous run was an omission — a missing code, never a wrong one. The conformance guard is structurally blind to that: it answers "is every tag on this record allowed?", and a record with three correct tags and one gap passes every check in it.
| Measure | Value | Reading |
|---|---|---|
| Records blocked from auto-publish | 5 | Held back because a cue fired and its code was absent. |
| Silent errors prevented | 5 | Every blocked record genuinely had a tag error. |
| Blocked but actually clean | 0 | No editor minutes spent on a false alarm. |
| Gate precision | 100.0% | On 24 records. Expect this to fall on real volume. |
The cues are deliberately dumb: place names and demonyms mapped to geography codes, plus regex for regulators, courts and tax. They never add a tag. Adding one would put an unreviewed guess into the index, which is the exact thing this pipeline exists to prevent — so they only withhold auto-publish. That means precision is untouched by design and the entire cost is paid in editor time, which is a number the business can decide about.
The one remaining silent error is arguably mine, not the pipeline's. On A01 the model tagged SUBJ-ANTI and SUBJ-REG because the article says the deal "remains subject to antitrust clearance." My key lists only SUBJ-MNA. I think the model is right and the key is wrong — and I left it in rather than editing the key a third time, because tuning ground truth until the score looks good is how you end up measuring nothing at all. That call belongs to an editorial lead, not to me.
Scored against a hand-assigned key carried through the pipeline on every record. Precision is codes applied that the key agrees with; recall is codes in the key that were applied.
| Facet | Precision | Recall | F1 | Exact set | Failure mode |
|---|---|---|---|---|---|
| COMPANY | 100.0% | 100.0% | 1.000 | 100.0% | None on this set. Both disambiguation traps passed. |
| GEOGRAPHY | 100.0% | 95.0% | 0.974 | 90.9% | Misses a jurisdiction named only in passing. |
| INDUSTRY | 100.0% | 79.3% | 0.885 | 81.8% | Will not infer sector from a company it knows. |
| SUBJECT | 93.5% | 70.7% | 0.806 | 50.0% | Drops secondary subjects; SUBJ-REG most often. |
Recall is the weak axis on every facet, and that is the right shape rather than an accident — the confidence floor, the evidence check and the vocabulary membership test all trade recall away to buy precision. A researcher who gets an incomplete result set refines the query; one who gets a wrongly-tagged document stops trusting the index. 11 of the 12 SUBJECT misses landed in the editor queue, where a human sees them. The two SUBJECT false positives are both on one record and both defensible — see the note under the safety metric.
Every code the model proposes is tested against the vocabulary before it can reach the index: membership, facet match, verbatim evidence, retirement status, confidence floor.
| Check | Count | Reading |
|---|---|---|
| Codes proposed by the model | 98 | Across 22 enriched records. |
| Accepted from the model | 83 | 15.3% rejection rate. |
| Inherited from the taxonomy | 20 | Broader terms the guard derived, not model output. |
| Rejected below confidence floor | 15 | All 15 rejections were this check. |
| Hallucinated codes rejected | 0 | Model stayed in vocabulary on this set. |
| Ungrounded evidence rejected | 0 | Every quoted span was found verbatim. |
Two zeros that need saying out loud: on this set the model did not invent a code or fabricate a justification, so the guard never had to fire in anger. The guard is not there because the model is unreliable today — it is there so the answer to "what happens when it is" is a rejected tag rather than a corrupted index. To prove it live, delete a row from the vocabulary table and re-run: the next record that uses that code is rejected as HALLUCINATED_CODE and routed to a human, with no code change.
Four runs against the same key. Each finding came from the scorecard, not from reading outputs by eye — which is the point of building the scorecard first.
Auto-publish required the minimum confidence across every applied tag to clear 0.85. But a correctly applied broader term is legitimately tagged near 0.60, so a single valid roll-up disqualified an otherwise clean record. 87.5% of the queue went to human review.
Gate now tests the lead tag of each required facet → editor touch 87.5% to 58.3%The prompt told the model to add broader terms alongside narrower ones. It applied that inconsistently in both directions — adding IND-TECH where the key did not want it, omitting GEO-EU where it did. The vocabulary already encodes broader/narrower, so this never needed a judgement call.
"Return only the most specific code" was read as "return one code", and the model started dropping co-equal secondary subjects. An environmental enforcement action is also a regulatory action; neither is the other's ancestor. One instruction was doing two jobs.
Rule split into two explicit halves → SUBJECT recall 65.9% to 70.7%, precision to 100%Codes applied included taxonomy-inherited broader terms; codes proposed did not. The metric compared two different populations. A reporting bug, but the kind that quietly makes a dashboard untrustworthy.
Rejection rate now counts model-originated codes only → 15.3%, inherited reported separatelyEvery error that reached auto-publish was an omission, and the guard only tests tags that are present. A record can be three-quarters right and pass every check. So the fix was not a better prompt — it was a second question the system had never been asked: does this record look under-tagged?
Deterministic completeness cues block auto-publish → silent error rate 37.5% to 16.7%, review queue precision 78.6% to 87.5%Four of these five findings were defects in how I measured, not in what the model produced. That ratio is normal and it is the argument for building the scorecard before tuning anything: without a key carried through every record, run 1's 87.5% touch rate looks like a model problem and gets answered with prompt engineering that would never have fixed it.
A queue full of records a human would not have changed teaches editors to click through, and then the gate protects nothing. This is the metric that stops the threshold from being set too conservatively.
| Measure | Value | What it means |
|---|---|---|
| Review queue precision | 87.5% | 14 of 16 routed records genuinely needed a human. |
| Unnecessary reviews | 2 | Records a human would have approved unchanged. |
| Duplicates suppressed | 2 | 8.3% of the queue; both correctly matched to their tier-1 canonical. |
| Enrichment failures | 0 | Failures route to specialist review; there is no path from error to auto-publish. |
Both unnecessary reviews were triggered by the same thing — an INDUSTRY lead tag between 0.70 and 0.85 on a record that was otherwise correct. That is a per-facet threshold problem, not a model problem, and it is the first thing I would tune with a larger key. Note that adding the completeness layer raised queue precision rather than diluting it: every record it added to the queue belonged there.
The pipeline reports concepts it could not express, which turns taxonomy change requests into an evidenced queue rather than an argument.
IND-MEDIA — which is the behaviour that makes the gap visible instead of hiding it inside a wrong tag.An earlier prompt revision produced 21 gap reports on 24 records by flagging every uncodeable detail — figures, dates, sub-aspects of concepts already coded. Tightening the instruction to "a central subject with no code at all" took it to one or two per run. A gap report that cries wolf costs a taxonomist real time, so its precision matters as much as the tagging does.
Each phase ends at a measurement, not a date. Nothing widens until the previous gate holds for two consecutive weeks on live volume.
Runs on live ingest and writes to a staging index nobody reads. Editors tag as they do today; the two outputs are compared nightly. Builds the golden set as a by-product of work already happening, at 300–500 records per language.
Gate Precision ≥ 97% per facet and a signed-off key large enough to detect a 3-point drift.Auto-publish enabled for a single content stream with the highest measured precision. Editors get a one-click reject that writes straight back to the key. Thresholds tuned per facet using the review-queue precision number, not per pipeline.
Gate Silent error rate under 2% and review queue precision above 70% — editors must not learn to rubber-stamp.Extend to the remaining streams and offshore shifts, with the scorecard sliced by vendor, shift and time zone. Taxonomy governance takes ownership of the vocabulary table; the pipeline stops being something engineering has to redeploy.
Gate Quality holds within 2 points across every shift, and a taxonomist has shipped a vocabulary change with no engineering involvement.Stated plainly, because the gap between a 24-record prototype and a global content operation is where the actual work is.
GEO-US-DC for "published in Washington") that two competent indexers would split on. Until I know the human baseline, I do not know whether 70.7% SUBJECT recall is bad or roughly human.