Bar-Compliance Content Reviewer
Checks law-firm marketing copy against attorney advertising rules before it goes live, and reports its own accuracy.
For a businessCatches risky claims in law-firm marketing before it goes live, so reviewers spend their time on real problems and clients stay clear of advertising-rule violations.
- Status
- Spec build: built to a real brief, tested on an answer key I wrote
- Result
- 9.8% of the copy it let through on a 500-document stress test still broke a rule, and 15 of those 25 misses were award claims
- How it was measured
- The evaluation harness, against a 500-document answer key with traps written to slip past keywords
- Biggest limit
- The fix for award claims is a what-if on the same 500 documents; it still needs documents it wasn’t designed on
01 · Context The problem
Attorney advertising rules change from state to state, and the violations that matter most are the ones that are only implied. A keyword list catches "guaranteed results" but not copy that makes the same promise without the banned words. That takes judgment, and judgment needs a confidence attached to it.
Flagging everything isn't the answer either. A reviewer who keeps getting blocked on clean copy stops trusting the tool.
02 · Pipeline How it works
- Load clients and test set
Each client carries its state and its own confidence threshold.
- Strip disclaimers
A compliant disclaimer itself contains the word "guarantee," so disclaimers are removed before scanning.
- Layer 1: regex
Catches banned terms and required labels, such as New York's "Attorney Advertising" requirement under 7.1(f).
- Layer 2: implied violations
Claude looks only for what regex can't see: implied guarantees, implied superiority, unverifiable comparisons, statistics without context and award claims. Its first prompt told it simply to be conservative, and it hedged the confidence of what it found; it now works from an anchored confidence scale (see the tuning log).
- Merge and route
A per-client threshold (0.80–0.90) separates blocking findings from advisory notes:
BLOCK,REVIEW_REQUIRED,PASS_WITH_NOTESorPASS. If the LLM call fails, the item goes to review. - Log every note
Copy that passes with notes goes out without review, so each note is written to a log table with the finding, its confidence, how far below the threshold it fell, and the words at issue. Sorted by code, it shows which problems keep coming back.
- Score the run
The silent error rate leads: a note no longer counts as a catch, because the copy still went out. Then catch rate, false alarms on clean copy, review precision, straight-through rate and projected cost per 1,000 documents.
03 · Choices Design decisions
False positives first, then silent errors
It was built to lead with how often clean copy gets blocked, because that is what makes reviewers ignore a tool. Scoring it externally showed the larger risk ran the other way: real violations going live as notes nobody reads. Both are reported now, and the silent error rate leads.
Every catch is attributed
The scorecard shows whether regex, the LLM or both caught each violation. If the LLM never catches anything regex missed, it isn't earning its cost.
State nuance stays advisory
Florida's pre-filing review is treated as advisory rather than blocking.
One bad call can't stop a batch
Requests run in batches of 5 with 3 retries, and errors route to review instead of halting the run.
04 · Evidence How I tested it
40 documents with ground truth across 10 clients in 5 states, including adversarial cases written to slip past keyword matching.
The workflow’s own scorecard counted a finding stated below the client’s threshold as a catch, and reported a 100% catch rate. Scored by the evaluation harness against the same key, 6 of the 20 documents that went live with no one reviewing them needed a person: a 30% silent error rate, and 22–30% across three runs of identical code.
Every one of the six was a violation the model had found and stated just below the threshold, so the fix belonged in the prompt, not the threshold. The same runs showed the answer key itself was incomplete: the improved prompt named real violations the key had never listed.
Then the answer key grew to 500 documents: 243 violations and 257 clean, 112 of them written to look like violations. The silent error rate there is 9.8% (25 of 256, likely 6.7–14.0%), and 218 of the 243 violations (89.7%) reached a person. The 6.7% on 40 documents sat inside its own range; the larger set shows what was behind it.
15 of the 25 silent errors are award claims (“Named to Super Lawyers”). The model rates them 0.75 almost every time and never above 0.80, under every client’s threshold. On the other side, 23 of the 26 clean documents sent to a person were flagged by regex rules that cannot read context: a fee counted as a result, “We cannot guarantee” read as a guarantee. The model earns its cost: 91 of the 218 violations that reached a person were found by the model alone.
Because every finding is recorded, another routing policy can be recomputed exactly with no new run. Capping thresholds at 0.85 and sending every award claim to a person gives 1.3% (3 of 234) with no new unnecessary review. That is measured on the same documents it would be tuned on, so a fresh run has to confirm it before it counts.
Token counts are recorded so cost per 1,000 documents can be projected.
05 · Production readiness What it would take to run at scale
What the build already does, and what I would add before it carried real volume. The second column is a plan, not a claim.
Monitoring
The scorecard leads with the silent error rate. Every note on copy that went out is written to a log table with its code, confidence and how far below the threshold it fell.
A weekly read of the notes log sorted by code: a code that keeps appearing on clean copy means a threshold is too low; on violations, too high.
Failures and retries
Requests run in batches of 5 with 3 retries, and a failed model call sends the copy to review. In the 500-document run, 2 calls failed and both went to a person.
A cap on the share of failed calls per run. Three earlier runs lost 20–47.5% of their calls to an overloaded API and were discarded; production should stop rather than report on them.
Drift
A frozen, fingerprinted 500-document answer key, and a list of values the model states that the key lacks, for a person to rule on.
State rules change. Rerun the golden set on every rule-set or prompt change, with the harness failing the release when a serious silent error gets through.
Cost at 100×
Token counts are recorded so cost per 1,000 documents can be projected. Today the model reviews every document.
Skip the model on copy the rule layer already blocks, since a person sees it anyway, and send the model only what could still go out unreviewed.
06 · Security Untrusted input and what the model can touch
- Marketing copy reaches the model as untrusted text. The model only returns findings; the route is decided by code against each client’s threshold, and the regex layer runs without the model, so banned words are caught even if the copy manipulates it.
- The gap: an instruction hidden in the copy could suppress the model’s own findings. A check that holds any copy containing instruction-like text for a person is the next safeguard I would add.
- Anything blocked or flagged goes to a person who signs off.
- The exported workflow carries no credentials, webhook IDs or internal URLs.
07 · Iteration Tuning log
- Prompt v1Told to be conservative, the model hedged the confidence of findings it did report. Silent error rate 22–30% across three runs.
- Prompt v2Anchored what each confidence level means, and kept conservatism for whether to report, not how confidently. 6.7% in four runs out of four, beyond the old version’s noise: five routes fixed, none broken, no new false alarms on clean copy.
- Key v2Nine violations the answer key had never listed were added after review, so the new prompt stopped being scored as wrong for being right.
- Golden 500Scored on 500 documents: 9.8%, with award claims 15 of the 25 silent errors. The recorded findings put a fix at 1.3%, pending a fresh run.
08 · Honest notes Known limits
- Rule sets cover 5 states.
- Award claims are now a pattern across 31 documents, not one document, so one rule can fix them instead of case-by-case patches. The fix is measured as a what-if on these same 500 documents and not yet run, so it still needs documents it was not designed on.
- Four regex rules cause 21 of the 26 unnecessary reviews: fees read as results, certifications that name their certifying body, certified specialists and expert witnesses, and denials of a guarantee.
- It flags and routes. A person still signs off on anything blocked or marked for review.
Want to see this one run?
I'll screen-share the workflow, the test set and the results, including what didn't work.