Bar-Compliance Content Reviewer
Checks law-firm marketing copy against attorney advertising rules before it goes live, and reports its own accuracy.
For a businessCatches risky claims in law-firm marketing before it goes live, so reviewers spend their time on real problems and clients stay clear of advertising-rule violations.
01 · ContextThe problem
Law-firm marketing has to follow attorney advertising rules that change from state to state. Keyword lists catch "guaranteed results" but miss copy that implies the same promise without the banned words.
Flagging everything isn't the answer either. A reviewer who keeps getting blocked on clean copy stops trusting the tool.
02 · PipelineHow it works
- Load clients and test set
Each client carries its state and its own confidence threshold.
- Strip disclaimers
A compliant disclaimer itself contains the word "guarantee," so disclaimers are removed before scanning.
- Layer 1: regex
Catches banned terms and required labels, such as New York's "Attorney Advertising" requirement under 7.1(f).
- Layer 2: implied violations
Claude looks only for what regex can't see: implied guarantees, implied superiority, unverifiable comparisons, statistics without context and award claims. It is told to be conservative.
- Merge and route
A per-client threshold (0.80–0.90) separates blocking findings from advisory notes:
BLOCK,REVIEW_REQUIRED,PASS_WITH_NOTESorPASS. If the LLM call fails, the item goes to review. - Score the run
Catch rate, false-positive rate, precision, straight-through rate, error rate and projected cost per 1,000 documents.
03 · ChoicesDesign decisions
Gating false positives come first
The headline metric is how often clean copy gets blocked, because that is what makes reviewers ignore a tool.
Every catch is attributed
The scorecard shows whether regex, the LLM or both caught each violation. If the LLM never catches anything regex missed, it isn't earning its cost.
State nuance stays advisory
Florida's pre-filing review is treated as advisory rather than blocking.
One bad call can't stop a batch
Requests run in batches of 5 with 3 retries, and errors route to review instead of halting the run.
04 · EvidenceHow I tested it
40 documents with ground truth across 10 clients in 5 states, including adversarial cases written to slip past keyword matching.
Costed at $2 / $10 per million tokens to project cost per 1,000 documents.
05 · Honest notesKnown limits
- Rule sets cover 5 states.
- It flags and routes. A person still signs off on anything blocked or marked for review.
Want to see this one run?
I'll screenshare the workflow, the test set and the results, including what didn't work.