Assessment Template Test Harness
Runs any conversation transcript through an assessment template, so prompt changes are regression-tested on real input instead of judged by eye.
For a businessLets a product team change its AI prompts with confidence: every change is checked against real conversations before it can affect a social worker's records.
01 · ContextThe problem
The product turns recorded social-care conversations into structured assessment records. Prompt changes were being judged by reading a few outputs, which makes it easy to fix one field and quietly break another.
In social care that matters: a softened risk or an invented diagnosis in a record is a real harm, not a formatting bug.
02 · PipelineHow it works
- Paste a transcript
A form trigger takes any transcript, so the same input can be rerun after every prompt change.
- Fill the template
Claude applies an occupational therapy assessment template at temperature 0.1 with a 2,000-token limit.
- Run the self-check
The prompt ends with an 11-item checklist the model works through before it returns anything.
- Return the record
The structured record comes back in the form: 5 output fields across 6 allowed assessment types.
03 · ChoicesDesign decisions
Source-only, always
Use only what is in the transcript. Never infer a diagnosis, mental capacity, risk or eligibility.
Never soften a risk
If someone states a risk, it goes in the record as stated. Contradictions are recorded with who said each side, not resolved.
Exact figures, safe redaction
Numbers are kept exactly as spoken, and redaction tokens pass through untouched.
Voice set per field
Third person for the reason for contact, first person for the strengths fields, strengths-based wording under the Care Act 2014. Undiscussed fields get a fixed "Not discussed in this conversation."
04 · EvidenceHow I tested it
The companion dashboard was built from five table extracts: 200 recorded conversations, 25 practitioners and 3 organizations. Every figure is computed in the browser from the raw rows.
It tested 15 candidate drivers of summary quality with permutation tests. None survived multiple-comparison correction, and the dashboard reports that instead of publishing a result that only looked significant.
A separate verification script re-derives every headline figure from the source data.
05 · Honest notesKnown limits
- The harness covers one template (v1).
- It is deliberately small: two nodes. The value is in the prompt rules and a repeatable input, not the plumbing.
Want to see this one run?
I'll screenshare the workflow, the test set and the results, including what didn't work.