Assessment Template Test Harness
Runs any conversation transcript through an assessment template, so a prompt change can be re-run on the same real input and compared, instead of judged once by eye.
For a businessLets a product team change its AI prompts with confidence: every change is checked against real conversations before it can affect someone's record.
- Status
- Take-home assignment
- Result
- Not measured on its own: it re-runs one prompt on the same transcript so changes can be compared
- How it was measured
- The measured piece of the same assignment is the summary-quality analysis
- Biggest limit
- It covers one template, and comparing outputs is still done by eye
01 · Context The problem
The product turns recorded conversations into structured assessment records. Prompt changes were being judged by reading a few outputs, which makes it easy to fix one field and quietly break another.
In a record about a real person, that matters: a softened risk or an invented diagnosis is a real harm, not a formatting bug.
02 · Pipeline How it works
- Paste a transcript
A form trigger takes any transcript, so the same input can be rerun after every prompt change.
- Fill the template
Claude applies a structured assessment template at temperature 0.1 with a 2,000-token limit.
- Run the self-check
The prompt ends with an 11-item checklist the model works through before it returns anything.
- Return the record
The structured record comes back in the form: 5 output fields across 6 allowed assessment types.
03 · Choices Design decisions
Source-only, always
Use only what is in the transcript. Never infer a diagnosis, mental capacity, risk or eligibility.
Never soften a risk
If someone states a risk, it goes in the record as stated. Contradictions are recorded with who said each side, not resolved.
Exact figures, safe redaction
Numbers are kept exactly as spoken, and redaction tokens pass through untouched.
Voice set per field
Third person for the reason for contact, first person for the strengths fields, strengths-based wording where the template calls for it. Undiscussed fields get a fixed "Not discussed in this conversation."
04 · Evidence How I tested it
The same assignment included an analysis of 200 recorded conversations: 15 candidate drivers of summary quality tested with permutation tests. None survived multiple-comparison correction, and the report says so instead of publishing a result that only looked significant.
A separate verification script re-derived every headline figure from the source data. The company is not named here and its data is not published.
05 · Honest notes Known limits
- The harness covers one template (v1).
- It is deliberately small: two nodes. The value is in the prompt rules and a repeatable input, not the plumbing.
Want to see this one run?
I'll screen-share the workflow, the test set and the results, including what didn't work.