Fifteen hypotheses. None survived.
A take-home for an AI documentation product that turns recorded conversations into written summaries. The question of what makes those summaries better or worse was not in the brief — I asked it, tested fifteen candidates across 200 conversations, and the honest answer was that none of them held up. This is what I reported instead. The company is not named and its data is not published.
01 · The question What makes a summary good?
Users rate each AI-written summary. The obvious deliverable is a chart that ranks what drives those ratings — this model is better, longer conversations score worse, that team is struggling. It is the kind of chart a leadership team would act on.
It is also the chart most likely to be wrong. Ask fifteen questions of a small dataset and some answers will look real by chance alone. The only way to know which are real is to decide in advance how you will test them, and then accept the answer.
02 · How I tested it Five steps, fixed before looking
- Work from the raw rows
Five table extracts — transcripts, summaries, prompt templates, users and feedback. Every figure is computed in the browser from those rows, not typed in from a summary, so anyone can change a filter and watch the numbers move.
- Name the candidates before looking
Fifteen things that could plausibly drive how users rate a summary: conversation duration and length, the model, its temperature, the team, the topic, how much rework a summary needed, and others.
- Test each one without assuming a shape
Permutation tests: shuffle the ratings over and over and ask how often chance alone produces a difference this large. No assumption that the data is normally distributed — which matters when a single template rests on 13 to 25 ratings.
- Correct for having asked fifteen times
Run fifteen tests at the usual 5% threshold and roughly one will pass by luck. Every result was checked under two corrections, Benjamini–Hochberg and the stricter Bonferroni, at α = 0.05.
- Check the arithmetic independently
A separate verification script re-derives every headline figure from the source extracts. The dashboard and the script have to agree before anything is reported.
03 · The result Nothing held up
Three of the fifteen relationships reached p < 0.05 on their own. Across fifteen tests you would expect roughly one to do that by chance, and none cleared the corrected threshold.
| Claim | Evidence | Verdict |
|---|---|---|
| Rework predicts lower ratings | r = −0.17, p = 0.043 | Fails correction |
| Some models fail more than others | p = 0.37 | Not supported |
| Quality is worse in non-English conversations | Under 20 rated conversations | Too few to test |
| Ratings disagree with comment sentiment | Comments drawn from 15 repeated strings | An artifact of the extract |
The failed tests were shown in the delivered dashboard on purpose. A version that rendered them as confident bars would have looked more useful, and it would have sent a leadership team planning around noise.
Why a null result is worth reporting
“We found nothing” reads like a failed analysis. It is the opposite: it shows the data don’t support fifteen stories someone would otherwise have told, several of which were one uncorrected test away from being presented as findings.
It also says something specific about the data. Ratings on 13 to 25 summaries per template cannot separate small effects from chance. The next step is not a cleverer test on the same data; it is collecting the signal that would actually answer the question — which is section 05.
04 · Worth acting on Counts, which need no inference
The findings a team could act on by Monday were not statistical at all. They were things you can count, and none of them depend on a significance test. The report counted each of these; the counts belong to the company, so only what was counted is shown here.
05 · What to measure next Three columns would answer the question
The question stayed open because the data could not answer it. These would change that, and none of them needs a model:
- An owner on every transcript
So a stuck or repeated job can be traced to someone who can act on it.
- A completion timestamp on every job
So “stuck for over a week” is something the system reports, not something an analyst reconstructs.
- The edit distance between the AI’s summary and what the user saved
The strongest quality signal available, and it is not captured today. A rating says how someone felt; the edits say what was actually wrong.