Podcast Archive Assistant
Colleagues ask ten years of the podcast a question in their own Claude and get an answer built only from the transcripts. Code checks every quote against its source before the answer goes out, and a question the archive can’t answer is declined, not guessed.
For a businessStaff can ask ten years of interviews a question and get a sourced answer in seconds, with every quote checked against the transcript, so nobody repeats a guest’s words wrong or builds on something the show never said.
- Status
- Released to colleagues at my employer, who use it and rated its first answers
- Result
- 157 of 161 test questions handled as the answer key says, none of 33 trick questions answered as if true, and a 4.7% silent error rate
- How it was measured
- An answer key of 161 questions I wrote and checked by hand, scored in code; search is scored on a held-out half
- Biggest limit
- Search finds the right passage in its top five for 73% of held-out questions, against an 85% target
01 · Context The problem
Ten years of interviews hold answers to questions people at my employer ask every week: has the show covered this, who said it, and what exactly did they say. Finding out meant searching transcripts by hand. A chatbot answering from memory is worse than nothing here: it will invent a quote, credit the wrong guest, or agree with a question built on a false premise.
02 · Pipeline How it works
- Audit the transcripts first
Before anything was indexed, every transcript was compared word by word with its live episode page. 38 of 153 had been damaged when they were copied into the sheet (sections moved, speech dropped, fragments spliced), and were repaired at the source rather than patched in the index.
- Index two ways
Transcripts are cut into passages and indexed for keyword search, which catches exact names, and for meaning, with a small open-source model. The two rankings are combined.
- Answer in the colleague’s own Claude
The archive is 14 tools Claude can call: search, read a passage in context, find episodes by guest or topic, and check quotes. Each colleague’s Claude does the reading and writing, so a question costs the archive nothing.
- Check every quote in code
Before an answer goes out, code confirms each quote appears in the passage it cites, ignoring only case, curly quotes and line breaks. If every quote passes it is answered; if one fails it is flagged; a question the archive doesn’t cover is declined.
- Fix transcription slips under rules
Automatic transcripts mishear words. A quote may fix a slip in [brackets], but code decides which fixes are allowed: never a number, a “not” or a pronoun, at most two per quote, and the answer tells the reader to listen.
- Host it for the team
It runs on Google Cloud Run behind a key, and a missing or wrong key is refused. A small kit adds it to a colleague’s Claude, with a setup sheet and a guide.
03 · Choices Design decisions
The server never calls a model
It searches, fetches and checks. Each colleague’s own Claude reads and writes, so nobody’s plan is shared and the archive has no model bill.
Quotes are checked in code, not trusted
A model can say it quoted accurately and be wrong. A string check against the cited passage can’t be talked out of its answer.
Declining is a correct answer
A question built on a false premise gets the premise named and corrected with quotes, or a decline. Answering it as asked counts as an error.
Scored on questions nothing was tuned on
The answer key is split in two. Settings are chosen on one half, and the headline search number comes from the half no setting was chosen on.
Keyword search won, and it says so
The design assumed combining keyword and meaning search would beat either one. On this archive keyword search alone did better, so that is the first thing to fix, not to hide.
A person confirms what code can’t
Answer-key gaps, speaker labels and listening fixes are proposed by code or Claude and confirmed by a person, and each ruling is recorded.
04 · Evidence How I tested it
An answer key of 161 questions, each checked by hand: 128 the archive can answer and 33 it can’t, including false premises and topics it never covered. Scored in code, the same way as the measured agents.
One full run of all 161: 157 routed as the key says (97.5%, likely range 93.8–99.0%), and none of the 33 trick questions answered as if true (likely range 0–10.4%). 14 of 127 answers cited an episode the key didn’t list; after I ruled on 26 possible gaps in the key (19 added, 7 rejected), 6 of 127 still do, a 4.7% silent error rate (likely 2.2–9.9%). The assistant didn’t change between the two scorings, only the key.
Search alone, on the held-out half: keyword search finds a passage with the answer in its top five for 73% of questions (likely 62–83%). Combined with meaning search it does worse, 64%.
183 automated tests, run on every push.
05 · Outcomes What it has done so far
- The CEO and the editor rated its first 9 answers: 5 useful as is, 4 useful with edits, none not useful. A small sample, so their notes count for more than the rate.
- Released to colleagues at my current employer: each has it in their own Claude, and all of them have used it.
06 · Security Untrusted input and what the model can touch
- Transcripts and questions are treated as data: the tools return text, and only code decides whether a quote stands.
- The hosted server needs a key kept in Google’s secret store; a missing or wrong key is refused.
- The usage log records searches and quote checks, never the answer text, and can be switched off.
07 · Honest notes Known limits
- Search misses too often: the right passage is in the top five for 73% of held-out questions, against an 85% target.
- Some answerable questions are declined when a guest said it in different words than the question uses: 4 in the full run.
- The answer key was written when 223 episodes had transcripts. 119 more have arrived since, so the next test run will re-check it.
- The first question after a quiet spell waits about 8.5 seconds while the server starts; later searches take 0.2–2 seconds. One measurement each.
- I wrote and checked the answer key myself; a second reviewer would make it stronger.
Want to see this one run?
I'll screen-share the workflow, the test set and the results, including what didn't work.