A ten-minute call is a context problem, not a speech problem

Voice agents are usually evaluated on the first thirty seconds. The greeting is clean, the transcription is accurate, the latency is good, and the demo ends. The failures live at minute seven, when the caller refers to something they said at minute two, the agent does not have it any more, and asks them to repeat it. At that point the caller stops treating the system as competent, and no amount of speech quality recovers it.

• The window fills with verbatim transcript. Ten to fifteen minutes of speech is a lot of tokens, most of them filler. Naively appended, the useful facts get buried among “um”, false starts and repeated confirmations long before the token limit is reached.
• Summarisation is used to compress, and it eats specifics. Under a latency budget, the summary is generated fast and cheap, and the thing that vanishes is exactly the material detail: a number, a date, a caveat the caller added in passing.
• State is kept in prose rather than in fields. If “what we know about this caller” is a paragraph, the agent cannot check what is still missing, and cannot tell what it has already promised.

All three are the same failure at different depths. The conversation is being stored as a recording of itself rather than as a record of what it established.

The design that holds is a structured state object updated every turn, alongside the transcript rather than instead of it. Concretely: the fields this call is trying to fill, their current values, their confidence, what has been explicitly confirmed by the caller, what the agent has committed to, and what remains open. The prompt for each turn is built from that object plus a short window of recent speech, not from the whole conversation.

This is also where the guardrails live. On a healthcare intake build, the agent worked to structured templates it fills in rather than free-writes, with hard limits on what it was permitted to ask, and ten-to-fifteen-minute intake calls were held in context. Templates are not only a safety control. They are the schema that makes long context tractable, because a field either has a value or it does not, and the model never has to re-read the call to find out.

Store what the conversation established, not a recording of the conversation establishing it.

Every voice agent will hand off. The measure of the system is what the person receives at that moment. Forwarding the call with a transcript attached is not a hand-off; it asks a human to do the reading the agent was supposed to have done. Six things have to arrive with the transfer, and a structured state object is the only way they can:

1. Who the caller is, and whether that identity was verified or merely asserted.
2. What they want, in one line, in their own words rather than a category.
3. What the agent has already done or promised: the commitments a human must not contradict.
4. The fields already filled, so nothing gets asked twice.
5. Why it escalated, by name: the caller asked, a guardrail tripped, confidence was low, or a limit was reached.
6. A link to the audio and the transcript, for the detail the summary omitted.

Get that right and the caller experiences one continuous conversation that changed voice. Get it wrong and they experience two conversations, the second beginning with “can you tell me what this is about?”, which is the exact moment the automation stops being an improvement.

The last piece is vocabulary, and it rewards being narrow first. On the clinical build, the system was trained on real dentistry vocabulary and note structure before anything else, and it was that depth in one specialty, rather than a general model applied thinly, that let quality hold across eight further specialties. Domain nouns are where transcription errors cluster, and a domain vocabulary is a smaller, more tractable artefact than a better model.

Test accordingly. The calls worth running against the system are not the clean ones: an accent the model was not trained on, a caller who interrupts, a busy room, and a caller who changes their mind at minute eight. That last one is the real test of whether you built a record or a recording. If you are looking at a voice agent for a front desk, ask to see that call rather than the demo, and ask what the human receives when it transfers.