The model that writes the best support email is not automatically the model that handles the best phone call.
A spoken turn puts constraints on a language model that a text box never does, and most of those constraints are about behavior rather than knowledge. Knowing which behaviors to look for turns model selection from a taste test into something you can measure.
- Behavior beats capability On a call the deciding factors are timing, brevity and restraint, not the reasoning depth a benchmark measures.
- Test the opening, not the essay Callers hear the first few seconds of a reply. Score how a model starts and how soon it stops rather than the whole answer.
- Long calls expose different faults Instruction drift, repetition and register changes only appear after many turns, so evaluations built on single exchanges miss them.
Leaderboards answer a different question. They score reasoning, coding and long-form writing, all of which a caller waiting in silence never sees. A model can sit near the top of every one of them and still make an agent that people ask to be transferred out of.
What separates a good call model is a short list of habits under pressure: how quickly it starts, how soon it stops, what it does when it is cut off, and what it does when the material it was given does not contain the answer.
Why a Strong Text Model Can Still Be a Weak Call Model
A reader controls a page. They skim, jump to the part they want, scroll back to re-read a number and take as long as they like. Every one of those affordances disappears on a phone call, and the model's output has to work without them.
- Delivery is linear. The caller hears the reply in the order it was written and cannot look ahead.
- Delivery is timed. A long answer occupies the line whether or not the caller still wants it.
- There is no formatting. Headings, bullets and bold text either vanish or are read aloud as noise.
- There is no scrollback. A reference number said once and not repeated is gone.
- Silence is expensive. Any second spent thinking is a second the caller hears as nothing happening.
Training that optimizes for text helpfulness tends to push in exactly the wrong direction here. Thoroughness, structure and careful hedging are rewarded on a page and punished on a line.
The selection question is not which model is strongest. It is which model behaves well when it cannot be thorough, cannot be formatted, and cannot make the listener wait.
The Behaviors a Spoken Turn Demands
Six behaviors account for most of the difference between a model that sounds natural on a call and one that does not. Each is testable, and none of them is what a general benchmark reports.
The first token arrives soon after the prompt is complete, because synthesis cannot begin until there are words to say.
Answers the question and stops, rather than adding the three related things a reader might also want.
Treats a caller cutting in as the new turn and abandons the half-spoken sentence instead of finishing it.
Says it does not have the detail when the material it was given is thin, instead of producing something plausible.
Produces numbers, dates and names in forms a voice engine pronounces correctly on the first attempt.
Still following the rules it was given at the start of the call on the twentieth turn, not only the second.
Some of these can be improved with prompting and some cannot. A model with a strong pull toward long structured answers can be shortened for a few turns, then drifts back as the conversation history grows and the original instruction sinks further into the context.
Starting Fast and Stopping Early
The timing number to compare candidates on is how long until the first token, not throughput once generation is under way. A model that begins promptly beats one that finishes promptly, because the voice step can already be working on the first clause while the remainder is still being produced.
Prompt size usually moves this number more than the choice of candidate does, and that breakdown sits in where the delay in a spoken turn actually comes from. What belongs to the candidate itself is steadiness: whether it starts at a similar point every time or is erratic about it.
Stopping is the other half, and it is the tendency most models have to be argued out of. Length is not neutral on a call. Every extra clause is time the caller cannot interrupt without feeling rude, and information they have already stopped retaining.
| Convention that reads well on a page | What it does on a call | What to look for instead |
|---|---|---|
| Restating the question first | A whole clause before any new information | A reply that opens with the answer |
| Bulleted lists with headings | Becomes an unbroken run of clauses when spoken | One fact per turn and an offer to continue |
| Caveats placed up front | Buries the answer past the caller's patience | The qualification after the answer, if at all |
| Long complete sentences | Leaves no natural place for the caller to come in | Short sentences with real pauses between them |
| Markdown and symbols | Voiced as stray characters or silently dropped | Plain prose the synthesis step can voice |
Score this by counting words in the first reply to a set of real caller questions, and again on the tenth reply of the same conversation. A model that holds brevity across both is worth far more than one that manages it once.
Interruption Recovery and Instruction Persistence
When a caller talks over the agent, the audio stops. The mechanics of detecting that are covered in how an agent notices it is being spoken over. What reaches the model afterwards is an awkward state: it produced half a sentence, the caller heard some unknown fraction of it, and there is now a new utterance to answer.
Models handle that state differently, and the differences are visible immediately. Some resume the abandoned sentence as though nothing happened. Some answer the interruption but then append the rest of what they were saying. The behavior you want is to drop the unfinished thought entirely and respond to what the caller actually said.
A related fault shows up when a caller asks for something again. Asked to repeat, a weaker model replays the previous sentence word for word. The caller who did not understand it the first time hears exactly the same words at the same speed, which is the moment many people give up and ask for a person.
Instruction persistence is the slower failure. Rules set at the start of a call, such as always confirming a reference number back or never quoting an amount, can be followed for several turns and quietly abandoned once the history grows. How that history is assembled matters too, and is covered in how a voice agent carries context across turns.
Both faults are invisible in short tests. They need a scripted conversation of fifteen or twenty turns, with the rule-breaking opportunity placed near the end rather than the beginning.
Grounding, Numbers and Names
The most consequential behavior is what a model does when the material it was handed does not contain the answer. Some say so. Some produce something that fits the shape of an answer and sounds entirely confident.
Voice sharpens the cost of that. There is no citation to click, no source shown alongside, no visible hesitation. A spoken answer arrives with the same tone whether it came from a document or from nowhere, and the caller has no way to tell the difference. The countermeasures are set out in keeping a voice agent from inventing answers.
Test it deliberately. Give the model a retrieval result that is genuinely off-topic, or one that answers a neighboring question rather than the one asked, and see what comes back. How the retrieval itself is built is a separate subject, covered in grounding answers in documents during a live call.
Output That Survives the Voice Step
A model writes text and something else has to say it. How the text is written changes how it sounds, and models vary in how carefully they produce forms that speak well.
- Dates written in numeric form, which different engines read in different orders.
- Reference numbers run together as one long string rather than grouped into speakable chunks.
- Currency and unit symbols, which may be spoken, skipped or mispronounced.
- Abbreviations that are ambiguous when spoken, where the same letters are a word and a shortening.
- Names copied from a document with formatting still attached to them.
Most of this is fixable with instructions and a normalization pass before the text reaches the voice. It is still worth knowing which candidate needs the least of that work, because every rule added to the prompt is prompt the model reads before it can start speaking.
Building an Evaluation From Your Own Calls
None of the above can be read off a published score. It has to be measured against the calls you actually receive, using the prompt and knowledge you actually intend to ship.
- Build fixtures from real transcriptsTake genuine caller turns, including the messy ones with self-corrections, background noise artifacts and half-finished sentences.
- Score the opening, not the essayMeasure time to the first token and the word count of the first sentence, since that is what the caller experiences.
- Run long conversationsFifteen turns or more, with the instruction you care about tested at the end rather than the start.
- Starve the context on purposeInclude cases where retrieval returns nothing useful, and record whether the model declines or fills the gap.
- Listen, do not readPlay the candidate output through the voice you intend to ship. Problems with numbers and names are audible and invisible on the page.
Re-run the whole set whenever the prompt changes, not only when the model does. A prompt edit made to fix one behavior routinely breaks another, and on a call the broken one is usually brevity.
One more result is worth expecting: no single candidate wins every row. Sending different turns to different models is a design in its own right, described in how one agent uses more than one model.
Conclusion
Choosing the best LLM for a voice agent requires more than comparing model size, reasoning benchmarks, or text generation quality. Real-world voice performance depends on fast responses, concise conversational answers, interruption recovery, reliable grounding, and consistent instruction following.
Evaluate candidates using realistic caller scenarios, extended conversations, and measurable response-quality metrics. By testing both model behavior and the complete voice pipeline, teams can choose a model that supports accurate, responsive, and natural customer conversations.
Frequently Asked Questions
Not reliably. Size tends to help with reasoning and knowledge, neither of which is usually the constraint on a call. Larger models are often slower to produce a first token and more inclined to write long, thorough answers, both of which work against a spoken turn. Judge candidates on timing and restraint first, then on capability.
Partly. Instructions and examples shorten replies noticeably in the first few turns. The question is whether brevity holds after fifteen turns, once the conversation history has grown and the original instruction is a long way back in the context. Test late in a conversation rather than early, because that is where the difference appears.
Compare time to the first token rather than total generation time, and record the spread rather than a single figure, because an erratic candidate is harder to live with than a uniformly slower one. Keep the prompt identical across candidates, or the exercise is measuring your prompt instead of the models.
Answering from nothing. When retrieved material does not contain the detail, some models produce a confident-sounding answer anyway, and a spoken reply carries no citation or visible source to give the caller a clue. Build fixtures that deliberately return thin or off-topic context and record which candidates decline.
Those are different steps with their own engines, and the criteria here apply only to the model that decides what to say. Recognition quality and voice quality are chosen separately and affect different parts of the call. The one overlap worth noting is that the wording a model produces changes how well any voice speaks it.
