Latency in a voice agent is not one number.
It is a chain of stages between the moment a caller stops speaking and the moment they hear a reply, and each stage can be fast on its own while the conversation still feels slow. Understanding which stage owns which part of the delay is what makes the problem fixable.
- Measured from the last word Latency is the gap between the caller finishing and the first audio of the reply, not total processing time.
- The stages overlap Streaming means recognition, generation and synthesis run partly in parallel rather than one after another.
- Endpointing dominates Deciding the caller has stopped talking is usually the largest single source of perceived delay, and is paid on every turn.
Text chat forgives delay. A reply that takes two seconds to appear reads as normal. On a phone call the same two seconds is an awkward silence, and the caller starts talking again, which creates a second problem on top of the first.
The chain has four working stages and one that people forget: deciding the caller has actually finished. That last one is usually the largest single contributor, and it is the one least often measured.
The Chain a Single Spoken Turn Passes Through
A turn is not a request and a response. It is five decisions in sequence, and a caller experiences only the gap between the last word they spoke and the first word they hear.

- Audio transport: the call carries the caller's voice from their handset to wherever the agent runs.
- Speech recognition: audio becomes text, usually as a stream of partial results rather than one block at the end.
- Endpointing: something decides the caller has finished a turn rather than paused mid-sentence.
- Generation: the model decides what to say, producing text token by token.
- Speech synthesis and playout: text becomes audio and that audio starts reaching the caller.
Measured badly, this looks like five numbers that add up. Measured properly, several of them overlap, which is why a system with a slow-looking model can still feel responsive and a system with fast components can still feel sluggish.
The number that matters is time to first audio: the gap between the caller's last word and the first sound of the reply. Total processing time is an engineering metric, not the caller's experience.
Speech Recognition and the Endpointing Decision
Modern recognition is streaming. Text starts appearing while the caller is still speaking, so recognition itself rarely dominates the delay. What dominates is deciding when the caller has finished.
That decision is a trade. Wait longer and you are confident the turn is over, but every caller pays that wait on every turn. Wait less and the agent interrupts someone who was drawing breath mid-sentence, which costs far more than the milliseconds it saved.
The cost is asymmetric, and this is what teams get wrong. A short extra pause is mildly annoying. Being cut off while reading out an order number is the thing that makes people ask for a person.
Callers reading a number, a reference or an address pause differently from callers describing a problem. An endpointing setting tuned on conversational speech will cut off people reading data aloud, so this is worth testing against the calls you actually receive rather than a generic sample. The mechanics of this are covered in turning spoken requests into structured data.
The Model Step: First Token, Not Total Tokens
A language model produces text one token at a time. The useful measurement is therefore how long until the first token arrives, because synthesis can begin on the opening words while the rest is still being generated.
Two things push that first token later. The first is prompt size: everything the model has to read before it can start writing. A long system prompt, a full conversation history and several retrieved documents all delay the opening word.
The second is anything that has to happen before generation starts. A knowledge lookup, a live API call to check an order, a policy check against business rules: each one is a round trip the caller waits through in silence unless the agent says something first.
| What adds the delay | Why it costs time | What usually helps |
|---|---|---|
| Long system prompt | The model reads it before writing anything | Move stable instructions out of the per-turn prompt |
| Full conversation history | Grows with every turn of the call | Summarize earlier turns rather than replaying them |
| Retrieved documents | Retrieval runs before generation starts | Retrieve fewer, shorter passages |
| Live data lookup | A network round trip in the middle of a turn | Start the lookup as soon as the intent is clear |
| Long replies | Only affects the tail, not the first word | Usually not worth optimizing for latency |
That last row matters because it is counter-intuitive. Shortening a reply from forty words to twenty barely changes what the caller perceives, since they heard the first word at the same moment either way. How retrieval fits into this is covered in retrieval-augmented generation for voice agents.
Speech Synthesis and the First Sound the Caller Hears
Synthesis has the same shape as generation: what matters is when the first audio is ready, not how long the full sentence takes to render.
A synthesis engine that streams can begin producing audio from the opening words. One that renders a complete sentence before returning anything forces the caller to wait for the whole reply to be built, even though they will only hear the beginning of it for another second.
There is a quality trade here. Rendering a full sentence at once lets the engine place emphasis and intonation across the whole phrase, because it knows where the sentence is going. Streaming from the first words gives that up in exchange for speed.
Voice choice interacts with this too, and not only through raw speed: a voice that reads numbers and addresses clearly saves far more time than it costs, because it prevents the repeat request that adds an entire extra turn.
What a Caller Actually Perceives
Perceived responsiveness is not the same as measured latency, and the gap between them is where most of the practical wins are.
- Silence feels longer than sound. An agent that acknowledges the request before looking up the answer buys real time.
- Consistency matters more than speed. A steady rhythm feels better than a system that is usually instant and occasionally slow.
- The first turn is judged hardest. A caller who waits after the greeting decides the system is slow and hears everything after that through it.
- A wrong answer delivered fast is worse than a right answer delivered slowly, because it costs a correction turn and some trust.
The acknowledgement point deserves care. Filling every gap with stock phrases becomes its own irritation, and a filler played before a lookup that returns instantly makes the agent sound slower than it is.
Whether the agent should be interruptible during these moments is a design decision in its own right, covered in how a live call is handed to a person when the answer is that a person should take over.
Measuring It Honestly
Component benchmarks are easy to collect and easy to be misled by. A recognition engine and a synthesis engine can each be quick while the assembled system is slow, because the delay lives in the joins and in the endpointing decision.
- Measure end to endTime from the caller's last word to the first audio of the reply, on a real call, not from a component log.
- Measure the distributionReport the slow end as well as the average. Callers remember the worst turn of a call, not the median one.
- Separate the first turnConnection and warm-up costs land on the opening turn and distort the overall average if they are mixed in.
- Test with real call audioBackground noise, accents and people reading numbers aloud all change endpointing behavior.

Which numbers belong on an operational dashboard rather than in a one-off test is covered in the metrics worth tracking for a voice agent.
Conclusion
Voice AI latency depends on more than how quickly speech recognition, language models and speech synthesis operate individually. Callers experience the combined effects of endpointing, model response initiation, audio delivery and network conditions.
To improve responsiveness, measure the time between a caller's final word and the first audible reply on real calls. Track slow-turn patterns, investigate delays across STT, LLM and TTS, and tune endpointing without cutting callers off mid-sentence.
With effective streaming, reliable interruption handling and consistent monitoring, teams can make AI voice conversations feel more natural while maintaining answer quality and caller trust.
Frequently Asked Questions
There is no single correct figure, because it depends on the call type and what the agent is doing. Set a target from your own calls: measure the gap between a caller's last word and the first audio of the reply, and look at the slow end rather than the average. A lookup against a live system will always be slower than an answer from a document, so one target across all turns is the wrong shape.
In most assembled systems it is not a component at all. It is endpointing, the decision that the caller has finished speaking, because that wait is paid on every single turn. After that, the usual candidates are anything that runs before generation starts, such as a knowledge lookup or a live API call, and a synthesis step that renders a whole sentence before returning any audio.
Not necessarily. What the caller hears is the first word of the reply, so the measurement that matters is how long until the model produces its first token. A model that starts quickly but writes slowly can feel more responsive than one that computes the whole answer fast but only returns it at the end. Prompt length and anything that runs before generation often matter more than the model itself.
Because endpointing decided the turn was over during a pause. People pause mid-sentence for many reasons, particularly when reading out a number, a reference or an address. Tuning the wait shorter to reduce delay makes this more likely, which is why the setting is a trade rather than something to minimize. It is worth testing against recordings of the calls you actually receive.
They help when the wait is genuinely long, such as a lookup against a slow system, because silence feels longer than sound. They hurt when they are used on every turn, or when they play before something that returns instantly, since the filler itself becomes the delay. Use them where a measurable wait exists rather than as a general habit.







