How to reduce voice AI agent latency

Everything else about a voice agent can be excellent and it will still feel broken if it pauses too long before answering. Latency is not one metric among many in voice AI. It is the metric that decides whether a caller believes they are in a conversation or believes they are waiting on a machine.

This guide breaks down where the time actually goes, what you can do about each piece, and what you give up in exchange.

Why latency is the failure mode

Human conversation has an extremely tight turn-taking rhythm. The gap between one speaker finishing and the next starting is on the order of a couple hundred milliseconds — fast enough that listeners are effectively predicting the end of your sentence before you get there. That rhythm is deeply wired in. When it breaks, people notice immediately, even if they can't articulate what felt wrong.

A voice agent that takes two seconds to start replying breaks it badly. Three specific things happen, in order:

The caller repeats themselves. Silence on a phone line reads as "they didn't hear me." So the caller says it again, which arrives mid-inference, which either gets discarded or corrupts the turn.

The caller talks over the agent. They start their next sentence just as the agent finally starts its reply. Now you're handling barge-in on a turn that shouldn't have needed it.

The caller stops treating it as a conversation. They shift into a slower, more clipped, menu-navigating register — short phrases, loud enunciation, waiting after each one. The agent may still work, but the product experience has collapsed into an IVR with extra steps.

None of these are model quality problems. A more capable model that is slower makes all three worse. This is why latency work has to happen at the pipeline level and not just at the prompt level.

Where the time actually goes

A voice pipeline is a chain: telephony carries the audio in, speech-to-text turns it into words, an LLM decides what to say, text-to-speech turns that into audio, and the telephony layer carries it back out. Every link adds time, and the total is what the caller feels.

It's worth being precise about what you're measuring. The number that matters is time-to-first-audio: from the moment the caller actually stops speaking to the moment the first sound of the reply reaches their ear. Not time to a complete response. Not time from when your server got a transcript. The caller experiences the gap, so measure the gap.

Broken down, here's where it accumulates.

Endpointing (deciding the caller is done)

This is the most underrated source of latency, and the one people almost never instrument. Before your STT can hand you a final transcript, something has to decide the caller has finished their turn. Usually that's a voice-activity threshold: some number of milliseconds of silence.

Set that window too long and you have added dead time to every single turn before any inference has even started. Set it too short and the agent interrupts people mid-sentence, which is worse than being slow. This single setting often accounts for more perceived latency than the model does.

Speech-to-text

If you're waiting for a complete utterance to be uploaded and transcribed as a batch, you're paying for the full duration of the caller's speech plus transcription time. Streaming STT eliminates most of this: partial transcripts arrive while the caller is still talking, so when they stop, the text is essentially already there.

LLM inference

What matters here is time-to-first-token, not total generation time — because you're not going to wait for the full response anyway (see below). Time-to-first-token is driven by model size, prompt length, whether there's a queue, and whether you're doing tool calls or retrieval before generating.

That last one is a trap. A retrieval step or a function call inserted before the first token is a full extra round trip on the critical path, and it's invisible in your model-latency dashboards.

Text-to-speech

Same principle: time-to-first-audio-chunk, not time to synthesize the whole utterance. Higher-quality, more expressive voices generally take longer to start producing audio than simpler ones. This is a real, direct quality-versus-speed lever.

Network and telephony

Every hop between the caller, the carrier, your media server, your STT provider, your model provider, and your TTS provider is a round trip. Individually these are small. Chained across five services in three regions, they are not.

The practical fix is unglamorous: colocate. Keep the media path and the inference path in the same region wherever possible, and avoid architectures where audio crosses a continent to reach a model and crosses back.

A worked latency budget

It helps to think in terms of a budget rather than a single number, because a budget tells you which line item to attack.

Say your target is one second of time-to-first-audio. That second has to cover, roughly in order:

• Endpoint detection. However long your silence window is, that time is spent before anything else starts. If it's 700ms, you have 300ms left for everything else, which is not a budget — it's a rounding error. This is why endpointing tuning comes first.
• Final transcript delivery. With streaming STT this is small, because the words were transcribed as they were spoken. With batch STT it scales with utterance length, which means your latency gets worse the more the caller says. That's a bad property.
• Any pre-generation work. Retrieval, tool calls, database lookups, authorization checks. Each one is a full round trip on the critical path.
• Time-to-first-token from the model. Driven by model size, prompt length, and queueing.
• Time-to-first-audio from TTS, given the first clause of text.
• Network transit back through your media path to the caller.

Two observations fall out of writing it down this way. First, three of those six line items have nothing to do with the LLM, which is where most teams spend their optimization effort. Second, several of them are constant per turn regardless of how hard the question is — meaning they set a floor you cannot prompt your way under.

The techniques that actually work

Stream everything, wait for nothing

This is the single biggest structural win, and it's worth stating as a rule: no stage in the pipeline should wait for the previous stage to finish.

Concretely, that means three things.

Streaming STT, so partial transcripts accumulate during the caller's speech rather than after it.

Streaming LLM output, so tokens are emitted as they're generated.

And most importantly, piping LLM tokens into TTS as they arrive rather than waiting for the complete response. As soon as you have a full clause or sentence, start synthesizing it. The caller hears the beginning of the answer while the model is still writing the end of it.

That last one changes the shape of the latency budget entirely. Without it, your time-to-first-audio includes full response generation. With it, it includes only the first clause. On a long answer, the difference is seconds.

The complication is sentence boundaries. Naively chunking on every token produces choppy prosody; chunking on periods works but stalls on long clauses. Chunk on clause boundaries and let the TTS handle the joins.

Tune endpointing deliberately

Treat silence-detection thresholds as a tunable product parameter, not a library default. There's no universally correct value — a support line where callers give short factual answers wants a tighter window than one where people tell rambling stories.

Two things help beyond raw threshold tuning: speculatively starting inference on a partial transcript when the caller pauses briefly, and cancelling if they resume; and using a filler acknowledgment (a brief "mm-hm" or "let me check that") to cover a known-slow turn like a database lookup. The second is a perception fix rather than a real one, but perception is the thing you're optimizing.

Pick the model for the job, not the hardest job

Most turns in a real conversation are not hard. Greetings, confirmations, "yes that's right," collecting a phone number — none of these need your most capable model. Routing simple turns to a faster, smaller model and reserving the larger one for genuinely complex reasoning cuts average latency substantially without touching the ceiling on quality.

This works best when the routing decision itself is cheap. If deciding which model to use costs a model call, you've added a round trip to save one.

Keep the agent's turns short

A voice agent that answers in three sentences instead of ten is faster in two ways: less to generate, and less to synthesize. It's also just better voice design — nobody wants a paragraph read at them over the phone. Prompt for brevity explicitly and enforce it, because models drift toward long answers.

Move work off the critical path

Anything that doesn't have to happen before the first word shouldn't. Logging, CRM writes, analytics, summarization, notification sends — push all of it to after the turn, or to a background job. Audit and compliance logging in particular is often written synchronously by default, and it does not need to be.

Handle barge-in as a latency feature

Barge-in — letting the caller interrupt the agent mid-sentence — is usually filed under conversational polish. It is also a latency feature, and an underrated one.

If the agent starts talking quickly and the caller can cut it off the instant it goes in the wrong direction, a slightly wrong fast answer costs almost nothing. If the caller has to sit through forty seconds of a wrong answer with no way to interject, every mistake is expensive and the agent has to be more careful, which makes it slower. Reliable barge-in buys you permission to be fast.

Implementing it properly means being able to stop TTS playback mid-chunk, cancel the in-flight generation, discard the partial turn, and start listening again — all without leaving orphaned audio in the buffer. This is one of the genuinely fiddly parts of a streaming pipeline, and it is worth the effort.

Measure per stage, in production

You cannot fix what you haven't attributed. Instrument each boundary — audio in, endpoint detected, transcript final, first token, first audio chunk, audio out — and look at the p95, not the mean. Voice latency problems live in the tail. The average turn being fast is no comfort to the caller who hit the slow one, and slow turns cluster around exactly the moments that matter most: the complex questions.

The honest tradeoff

Latency is not free to reduce. Every technique above spends something.

Faster models are generally less capable. A smaller model routed onto simple turns will occasionally get a turn that turned out not to be simple, and it will handle it worse than the large model would have.

Faster TTS voices generally sound less natural. Expressiveness costs synthesis time. You are trading one dimension of perceived quality for another, and which one wins depends on your callers.

Shorter agent turns carry less information. Sometimes the ten-sentence answer was the right answer.

Tighter endpointing interrupts people. Push it too far and you've traded a latency problem for a rudeness problem, which is harder to detect in metrics and worse in practice.

Streaming architectures are harder to build and debug. Cancellation, barge-in mid-synthesis, partial state, and out-of-order events all become real concerns. A batch pipeline is much simpler to reason about; it's just slower.

The reason to be explicit about this is that "reduce latency" gets treated as a pure win, and teams over-optimize into an agent that responds instantly with mediocre answers in a robotic voice. The goal is a conversation that feels natural, and latency is one input to that, not the whole of it.

Where to start

If you're staring at a slow agent and don't know where to begin, do these in order:

1. Instrument the five pipeline boundaries and get a p95 for each. Most teams discover the bottleneck isn't where they assumed.
2. Check your endpointing threshold. This is frequently the largest single number and the cheapest to change.
3. Pipe LLM tokens into TTS incrementally if you aren't already. Biggest structural win available.
4. Move every non-essential operation off the critical path.
5. Only then start swapping models or voices — that's the step with the real quality cost, so it should come last, not first.

Voice latency work is mostly plumbing, not machine learning. The model is rarely the whole problem, and it's almost never the first thing to change.