Voice agents feel fast only when the whole turn, from user speech to audible reply, stays under about 800 ms. Teams often look at time‑to‑first‑token (TTFT) as the sole speed indicator, but TTFT only marks when the language model starts generating. A text‑to‑speech engine cannot speak until it has a complete clause, so the user actually experiences time‑to‑first‑sentence (TTFS). Optimizing TTFT alone can leave the agent feeling sluggish because the sentence completion step may still be slow.
To pick a provider that truly serves voice, look at two numbers together: TTFT and tokens‑per‑second. A model with a quick first token but low throughput will still delay the first sentence. Conversely, a model with high throughput but a long TTFT wastes the latency budget before any generation begins. The sweet spot is a sub‑500 ms TTFT paired with at least 250 tokens per second, which lets a typical sentence finish in roughly 100 ms.
Latency is not just the model. Speech‑to‑text adds the delay between when the user stops talking and when the pipeline knows they are done. End‑of‑turn detection that runs inside the recognizer (e.g., models that emit an eager end‑of‑turn signal) can shave 200‑600 ms off the critical path by letting the LLM start early. Text‑to‑speech latency quotes often exclude network round‑trip and player buffering; add 150‑300 ms for those pieces when budgeting.
Reasoning effort is the biggest lever. Switching a model from minimal to high reasoning can triple TTFT, blowing past the 800 ms target. Cap reasoning explicitly in your API calls and keep it low for voice unless the task truly needs deep analysis.
Tool calls roughly double LLM latency. Limit the number of steps, batch external requests, and play a soft thinking sound so users perceive progress during unavoidable delays.
Before tuning, instrument each stage. Capture end‑to‑end latency, TTFT, and time‑to‑first‑audio per turn. Log the data externally and watch for regressions. Focus on tail latency (p95) because occasional long turns break the conversational feel more than average speed.
Finally, colocation matters more than picking the fastest model. Hosting the agent, STT, LLM, and TTS in the same region or on the same network cuts transport jitter and makes the latency budget predictable.
#AI #Product #VoiceAI #Latency #LLM #MLOps