Voice Agents Fail in the Media Plane, Not the Model

Every voice AI team I talk to benchmarks the same three numbers: time to first token, tokens per second, and end-to-end response latency in their test environment. Then they ship, and users say the agent feels slow, interrupts them, or talks over itself — even though the dashboards look great.

That's because the dashboards are measuring the model. The user is experiencing the media plane.

The part of the stack nobody instruments

A production voice agent is a real-time media system with an LLM inside it, not an LLM with a microphone attached. Between the user's mouth and the model sit capture, encoding, packetization, a network path with real jitter and loss, a jitter buffer, VAD, endpointing, ASR streaming — and the whole chain again on the way back. Each stage adds latency, and more importantly, adds variance. Humans are remarkably tolerant of consistent delay and remarkably intolerant of inconsistent delay.

Three failure modes dominate, and none of them show up in a token benchmark:

Endpointing errors. Decide too early that the user finished speaking and the agent interrupts them. Decide too late and the agent feels sluggish. Static silence thresholds fail across accents, thinking pauses, and noisy environments — and this single parameter shapes perceived intelligence more than model choice does.

Barge-in handling. When a user talks over the agent, the system must detect it, stop TTS playback, flush queued audio, and cancel in-flight generation — in under ~200ms across the full pipeline. Most stacks handle the detection and fumble the flush, so the agent keeps talking. Users read that as "it isn't listening."

Network reality. Demos run on office Wi-Fi. Users are on cellular in cars, on hotel networks, behind aggressive NATs. Packet loss degrades ASR accuracy before it degrades audibility — the transcript quietly gets worse, the model answers the wrong question, and the team blames the prompt.

Why this matters commercially

The teams winning voice AI deals right now are winning on developer ergonomics. But the deals that renew are decided in the media plane, because that's where the 95th-percentile experience lives — and enterprises buy the 95th percentile. Testing and QA tools that evaluate transcripts catch none of this: two identical transcripts can represent one delightful conversation and one broken one.

If you own a voice agent product, the highest-leverage question isn't "which model?" It's: what is our latency variance under 2% packet loss, how fast do we stop talking when the user starts, and who on the team owns endpointing? If nobody can answer, that's the gap.

I've spent 15+ years building real-time media systems — across 20M+ devices, five patents in connectivity and NAT traversal, and voice/video APIs at enterprise scale. This boundary between media and intelligence is exactly where I consult.