OpenAIが大規模に低遅延音声AIを実現する仕組み How OpenAI delivers low-latency voice AI at scale

OpenAI has shared a behind-the-scenes look at the engineering required to deliver its voice-based AI experiences—such as ChatGPT's voice mode and the Realtime API—at low latency for a global user base. In conversational AI, response delay is one of the most decisive factors shaping user experience, and minimizing it has become a key technical frontier over the past year.

In natural human conversation, the gap between turns averages around 200 milliseconds. Anything significantly longer feels awkward or robotic. To approach this threshold, OpenAI appears to stream audio input incrementally rather than waiting for an utterance to complete, parallelizing transcription, inference, and speech synthesis stages of the pipeline. Traditional cascaded systems that wait for end-of-speech detection before kicking off downstream processing typically incur multi-second delays. End-to-end speech models in the GPT-4o family reduce this by collapsing intermediate steps, so audio can be ingested and produced directly without round-trips through separate ASR and TTS components.

Delivering this experience at scale introduces a different set of problems. Users distributed across continents need consistent low latency, which depends on geographic placement of inference capacity close to the network edge, careful GPU batching strategies that preserve per-request responsiveness, and protocol choices designed for persistent low-overhead connections. WebSocket has long been used for streaming AI workloads, but the Realtime API has added WebRTC support, which is better suited to audio because it tolerates jitter and packet loss more gracefully and benefits from decades of optimization in real-time communications stacks.

The combination of model-level changes and transport-level changes matters because each layer compounds delay. Even a well-optimized model can feel sluggish if audio frames are buffered inefficiently or routed through distant data centers. Conversely, a fast network path is wasted if the model itself cannot begin generating speech until the user finishes speaking. OpenAI's architecture appears to address both ends of the chain, with continuous streaming and edge-aware routing as recurring design themes.

The broader ecosystem is moving in the same direction. Google's Gemini Live, voice-capable systems from Anthropic and xAI, and a growing crop of voice infrastructure specialists such as Deepgram and LiveKit are all competing to define how real-time voice agents are built and deployed. LiveKit has reportedly been adopted as part of the underlying transport for OpenAI's Realtime API, illustrating how core voice AI providers increasingly rely on dedicated real-time communication platforms rather than building everything in-house.

The commercial implications are significant. Customer support, voice agents for scheduling and sales, in-car assistants, and accessibility tools all depend on conversational latency that feels natural enough to sustain attention. Without sub-second responsiveness, voice AI tends to revert to a turn-taking IVR-like experience that limits its usefulness. As model providers and infrastructure vendors converge on streaming end-to-end architectures with WebRTC transport, the technical baseline for production voice AI is shifting rapidly.

Looking ahead, latency optimization may become as much a competitive differentiator as raw model quality. The next round of progress is likely to come from tighter integration between models and transport layers, smarter handling of interruptions and barge-in, and broader regional deployment of inference capacity. OpenAI's disclosure suggests that the company views these systems-level details—not just model scaling—as central to making voice AI a practical interface for everyday use.