The caller heard silence for t... Note

The caller heard silence for two seconds before the agent spoke

A voice agent had a critical bug where callers would speak over the greeting due to a two-second delay in audio playback. This delay occurred because the system waited for the entire LLM response to generate before sending it to text-to-speech. The initial investigation involved timing each stage of the audio pipeline: speech-to-text, LLM processing, text-to-speech, and network transmission. The analysis revealed that the LLM completion and TTS cold start were the primary contributors to the excessive latency.To fix this, the LLM's output was streamed to the TTS engine sentence by sentence, allowing audio generation to begin much earlier. Furthermore, the TTS connection was pre-warmed and kept open to reduce its startup time. A crucial improvement involved sending a short, pre-synthesized acknowledgement immediately after the caller finished speaking. This immediate audio feedback masked any remaining processing time, preventing the perception of dead air. These optimizations reduced the time to first audio from two seconds to approximately 150 milliseconds for the acknowledgement. The key takeaway was to treat latency as a budget across stages and to prioritize immediate audio feedback over waiting for a complete, perfect response. The strategy of streaming LLM output and providing instant acknowledgments resolved the collision issue and improved the user experience.