Why does TTS audio buffering make interruptions slow?

TTS (Text-to-Speech) audio buffering makes interruptions slow because the voice agent creates its speech faster than we can listen to it, and all that extra speech waits in buffers. When we interrupt, everything already waiting in those buffers still has to be thrown away, and whatever cannot be thrown away keeps playing.

In this blog, we will learn about why TTS audio buffering makes interruptions slow. We will also see how a TTS model creates audio faster than real time, where the extra audio waits before we hear it, how much delay a full buffer adds when we interrupt, why we cannot simply remove all the buffers, how pacing the audio keeps the buffers small, and where this works well and where it fails.

We will cover the following:

What is TTS audio buffering?
TTS creates audio faster than we can hear it
Where the extra audio waits
What happens when we interrupt
How much delay does the buffer add?
Why we cannot remove the buffers
The fix: send audio at the speed of speech
Clearing what is left
A complete example
Where it works well and where it fails

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is TTS audio buffering?

A voice agent is a system that talks with us over a phone call or in an app. A typical voice agent has three main parts:

STT (Speech-to-Text): It listens to our voice and turns it into text.
LLM (Large Language Model): It reads that text and writes a reply.
TTS (Text-to-Speech): It turns the reply text into audio.

When we start talking while the agent is still speaking, it is called an interruption, or barge-in. A good agent must go silent right away when we interrupt.

Now, a buffer is a place where data waits before it is used. Think of a water tank on the roof of a house. Water comes into the tank from the pipe, and it waits there until someone opens a tap.

TTS audio buffering means that the audio created by the TTS waits in one or more buffers before it reaches our speaker.

Once the water is in the tank, closing the pipe does not empty the tank. While the tap is open, the water already in the tank keeps flowing out until the tank is empty. In the same way, audio already in the buffer keeps playing even after the agent decides to stop.

To understand why so much audio ends up in the buffers, we must first see how fast the TTS works.

TTS creates audio faster than we can hear it

We listen to speech at a fixed speed. One second of audio takes one second to hear. This speed is called real time.

A modern TTS model, running on a good GPU, often creates audio much faster than real time. For the sake of example, let's say our TTS creates 1 second of audio in 0.2 seconds. Means, it is 5 times faster than real time.

This speed helps the agent start speaking quickly, because the first piece of audio is ready almost immediately.

But, this speed creates a side effect. The TTS keeps creating audio 5 times faster than we can listen to it. So, until the reply is complete, every second, 5 seconds of audio are created, but only 1 second is played. The other 4 seconds must wait somewhere.

The audio is created and sent in small pieces called chunks. On a phone call, a chunk is often 20 milliseconds of sound, and we will use the same size in this blog. These chunks pile up in the buffers, one after another.

Where the extra audio waits

The audio does not travel straight from the TTS to our ear. It passes through many waiting areas on the way.

The audio travels like below:

+----------------------+
|         TTS          |
|  creates the audio   |
+----------------------+
           ↓
+----------------------+
|    Server buffer     |
|   chunks to send     |
+----------------------+
           ↓
+----------------------+
|       Network        |
|  chunks on the way   |
+----------------------+
           ↓
+----------------------+
|    Device buffer     |
|   chunks to play     |
+----------------------+
           ↓
+----------------------+
|       Speaker        |
|   sound we hear      |
+----------------------+

Here, we can see the waiting areas:

Server buffer: Audio chunks that are ready on the agent's server and are waiting to be sent.
Network: Audio chunks that are already sent and are travelling to our device.
Device buffer: Audio chunks that reached our phone or browser and are waiting to be played.
Speaker: The sound that is playing right now.

On a phone call, there is one more waiting area. The telephony platform that connects the agent to the phone call also keeps its own buffer.

Here, we can notice that the agent's server fully controls only the first waiting area, the server buffer. The farther the audio travels, the harder it is for the agent to take it back.

What happens when we interrupt

Let's take an example. We call a voice agent to book a flight. The agent says, "We have flights at 6 AM, 9 AM, 1 PM, 5 PM, and 9 PM." After "9 AM", we say, "9 AM works."

The agent hears us and decides to stop. Now, the question is: when does the sound actually stop?

The answer is: only when every buffer between the agent and our ear is empty or cleared.

Suppose the agent stops the LLM and the TTS, but does nothing about the buffers. No new audio is created. But the device buffer still holds "1 PM, 5 PM, and 9 PM". So, our speaker keeps playing it. We hear the agent talk over us for 2 or 3 more seconds.

This makes a voice agent feel unnatural. We said "9 AM works", and the agent kept reading the list. Many people then talk louder or repeat themselves, and the agent has to deal with that extra speech too.

So, the slowness does not come from the LLM or the TTS being slow. It comes from the audio that was already created and already sent.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

AI Engineer
Forward Deployed Engineer
AI Agent Engineer
Harness Engineer
Voice AI Engineer
AI Architect
LLM Engineer
LLM Inference Engineer
Gen AI Engineer
MLOps Engineer
ML Engineer
Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

How much delay does the buffer add?

The best way to learn this is by taking an example with numbers.

Let's say the agent's full reply is 10 seconds long. The TTS is 5 times faster than real time. The server sends every chunk to the device the moment it is created.

Step 1: The TTS creates the full 10 seconds of audio in about 2 seconds.

Step 2: The server sends all of it right away. Playback starts almost immediately, so after about 2 seconds, we have heard 2 seconds, and the remaining 8 seconds are sitting in the device buffer.

Step 3: We interrupt at the 3-second mark. The device plays at real time, so we have heard 3 seconds of speech. The device buffer still holds 10 - 3 = 7 seconds of audio.

If the device has no way to empty its buffer, we hear the agent talk for 7 more seconds.

Now, let's compare it with a server that sends audio only a little ahead of playback. Let's say it keeps the device buffer at only 0.2 seconds of audio. Everything else stays on the server.

At the 3-second mark, the device buffer holds only 0.2 seconds of audio. The server throws away its own buffer instantly, because it fully controls it. So, the extra speech we hear is at most about 0.2 seconds.

We can write the rule like below:

Extra speech after interrupt = audio waiting where the agent cannot clear it

Here, we can notice that the delay is not about how fast the LLM or the TTS works. It is about where the extra audio is waiting.

Let me tabulate the differences between the two approaches for your better understanding.

Approach	Device buffer at 3 seconds	Extra speech after interrupt
Send everything immediately	7 seconds	Up to 7 seconds
Send only 0.2 seconds ahead	0.2 seconds	Up to 0.2 seconds

Why we cannot remove the buffers

Now, the next big question is: why not remove all the buffers? Then, there is nothing to throw away.

The answer is that the buffers protect the sound quality.

The network does not deliver chunks at a perfectly steady speed. Some chunks arrive a little late. Some arrive in a bunch. This unevenness in arrival time is called jitter.

If the device has no buffer, and the next chunk is late, the speaker has nothing to play. The sound breaks up for a moment. This is called an underrun. We hear a gap, a click, or a robotic stutter.

So, the device keeps a small buffer on purpose. It holds a little audio in reserve, so that a late chunk does not cause a gap. This is called a jitter buffer, and it usually holds somewhere between a few tens of milliseconds and a couple of hundred milliseconds of audio.

Here is the trade-off:

Big buffer: Smooth sound, but slow to stop when we interrupt.
Small buffer: Fast to stop, but the sound can break up when the network is uneven.

We cannot make the buffer zero. We can only make it as small as the network allows. This is why the goal is not "no buffers". The goal is "small buffers far from the agent, and big buffers only where the agent can clear them".

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

LLM
RAG
MCP
Agent
Fine-tuning
Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

The fix: send audio at the speed of speech

Here comes pacing into the picture.

Pacing means that the server sends audio to the device at the same speed it will be played, keeping only a small amount of audio ahead of the speaker.

The TTS can still create audio 5 times faster than real time. But the extra audio stays in the server buffer. The server releases it at real time, like a tap that lets out only a little water at a time.

Let's see how it works:

The TTS creates audio as fast as it can and puts it in the server buffer.
The server keeps track of how much audio the device has not played yet. This is called the lead.
If the lead is smaller than the target, for example 200 milliseconds, the server sends the next chunk.
If the lead is already at the target, the server waits.

At the very start, the lead is zero. So, the server sends the first 200 milliseconds of audio right away, and playback begins without any extra wait.

This way, the device always has enough audio to cover small network delays, but never several seconds of it.

The best part is that the server buffer is the easiest buffer to clear. It sits in the agent's own memory. When we interrupt, the agent empties it in a single step, and that audio is never sent.

Clearing what is left

Pacing keeps the far buffers small. But they are still not empty. So, when we interrupt, the agent must also clear what is left, in order from nearest to farthest:

Stop the TTS and the LLM: No new audio gets created.
Empty the server buffer: The agent drops every chunk that is waiting to be sent.
Tell the device to clear its buffer: In an app, the agent sends a message to the app, and the app stops its audio player and empties its buffer. On a phone call, some telephony platforms support a "clear" message that drops the audio they are holding.

One more thing to notice. Even after the agent sends the "clear" message, a few chunks of the old reply can still arrive. For example, a chunk that the sender was already sending at the moment of the stop can reach the device after the message. So, the device must ignore audio that belongs to the old reply. A common way is to give each reply a number. When the agent stops reply number 5, the device drops any chunk marked with number 5.

If we want to go deep into AI and ML System Design, including how to Design a Real-Time Voice AI Agent, check out our AI and Machine Learning Program at Outcome School.

A complete example

Let's see the code for a paced sender. It keeps only the important parts.

We can write the code as below:

import asyncio
import time

CHUNK_MS = 20         # each chunk is 20 milliseconds of audio
TARGET_LEAD_MS = 200  # keep only 200 ms ahead of the speaker

async def paced_sender(server_buffer, send_to_device, stop_event):
    start = time.monotonic()
    sent_ms = 0
    while not stop_event.is_set():
        played_ms = (time.monotonic() - start) * 1000
        lead_ms = sent_ms - played_ms
        if lead_ms >= TARGET_LEAD_MS or not server_buffer:
            await asyncio.sleep(0.005)  # wait a little and check again
            continue
        chunk = server_buffer.pop(0)
        await send_to_device(chunk)
        sent_ms += CHUNK_MS

Here, we have a loop that sends one chunk at a time:

sent_ms is how much audio we have sent to the device so far.
played_ms is how much audio the device has played so far. For the sake of understanding, we guess it from the time passed since the sender started, and we assume that playback starts at the same moment.
lead_ms is the audio waiting on the device. It is what we have sent minus what has been played.
If the lead is already 200 milliseconds or more, or the server buffer has no chunk yet, we wait for 5 milliseconds and check again. Otherwise, we send the next chunk.
The loop ends as soon as stop_event is set.

Now, let's see the code that runs when the user interrupts:

async def on_interrupt(tts_task, server_buffer, stop_event, device):
    tts_task.cancel()                       # stop creating new audio
    stop_event.set()                        # stop the paced sender
    dropped_ms = len(server_buffer) * CHUNK_MS
    server_buffer.clear()                   # drop audio not sent yet
    await device.send_clear()               # ask the device to drop its buffer
    print(f"Dropped {dropped_ms} ms of audio on the server")

Here, we can see that the agent clears the buffers from nearest to farthest. It stops the TTS, stops the sender, empties the server buffer, and asks the device to clear its own buffer.

Let's say the reply was 10 seconds long, and we interrupt at the 3-second mark. The TTS has already finished, so the server buffer holds about 10 - 3 - 0.2 = 6.8 seconds of audio. It will print the following:

Dropped 6800 ms of audio on the server

Here, we can notice that 6.8 seconds of audio were thrown away on the server, before they were ever sent. Without pacing, most of that audio would have been sitting on the device already. With pacing, the user hears at most about 0.2 seconds of the old reply, and often less, because the device also clears its buffer.

In a real system, the device reports how much it has actually played, instead of the server guessing it from the clock. But the idea stays the same.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Where it works well and where it fails

Pacing and clearing work well in these cases:

When the agent controls the full path, like its own app, it can clear every buffer and make the stop feel instant.
When the network is steady, the lead can be small, so the stop is fast.
When the TTS is much faster than real time, the server buffer always has audio ready. So, the paced sender never runs out, and the extra audio waits on the server, where it is easy to drop.

It fails, or works less well, in these cases:

On a phone call, the phone network and the telephony platform keep their own buffers. The agent cannot always clear them fully, so a small delay remains.
On a weak or uneven network, the lead must be bigger to avoid gaps in the sound. A bigger lead means a slower stop.
If the device or the platform does not support a "clear" message, the audio already sent will play out to the end.
If the agent guesses the playback position from the clock, and the playback started late or paused for a moment, the guess drifts. Then, the agent thinks the device has less audio than it really has.

Now we must have understood why TTS audio buffering makes interruptions slow, where the audio waits, and how pacing and clearing keep the agent quick to stop.

Frequently Asked Questions

Does a faster TTS model make interruptions faster?

No. A faster TTS model does not make interruptions faster on its own. The delay after an interruption is not about how fast the LLM or the TTS works. It is about where the extra audio is waiting. Audio waiting on the device keeps playing, but audio waiting on the server can be dropped by the agent in a single step.

What is a jitter buffer in a voice agent?

A jitter buffer is a small buffer that the device keeps on purpose, so that a late audio chunk does not cause a gap in the sound. The network does not deliver chunks at a perfectly steady speed. So, the jitter buffer holds a little audio in reserve, usually between a few tens of milliseconds and a couple of hundred milliseconds.

What is an underrun in audio playback?

An underrun happens when the speaker has nothing to play because the next audio chunk is late. The sound breaks up for a moment, and we hear a gap, a click, or a robotic stutter. This is why the device keeps a small buffer on purpose instead of having no buffer at all.

Why does a voice agent give each reply a number?

A voice agent gives each reply a number so that the device can ignore late audio from an old reply. Even after the agent sends the "clear" message, a chunk that was already being sent can reach the device. When the agent stops reply number 5, the device drops any chunk marked with number 5.

Can a voice agent clear all the buffered audio on a phone call?

Not always. On a phone call, the phone network and the telephony platform keep their own buffers. Some telephony platforms support a "clear" message that drops the audio they are holding, but the agent cannot always clear these buffers fully. So, a small delay remains.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.