How does a voice agent stop speaking instantly when interrupted?

Q: What is the clear message in Twilio Media Streams?

The clear message tells the phone platform to throw away all the audio it was holding for that call. On a phone call, our code cannot reach inside the user's phone, because the audio first goes to a phone platform that holds its own buffer. So, the agent sends clear to the platform, and the waiting audio is never played.

Q: How does a voice agent know how much of its reply was played on a phone call?

It uses the mark message. The agent sends a mark message after its audio chunks, and Twilio sends that mark back when the audio before it has finished playing. Some TTS services also return timestamps for each word, so the agent can match the played time with the words and find the last word the user heard.

How does a voice agent stop speaking instantly when interrupted?

A voice agent stops speaking instantly when interrupted by throwing away every piece of audio that is waiting to be played, at every step between the agent and our ear, and by cancelling the work that would create more audio.

In this blog, we will learn about how a voice agent stops speaking instantly when interrupted. We will also see where the agent's audio waits before we hear it, why stopping only the LLM is not enough, how the agent clears every waiting area in the right order, how it ignores late audio that is still on the way, how it remembers only what we actually heard, and where it works well and where it fails.

We will cover the following:

What does stopping instantly mean?
How a voice agent speaks
Where the audio waits before we hear it
Why stopping the LLM is not enough
Step 1: Stop the sound at our ear
Step 2: Clear the server buffer
Step 3: Cancel the work that makes new audio
Step 4: Ignore late audio with a reply number
Step 5: Remember only what was heard
Keeping the waiting areas small
A complete example
Where it works well and where it fails

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What does stopping instantly mean?

When we start speaking while the voice agent is still talking, it is called barge-in. The agent first has to detect that we really want to interrupt. That part is called barge-in detection.

This blog is about what happens right after that. The agent has decided, "The user is interrupting me." Now, it must go silent.

Stopping instantly means that the agent's voice goes quiet within a fraction of a second after the agent decides to stop.

Let's say we call a voice agent to book a flight. The agent says, "We have flights at 6 AM, 9 AM, 1 PM, and 5 PM." After "9 AM", we say, "9 AM works."

If the agent goes silent right away, the call feels natural. If it keeps talking for two more seconds, "1 PM, and 5 PM", we feel ignored. We start talking louder, or we repeat ourselves.

So, the question is: why would an agent keep talking after it has already decided to stop? To answer this, we must first see how a voice agent speaks.

How a voice agent speaks

A typical voice agent has three main parts:

STT (Speech-to-Text): It listens to our voice and turns it into text.
LLM (Large Language Model): It reads that text and writes a reply.
TTS (Text-to-Speech): It turns the reply text into audio.

The agent keeps listening to our microphone even while it is speaking. This is how the agent notices that we have started to talk.

The LLM does not write the full reply at once. It writes it token by token. The TTS does not wait for the full reply either. It starts making audio as soon as it gets the first few words. This is called streaming, and it helps the agent start speaking quickly.

The audio is made in small pieces called chunks. A chunk is often as small as 20 milliseconds of sound. These chunks travel from the agent's server, over the internet or the phone network, to our phone or browser, and finally to our speaker.

Now, the TTS often makes audio several times faster than we can listen to it. Means, the audio is created faster than it is played. So, the extra audio has to wait somewhere.

Where the audio waits before we hear it

A place where data waits before it is used is called a buffer. A voice agent has buffers at many steps.

The audio travels like below:

+--------------------+
|        LLM         |
|  reply text queue  |
+--------------------+
          ↓
+--------------------+
|        TTS         |
|  audio being made  |
+--------------------+
          ↓
+--------------------+
|   Server buffer    |
|  chunks to send    |
+--------------------+
          ↓
+--------------------+
|      Network       |
|  chunks on the way |
+--------------------+
          ↓
+--------------------+
|   Device buffer    |
|  chunks to play    |
+--------------------+
          ↓
+--------------------+
|      Speaker       |
|   sound we hear    |
+--------------------+

Here, we can see that the audio passes through many waiting areas:

LLM text queue: Words the LLM has written, but the TTS has not turned into audio yet.
TTS: Audio that the TTS is making right now.
Server buffer: Audio chunks ready on the server, waiting to be sent.
Network: Audio chunks already sent, travelling to our device.
Device buffer: Audio chunks that reached our phone or browser and are waiting to be played. The phone network and the browser keep a small buffer on purpose, so that the sound does not break up when some chunks arrive late. This is often called a jitter buffer. On a phone call, a similar buffer also sits on the phone platform that connects the agent to the call.
Speaker: The sound that is playing right now.

Each of these holds some audio. Together, they can hold several seconds of speech that the agent has already "said" but we have not heard yet.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

AI Engineer
Forward Deployed Engineer
AI Agent Engineer
Harness Engineer
Voice AI Engineer
AI Architect
LLM Engineer
LLM Inference Engineer
Gen AI Engineer
MLOps Engineer
ML Engineer
Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Why stopping the LLM is not enough

Most of the time, we do mistakes while building the stop logic for the first time. A common mistake is to stop only the LLM.

Let's see what happens. The agent detects barge-in and stops the LLM. No new words are written. But the TTS is still turning the last sentence into audio. The server buffer still has 2 seconds of audio. The device buffer still has half a second of audio.

So, the user hears the agent continue for 2 or 3 more seconds after saying "stop".

Another common mistake is to stop only on the server. The server stops sending, but the device still plays everything in its own buffer.

To stop instantly, the agent must empty every waiting area, starting from the one closest to our ear.

In practice, the agent does all the steps below within a few milliseconds of each other. But the order still matters, because the user notices the sound at the ear first. Now, let's go through each step.

Step 1: Stop the sound at our ear

The fastest way to make the agent go quiet is to stop the sound closest to the user. So, the first thing the agent does is stop the playback on the device and empty the device buffer.

If the device is a browser or a mobile app that we control, our code can do it directly. It stops the audio player and throws away the chunks waiting to be played.

If the user is on a phone call, our code cannot reach inside the user's phone. Here, the audio first goes to a phone platform, which holds its own buffer and then plays the audio into the call.

So, the agent sends a special message to the phone platform that says "clear". For example, Twilio Media Streams supports a clear message. When the platform gets it, it throws away all the audio it was holding for that call.

There is one more detail. If we cut a sound wave suddenly in the middle, the speaker makes a tiny click. So, the agent can fade the volume down to zero over a few milliseconds instead of cutting it. A few milliseconds is too short for us to notice as a delay, but long enough to remove the click.

Step 2: Clear the server buffer

Now, the sound at our ear has stopped. But the server buffer still has chunks that are ready to be sent. If we do nothing, the server will keep sending them, and the device will start playing again.

Let's say the server buffer has 2 seconds of audio for the words "1 PM, and 5 PM". So, the agent empties the server buffer as well. Every chunk that was waiting to be sent is thrown away, and those words never leave the server.

Many voice agent frameworks do this with a special interruption message that flows through the whole pipeline. Every part of the pipeline gets this message, drops whatever it was holding, and passes the message to the next part. This way, no part is forgotten.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

LLM
RAG
MCP
Agent
Fine-tuning
Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Step 3: Cancel the work that makes new audio

The waiting areas are empty now. But the LLM and the TTS are still working. If they keep going, they will fill the buffers again.

So, the agent cancels both of them:

It cancels the TTS request, so that no new audio is made.
It cancels the LLM request, so that no new words are written.

This has one more benefit. Many LLM and TTS services charge us for every token or every character they process. If the agent cancels them early, we do not pay for a reply that nobody will hear.

We have a detailed blog on how sentence chunking feeds LLM output into TTS faster that explains how the reply text moves from the LLM to the TTS step by step.

Step 4: Ignore late audio with a reply number

Now, there is one more case to handle. The agent cleared everything and cancelled the TTS. But a few chunks were already in the network when the agent cleared the buffers. They arrive at the device a moment later.

If the device plays them, the user hears a small piece of the old reply after silence.

So, here comes the reply number to the rescue. The agent gives every reply a number. Let's say the old reply was reply number 7. Every chunk of that reply carries the number 7. When the user interrupts, the agent moves to reply number 8 and tells the device about it together with the stop message.

Now, the device follows one rule: play a chunk only if its reply number matches the current reply. Any late chunk with number 7 is thrown away.

We can write the code on the device as below:

current_reply = 8

def on_audio_chunk(chunk):
    if chunk.reply_number != current_reply:
        return  # old reply, throw it away
    player.play(chunk.audio)

Here, we can see that the device checks the reply number of every chunk before playing it. A late chunk from reply 7 does not match the current reply 8, so it is dropped. This way, old audio is never played after the stop.

Step 5: Remember only what was heard

The agent is silent now. But there is one more thing to fix, and this one is about memory.

The LLM wrote the full sentence: "We have flights at 6 AM, 9 AM, 1 PM, and 5 PM." But the user only heard "We have flights at 6 AM, 9 AM". If the agent saves the full sentence in the conversation history, the LLM will believe it already told the user about 1 PM and 5 PM.

So, the agent must cut the saved reply to only the part that was actually played. To do this, it needs to know how much audio was played before the stop. In an app, the device keeps count of the milliseconds of audio it played. On a phone call, the agent can send a mark message after its chunks, and Twilio sends that mark back when the audio before it has finished playing. Some TTS services also return timestamps that tell when each word starts in the audio. By matching the played time with the word timestamps, the agent finds the last word the user heard.

Then, the agent saves only that part in the history, like below:

Agent: We have flights at 6 AM, 9 AM
User: 9 AM works.

Now, the LLM knows exactly what the user heard, and its next reply makes sense. Some realtime APIs give us a direct way to do this. For example, the OpenAI Realtime API has a conversation.item.truncate event, where we send how many milliseconds of the reply were played, and it removes the rest.

To learn Memory in Agents, Agent Architecture, and how to Design a Real-Time Voice AI Agent, check out our AI and Machine Learning Program at Outcome School.

Keeping the waiting areas small

Now, there is a better approach too: do not let the waiting areas grow big in the first place.

The server does not need to send all the audio to the device the moment it is ready. Instead, it can send the audio at the same speed at which it is played, with only a small amount kept ahead. This is called pacing. It matters even more on phone calls, because some phone platforms accept many minutes of audio at once and queue all of it.

Let's say the server keeps only about 200 milliseconds of audio ahead of the speaker. Then, even in the worst case, the device has very less audio to throw away. The rest stays on the server, where the agent can delete it right away.

But, if the waiting area is too small, a slow network makes the sound break up, because the next chunk does not arrive in time. So, there is a balance:

Big buffer: Smooth sound on a slow network, but a slow stop.
Small buffer: Fast stop, but the sound can break up on a slow network.

So, a good agent keeps a small buffer on the device and keeps the rest on the server.

A complete example

Let's put everything together with our flight booking call.

Step 1: The agent is saying, "We have flights at 6 AM, 9 AM, 1 PM, and 5 PM." This is reply number 7.

Step 2: Right after "9 AM", the user says, "9 AM works." The agent detects a real interruption.

Step 3: In an app, the agent fades out the sound in a few milliseconds and empties the device buffer. On a phone call, it sends the clear message instead.

Step 4: The agent empties the server buffer.

Step 5: The agent cancels the TTS request and the LLM request.

Step 6: The agent moves to reply number 8 and tells the device. A late chunk from reply 7 arrives at the device and is thrown away.

Step 7: The device reports that it played 2,400 milliseconds of audio. The agent matches this with the word timestamps and saves "We have flights at 6 AM, 9 AM" in the history.

Step 8: The agent listens to the user's full sentence and sends it to the LLM. Then, the agent replies: "Great, I will book the 9 AM flight for you."

We can write the stop logic of the agent as below:

def on_interruption():
    device.fade_out_and_clear()   # stop the sound at the ear
    server_buffer.clear()         # drop chunks waiting to be sent
    tts.cancel()                  # no new audio
    llm.cancel()                  # no new words
    agent.reply_number += 1       # late chunks will be ignored
    heard = text_played_so_far()  # only what the user heard
    history.save_agent_reply(heard)

Here, we can see that the lines follow Step 3 to Step 7 of the example, in the same order, starting from the one closest to the ear.

The user hears the agent stop right away, and the next reply makes sense. This is how a voice agent stops speaking instantly when interrupted.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Where it works well and where it fails

It works well when:

The agent controls the device, like a browser or a mobile app, so it can empty the device buffer directly.
The server paces the audio and keeps the device buffer small.
The TTS returns word timestamps, so the agent can save exactly what was heard.

It fails or gets harder when:

The phone network or the user's phone adds its own small buffer that no message from the agent can clear, so the user hears a tiny bit of extra audio.
The network is slow and the device buffer must be large to keep the sound smooth.
The TTS does not return word timestamps, so the agent can only estimate where the user stopped listening, for example by assuming that every word takes the same time.
One part of the pipeline does not handle the interruption message, so old audio leaks through from that part.
The barge-in detection fires on a cough or a background noise, so the agent stops for nothing and its reply is thrown away. This is why some agents first lower the volume and stop fully only after the interruption is confirmed.

Now we must have understood how a voice agent stops speaking instantly when interrupted, where the audio waits, how the agent clears it in the right order, and how it remembers only what we actually heard.

Frequently Asked Questions

What is the clear message in Twilio Media Streams?

The clear message tells the phone platform to throw away all the audio it was holding for that call. On a phone call, our code cannot reach inside the user's phone, because the audio first goes to a phone platform that holds its own buffer. So, the agent sends clear to the platform, and the waiting audio is never played.

Why does a voice agent fade out its voice instead of cutting it?

A voice agent fades out its voice to avoid a click. If we cut a sound wave suddenly in the middle, the speaker makes a tiny click. So, the agent fades the volume down to zero over a few milliseconds. This is too short for us to notice as a delay, but long enough to remove the click.

How does a voice agent know how much of its reply was played on a phone call?

It uses the mark message. The agent sends a mark message after its audio chunks, and Twilio sends that mark back when the audio before it has finished playing. Some TTS services also return timestamps for each word, so the agent can match the played time with the words and find the last word the user heard.

Does cancelling the LLM and the TTS early save money?

Yes. Many LLM and TTS services charge us for every token or every character they process. When the agent cancels them as soon as the user interrupts, they stop processing the rest of the reply. So, we do not pay for a reply that nobody will hear.

What is a jitter buffer in a voice agent?

A jitter buffer is a small buffer that the phone network and the browser keep on purpose, so that the sound does not break up when some audio chunks arrive late. It holds a little audio before playing it. But this also means that some of the agent's audio is still waiting there when the user interrupts.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.