What is barge-in in a voice AI agent?

Q: What does conversation.item.truncate do in the OpenAI Realtime API?

The conversation.item.truncate event lets the agent tell the API how many milliseconds of its audio were actually played. The API then cuts the rest of that reply, including the text of the part that was not played. This keeps the conversation history matched to what the user really heard.

Barge-in in a voice AI agent is the ability of the user to interrupt the agent while it is speaking, and the ability of the agent to stop talking right away and listen.

In this blog, we will learn about barge-in in a voice AI agent. We will also see where the word comes from, why it is hard for a machine to hear the user while it is talking, how the agent tells a real interruption apart from noise and "uh-huh", what the agent must do the moment the user barges in, how it remembers what the user actually heard, and why all of this must happen fast.

We will cover the following:

What is barge-in?
Where the word barge-in comes from
How a voice AI agent speaks and listens
Problem 1: The agent hears itself
Problem 2: Not every sound is an interruption
How the agent detects barge-in
What the agent does after barge-in
Remembering what the user actually heard
Why barge-in must be fast
A complete walkthrough

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is barge-in?

Barge-in is when the user starts speaking while the voice AI agent is still speaking, and the agent stops its own speech so that it can listen to the user.

A voice AI agent is a system that talks with us using voice. We speak, it listens, it thinks, and it speaks back. Customer support phone bots and voice assistants are voice AI agents.

The best way to learn this is by taking an example.

Let's say we call a food delivery support line and the agent starts speaking:

Agent: Thank you for calling. I can help you with new orders,
       order tracking, refunds, address changes, and...
User:  Refund.
Agent: Sure. Let's start your refund. What is your order number?

Here, the user already knew what they wanted and said "Refund" in the middle of the list. The agent stopped, listened, and moved to the refund.

This is barge-in.

Humans do this all the time. When a friend is giving us directions and we already know the place, we say, "Yes, I know it," and the friend stops. A good voice AI agent must behave the same way.

Barge-in is one part of a bigger job called turn-taking. We have a detailed blog on turn-taking in a voice AI agent that explains how the agent decides whose turn it is to speak.

Where the word barge-in comes from

To "barge in" means to enter suddenly and interrupt. The user "barges in" on the agent's turn.

The word comes from old telephone systems. We must have faced those phone menus that say, "Press 1 for billing, press 2 for support." These are called IVR systems. IVR stands for Interactive Voice Response. In early IVR systems, we had to wait for the full menu to finish before we could press a key. Later, IVR systems allowed the caller to press a key or speak in the middle of the message, and the system would stop playing it. This feature was called barge-in.

Modern voice AI agents use the same word for the same idea. The difference is that the agent now speaks sentences generated by an AI model, not a fixed recorded menu.

How a voice AI agent speaks and listens

Before jumping into how barge-in works, we must know how a voice AI agent is built.

Most voice AI agents follow a pipeline like below:

+-------------------+
|  User speaks      |
+-------------------+
          ↓
+-------------------+
|  STT              |
|  (speech to text) |
+-------------------+
          ↓
+-------------------+
|  LLM              |
|  (writes reply)   |
+-------------------+
          ↓
+-------------------+
|  TTS              |
|  (text to speech) |
+-------------------+
          ↓
+-------------------+
|  Speaker plays    |
|  the reply        |
+-------------------+

Here, we have:

STT (Speech to Text): It turns the user's voice into text.
LLM (Large Language Model): It reads the text and writes the reply.
TTS (Text to Speech): It turns the reply text into voice.

There is one more part that matters for barge-in. It is called VAD, which means Voice Activity Detection. VAD listens to the microphone audio in small pieces, for example every 20 or 30 milliseconds, and says only one thing: "Is a human speaking right now, yes or no?"

In a simple agent, the microphone is switched off, or ignored, while the agent is speaking. The agent talks, then it listens, then it talks again. This is like a walkie-talkie. Only one side can talk at a time.

With such an agent, barge-in is not possible. So, to support barge-in, the agent must keep listening even while it is speaking. Means, the microphone, the VAD, and the STT keep running during the agent's speech.

But the following issues arise.

Problem 1: The agent hears itself

Let's say we are using a voice agent on a phone with the speaker on. The microphone is a few centimeters away from the speaker. So, the microphone picks up the agent's own voice.

Now, the VAD hears a human voice. It is the agent's own voice, but the VAD does not know that. It says, "Yes, a human is speaking." The agent thinks the user has barged in and stops itself.

So, the agent keeps interrupting itself. This is called self-interruption.

The solution to this problem is Acoustic Echo Cancellation, usually called AEC.

The agent knows exactly what audio it is sending to the speaker, because it generated that audio. So, AEC takes that known audio, predicts how it will sound when it comes back into the microphone, and subtracts it from the microphone signal.

+--------------------------+
|  Microphone audio        |
|  (user + agent's echo)   |
+--------------------------+
             ↓
+--------------------------+
|  AEC subtracts the       |
|  agent's own audio       |
+--------------------------+
             ↓
+--------------------------+
|  Clean audio             |
|  (mostly the user)       |
+--------------------------+
             ↓
+--------------------------+
|  VAD                     |
+--------------------------+

Here, we can see that the agent's own voice is removed before the audio reaches the VAD. So, the VAD now hears mostly the user.

Phones, browsers, and operating systems often provide AEC. But it is never perfect. A loud speaker, a big empty room, or a low-quality phone can let some echo leak through. Hence, a good agent does not trust a single short burst of sound as a barge-in. This takes us to the next problem.

Problem 2: Not every sound is an interruption

Even with clean audio, not every sound that reaches the microphone means "stop talking."

Let's see a few cases:

Backchannels: While listening, people say small things like "uh-huh", "okay", "right", or "hmm". These mean "I am listening, please continue." They do not mean "stop."
Background noise: A door closes, a dog barks, a TV is on in the room, or a car horn sounds.
Coughs and breaths: The user coughs or clears the throat.
Other people: Someone else in the room talks to the user.

If the agent stops for every one of these, it keeps cutting its own sentences. When the agent stops for a sound that was not a real interruption, it is called a false barge-in.

So, the agent has two opposite mistakes to avoid:

Mistake	What happens	How it feels to the user
False barge-in	The agent stops for noise, a cough, or "uh-huh"	The agent keeps cutting itself
Missed barge-in	The agent keeps talking when the user wants to speak	The agent does not listen

How the agent detects barge-in

Let's see the approaches one by one, from the simplest to the most complete.

Approach 1: Stop on any voice

The agent stops the moment the VAD says "a human is speaking", even if the sound lasts only a fraction of a second.

Advantage: It is very fast. The user is heard right away.

Disadvantage: It stops for coughs, "uh-huh", and leaked echo. False barge-in is very common.

The issue with this approach is that it reacts to every tiny sound. Let's see how the next approach solve this issue.

Approach 2: Wait for enough speech

The agent stops only when the user has been speaking for a minimum amount of time, for example half a second, or has said a minimum number of words, for example two or three words. The word count comes from the STT, which keeps turning the user's speech into text even while the agent is talking.

So, a short cough or a quick "hmm" does not cross the limit. A real sentence like "Wait, I want to change my address" crosses it quickly.

Advantage: Far fewer false barge-ins.

Disadvantage: It adds a small delay before the agent stops. And a long backchannel like "okay, okay, yes" can still cross the limit.

The issue with this approach is that it only measures how much the user said, not what the user meant. Let's see how the next approach solve this issue.

Approach 3: Understand the meaning

The agent looks at the words the user said, and decides whether they are an interruption or a backchannel.

"uh-huh", "okay", "right", "yes, go on" -> backchannel, keep talking
"wait", "stop", "no, that is wrong", "actually I want a refund" -> interruption, stop and listen

This can be done with a simple list of backchannel words, or with a small model that is trained to label the user's speech as "interrupt" or "continue". Some newer systems use a model that looks at both the audio and the words together.

Advantage: It handles backchannels correctly.

Disadvantage: It needs the words first, so it is a little slower than the VAD alone.

In practice, agents combine these approaches. The VAD gives a fast first signal, and the agent gets ready to stop. Then the minimum speech check and the meaning check confirm whether it is a real barge-in. The meaning check also lets a single word like "stop" through, even though it is shorter than the minimum.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

LLM
RAG
MCP
Agent
Fine-tuning
Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

What the agent does after barge-in

Now, let's say the agent has confirmed that the user barged in. The agent must now stop everything that is producing its old reply.

The problem is that the old reply is not in one place. It is spread across the whole pipeline:

The LLM can still be generating the rest of the reply.
The TTS can still be turning text into audio.
Audio that is already generated can be waiting in a buffer, which is a small waiting area for audio that is ready to play.
The speaker can be in the middle of playing a piece of audio.

If the agent only stops the LLM, the audio that is already buffered will still play. The user will hear the agent continue talking after saying "stop." So, the agent must stop all of them, starting with the speaker, then the buffer, then the TTS, and then the LLM. After that, it listens to the user's full sentence and starts a new reply.

We can write the logic as below:

def on_user_started_speaking(agent):
    if not agent.is_speaking:
        return  # normal turn, nothing to interrupt

    if not is_real_interruption(agent.user_audio, agent.user_text):
        return  # cough, noise, or "uh-huh", keep talking

    agent.speaker.stop()        # stop the sound right away
    agent.audio_buffer.clear()  # drop audio waiting to play
    agent.tts.cancel()          # stop making new audio
    agent.llm.cancel()          # stop writing the old reply
    agent.mark_heard_part()     # save what the user actually heard
    agent.is_speaking = False
    agent.start_listening()

Here, we have:

agent.is_speaking tells us whether the agent is talking right now. If it is not, this is a normal user turn.
is_real_interruption is the check from the previous section. It returns False for noise and backchannels.
We stop the speaker first, so that the user stops hearing the agent as fast as possible.
agent.mark_heard_part() saves how much of the reply the user actually heard. We will learn about this in the next section.
Finally, agent.start_listening() lets the agent hear the user's full sentence.

Note: This code is just for the sake of understanding. A real agent runs these parts at the same time in separate tasks, and it uses its framework's own cancel functions.

Remembering what the user actually heard

Now, the next big question is: after barge-in, what does the agent remember about its own reply?

Let's say the LLM wrote this full reply:

Your order will arrive at 7 PM. The delivery partner is Ravi.
You can track the order in the app. Your total is 450 rupees.

The LLM usually generates text much faster than the speaker can play it as audio. So, the full text is often ready long before the audio finishes playing. But the user barged in after hearing only this:

Your order will arrive at 7 PM. The delivery partner is...

Now, if the agent saves the full reply in the conversation history, the LLM believes the user knows the delivery partner's name and the total amount. Later, the user asks, "How much is it?" and the agent says, "As I told you, it is 450 rupees." But the user never heard that.

So, the agent must cut its saved reply to only the part that was actually played.

+---------------------------------------+
|  Full reply written by the LLM        |
|  (4 sentences)                        |
+---------------------------------------+
                    ↓
+---------------------------------------+
|  Part played before barge-in          |
|  (1.5 sentences)                      |
+---------------------------------------+
                    ↓
+---------------------------------------+
|  Saved in conversation history        |
|  "Your order will arrive at 7 PM.     |
|   The delivery partner is..."         |
+---------------------------------------+

How does the agent know which part was played? The answer is timing. The agent knows when the audio started playing and when it stopped. Some TTS systems also return word-level timestamps, which tell the time at which each word is spoken. So, the agent can match the stop time to the last word that was spoken.

For example, the OpenAI Realtime API provides the conversation.item.truncate event. The agent sends how many milliseconds of its audio were played, and the API cuts the rest of that reply, including the text of the part that was not played.

Some agents also add a small note to the history, like "the user interrupted here", so that the LLM understands why its sentence is incomplete.

To learn Memory in Agents, Agent Architecture, and how to Design a Real-Time Voice AI Agent, check out our AI and Machine Learning Program at Outcome School.

Why barge-in must be fast

When a person keeps talking for two more seconds after we say "wait", we feel ignored. The same is true for a voice AI agent. The time between the moment the user starts speaking and the moment the agent goes quiet must be short. But many small delays add up in this chain:

The VAD needs a few small audio pieces to be sure there is speech.
The confirmation check needs some time, especially when it waits for words.
The audio already sent to the phone network or the speaker can still play for a moment.

So, a good agent keeps every part of this chain short. It uses small audio buffers, so that very less audio is waiting to play. It can also lower the volume as soon as the VAD fires, fully stop once the barge-in is confirmed, and go back to normal volume if it was a false alarm.

Here, we can see a trade-off. Stopping faster means more false barge-ins. Waiting longer before stopping means fewer false barge-ins, but a slower response. We must tune this based on our use case. A phone line with a lot of background noise needs to wait longer. A quiet in-app assistant can stop faster.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

A complete walkthrough

Now, let's put everything together with one full example.

The user calls a food delivery agent and asks, "Where is my order?"

Step 1: The agent's LLM writes the reply, and the TTS starts speaking: "Your order will arrive at 7 PM. The delivery partner is Ravi. You can track the order in the app..."

Step 2: The microphone, the VAD, and the STT keep running. AEC removes the agent's own voice from the microphone audio.

Step 3: The user says "uh-huh" after the first sentence. The VAD fires, but the check finds that it is a short backchannel. The agent keeps talking.

Step 4: The user then says, "Wait, can I change the address?" The VAD fires, the speech is long enough, and the words "wait" and "change the address" mean an interruption. Barge-in is confirmed.

Step 5: The agent stops the speaker, clears the audio buffer, cancels the TTS, and cancels the LLM.

Step 6: The agent saves only the part the user heard: "Your order will arrive at 7 PM. The delivery partner is Ravi. You can track..."

Step 7: The STT turns the full user sentence into text. The LLM gets the updated history and replies: "Sure. What is the new address?"

Now we must have understood what barge-in in a voice AI agent is, how the agent detects it, and what it does when the user interrupts.

Frequently Asked Questions

Why can't a simple voice AI agent support barge-in?

A simple voice AI agent switches off or ignores the microphone while it is speaking, like a walkie-talkie where only one side can talk at a time. So, it cannot hear the user during its own speech. To support barge-in, the microphone, the VAD, and the STT must keep running even while the agent is speaking.

What is a false barge-in?

A false barge-in is when the agent stops talking for a sound that was not a real interruption. This can be a backchannel like "uh-huh", a cough, background noise, someone else in the room, or the agent's own echo. Too many false barge-ins make the agent keep cutting its own sentences.

Should a voice AI agent stop talking when the user says "uh-huh"?

No. Words like "uh-huh", "okay", and "right" are backchannels. They mean "I am listening, please continue." The agent checks the words the user said, using a list of backchannel words or a small model, and keeps talking for backchannels while stopping for real interruptions like "wait" or "stop".

Why does the agent stop the speaker before cancelling the LLM?

The agent stops the speaker first so that the user stops hearing it as fast as possible. If the agent only stopped the LLM, the audio already waiting in the buffer would still play, and the user would hear the agent keep talking after saying "stop". So, the agent stops the speaker, then the buffer, then the TTS, and then the LLM.

What does conversation.item.truncate do in the OpenAI Realtime API?

The conversation.item.truncate event lets the agent tell the API how many milliseconds of its audio were actually played. The API then cuts the rest of that reply, including the text of the part that was not played. This keeps the conversation history matched to what the user really heard.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.