How does a voice agent handle the user and agent talking at the same time?

A voice agent handles the user and agent talking at the same time by listening even while it speaks, removing its own voice from what it hears, deciding what kind of overlap it is, and then either continuing to speak or stopping to give the turn to the user.

In this blog, we will learn about how a voice agent handles the user and agent talking at the same time. We will also see why overlapping speech is a hard problem, the different kinds of overlap, how the agent hears the user over its own voice and decides whether to stop or continue, how it keeps the conversation history correct after it stops, how full-duplex models handle overlap in a different way, and where it works well and where it fails.

We will cover the following:

• What is overlapping speech?
• Why is overlapping speech a hard problem for a voice agent?
• The different kinds of overlap
• Step 1: Keep listening while speaking
• Step 2: Remove the agent's own voice
• Step 3: Decide what kind of overlap it is
• Step 4: Act on the decision
• Step 5: Keep the conversation history correct
• What if both start speaking at the same moment?
• How full-duplex models handle overlap
• A complete example
• Where it works well and where it fails

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is overlapping speech?

Overlapping speech is the moment when the user and the voice agent are both speaking at the same time.

Let's say we call a voice agent to check our order. The agent says, "Your order has three items: a phone case, a charger, and a pair of headphones, and it will arrive on..." Right after "a charger", we say, "Wait, I want to cancel the charger." For a short time, two voices are on the line together.

Humans do this all the time. One person starts speaking a little before the other finishes, or says "yeah" while the other is talking.

A voice agent must learn to do the same.

Why is overlapping speech a hard problem for a voice agent?

Before jumping into the solution, we must know how a simple voice agent works.

Many voice agents work like a relay race with three runners:

• Speech-to-Text (STT) turns the user's voice into text.
• LLM (Large Language Model) reads that text and writes a reply.
• Text-to-Speech (TTS) turns the reply into voice.

In the simplest version, the agent and the user take turns. It listens, then it thinks, then it speaks. While it speaks, it does not listen. This is called half-duplex, which means only one side can talk at a time.

User speaks
     ↓
STT: voice to text
     ↓
LLM: writes the reply
     ↓
TTS: text to voice
     ↓
Agent speaks (not listening)

Here, we can see the problem. If the user speaks while the agent is speaking, nobody is listening. The agent keeps talking over the user, and the user's words are lost.

So, the agent must listen while it speaks. But the moment it does that, three new problems come into the picture:

• The microphone also picks up the agent's own voice from the speaker.
• Not every sound from the user means "stop". A "hmm" or "okay" means "keep going".
• After the agent stops, it must know exactly how much of its reply the user actually heard.

Let's understand the kinds of overlap first, and then we will see how the agent solves each problem.

The different kinds of overlap

The best way to learn this is by taking an example.

Suppose the agent is reading out the steps to reset a router.

Kind 1: Interruption. The user says, "Wait, I already did that." The user wants the agent to stop and listen. This is called barge-in.

Kind 2: Backchannel. The user says, "uh-huh" or "okay" or "right". The user is only saying "I am following you, please continue". The agent must not stop.

Kind 3: Noise and echo. A cough, a dog barking, a TV in the background, or the agent's own voice coming back through the microphone. The agent must ignore all of these.

Kind 4: Both start together. The agent finishes a sentence, takes a short pause, and starts the next sentence. At the same moment, the user also starts speaking. Both voices collide at the start.

Let me tabulate the kinds of overlap and what the agent must do for your better understanding.

Now, let's see step by step how the agent handles all of these.

Step 1: Keep listening while speaking

The agent keeps the microphone open the whole time, even while it is speaking.

So, now there are two streams of audio flowing at the same time:

• The output stream: the agent's voice going to the speaker.
• The input stream: whatever the microphone hears.

The agent keeps sending the input stream to a small detector, which we will see in Step 3, even while the output stream is playing.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Step 2: Remove the agent's own voice

Now, here is the first problem. When the agent's voice plays from the user's phone speaker, the user's microphone also picks up that sound. If the agent does not handle this, it hears its own voice and thinks the user is talking. It then stops itself in the middle of every sentence.

So, here comes Acoustic Echo Cancellation (AEC) to the rescue.

The agent knows exactly what it is playing, because it produced that audio itself. So, AEC takes the audio that was sent to the speaker, predicts how that audio will sound when it comes back through the microphone, and subtracts it from what the microphone heard.

Microphone hears: user voice + agent voice (echo)
     ↓
AEC subtracts: agent voice (known copy)
     ↓
Clean audio: user voice

Here, we can see that what remains after the subtraction is mostly the user's voice. The prediction is not perfect, because the sound bounces around the room and changes a little, but it removes most of the echo.

On phone calls, the phone or the calling platform often does this echo cancellation. In browser apps, the browser often provides it. Either way, the agent must make sure it is turned on.

Step 3: Decide what kind of overlap it is

Now, the agent has clean audio from the user. The next big question is: "Is this sound a real interruption, a backchannel, or noise?"

The agent answers this question in layers.

Layer 1: Is there any human speech? A small, fast model called Voice Activity Detection (VAD) checks each short piece of audio, usually a few tens of milliseconds long, and gives a score for how likely it is to contain human speech. A door slam or a beep gets a low score. A voice gets a high score.

Layer 2: Is it long enough? A cough or a short "hmm" lasts a very short time, and the agent must keep speaking for both. So, the agent waits until the speech has lasted for a minimum time before reacting, for example around 200 to 300 milliseconds. Short bursts are ignored.

Layer 3: What are the words? The agent sends the audio to STT and looks at the words. If the words are only "uh-huh", "yeah", "okay", or "right", it is a backchannel. If the words are "wait", "no", "stop", or a real sentence, it is an interruption. Getting the words takes a little extra time, so this layer adds some delay.

Layer 4: Context. Some voice agents also use a small model that looks at what the agent was saying and what the user said, and predicts whether the user wants the turn. Let's say the agent just asked, "Is that correct?" and the user says "yeah". That "yeah" is a real answer, not a backchannel.

We can write the decision logic as below:

def classify_overlap(audio, agent_is_speaking):
    if not agent_is_speaking:
        return "normal_turn"

    if vad_score(audio) < 0.5:
        return "noise"  # no human speech

    if speech_duration(audio) < 0.3:  # in seconds
        return "noise"  # too short, like a cough

    words = transcribe(audio).lower().strip()

    if words in ["uh-huh", "yeah", "okay", "right", "mm-hmm"]:
        return "backchannel"

    return "interruption"

Here, we have:

• First, if the agent is not speaking, there is no overlap at all, so it is a normal turn.
• Then, if VAD says there is no human speech, it is noise.
• After that, if the speech is too short, it is treated as noise, like a cough.
• Then, we look at the words. If they are only a backchannel word, the agent keeps speaking.
• Finally, anything else is treated as a real interruption.

Note: The numbers here, like 0.5 and 0.3 seconds, are only for the sake of example. Real systems tune these values based on their use case. Layer 4 is left out of this code for the sake of understanding.

We have a detailed blog on how a voice agent tells a real interruption from a cough or background noise that explains these layers in depth.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Step 4: Act on the decision

Now, the agent knows what kind of overlap it is. It acts as below:

• Noise: Do nothing. Keep speaking.
• Backchannel: Keep speaking. Do not add "okay" as a new user turn, so that the LLM does not reply to it.
• Interruption: Stop right away.

Stopping right away needs care, because at the moment of interruption, the runners of our relay race are often still busy:

• The TTS is playing audio, and some more audio is already waiting in a buffer to be played.
• The LLM is still writing the rest of the reply.

So, the agent must do the following, quickly:

• Stop the audio that is playing and throw away the audio waiting in the buffer.
• Cancel the LLM so that it stops writing the old reply.
• Cancel the TTS so that it stops making new audio.
• Start listening to the user's full sentence as a new turn.

If the agent stops the speaker but forgets to clear the buffer, the user hears a few more words after they interrupted. A good agent stops within a few hundred milliseconds of the user starting to speak.

Some agents also do a soft stop. When the user first starts speaking, the agent lowers its volume or pauses, instead of fully cancelling. If the sound turns out to be a cough or a backchannel, it continues. If it turns out to be a real interruption, it cancels. This way, a false alarm does not throw away the reply, and the agent still reacts quickly while it waits for the words from Layer 3.

Step 5: Keep the conversation history correct

The LLM usually writes the full reply as text much faster than the TTS can speak it. So, when the user interrupts, the LLM has written the full sentence, but the user has only heard part of it.

Let's say the LLM wrote:

"Your order has three items: a phone case, a charger, and a pair of headphones."

But the user said, "Wait, I want to cancel the charger," right after hearing "Your order has three items: a phone case, a charger". If the agent saves the full text in the conversation history, the LLM believes that the user also heard about the headphones. Later, it says, "As I mentioned, the headphones..." The user never heard that.

So, the agent must cut the saved reply down to only the part that was actually played. It does this by tracking how much audio was played before it stopped, and matching that to the words. Many TTS systems give a timestamp for each word, which makes this matching easier.

LLM wrote:   "Your order has three items: a phone case, a charger, and a pair of headphones."
User heard:  "Your order has three items: a phone case, a charger"
Saved:       "Your order has three items: a phone case, a charger"

Some systems also add a small note in the history that the agent was interrupted. Now, the LLM knows exactly where the conversation stands, and its next reply makes sense.

If we want to go deep into Memory in Agents, Context Engineering, and how to Design a Real-Time Voice AI Agent, our AI and Machine Learning Program at Outcome School covers them in depth.

What if both start speaking at the same moment?

Now, let's discuss Kind 4, where both voices begin at the same moment. The agent detects it with the same steps as an interruption, because it hears user speech while it is speaking.

A common rule for a voice agent is: the user always wins. The agent stops, and the user gets the turn.

This rule makes sense because the user's words are new information, while the agent can always repeat what it was going to say. Once the user finishes, the agent can reply to what the user said, and continue its old point only if it is still needed.

To reduce these collisions, the agent can also wait a little longer after a natural pause before it starts its next sentence. This gives the user a small window to jump in.

How full-duplex models handle overlap

Now, it's time to learn about a different design, where one model replaces the many separate parts of the relay race.

A full-duplex speech model is a single model that listens and speaks at the same time. Full-duplex means both sides can talk at once, just like a phone call between two people.

Moshi, a speech model from Kyutai, is an example. It models the user's audio and its own audio as two parallel streams, and at every small time step, about every 80 milliseconds, it processes both streams together. Because of this, it has no explicit speaker turns, and it does not need a separate VAD or a hand-written rule to decide when to stop.

Instead, the model learns from conversation data when people stop, when they say "uh-huh" and keep going, and when they speak over each other. So, overlap becomes a normal part of the conversation for the model, not a special case.

Let me tabulate the differences between the two designs for your better understanding.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

A complete example

Now, let's put everything together with one call.

The agent is reading the router reset steps: "First, unplug the router. Then, wait for thirty seconds. After that, plug it back in and..."

• The agent's own voice comes back through the microphone. AEC subtracts it, so the agent does not stop itself.
• After "wait for thirty seconds", the user says "okay". VAD finds speech, STT gives "okay", and the agent marks it as a backchannel. It keeps speaking, and "okay" is not added as a new turn.
• During "plug it back in", the user says, "Wait, which light should be green?" VAD finds speech, it lasts long enough, and the words are a real question. The agent marks it as an interruption.
• The agent stops the audio, clears the buffer, and cancels the LLM and TTS.
• The agent saves only "First, unplug the router. Then, wait for thirty seconds. After that, plug it back in" in the history.
• The agent listens to the full question and replies, "The power light should be green."

This is how a voice agent handles the user and the agent talking at the same time.

Where it works well and where it fails

It works well when:

• The audio is clean and AEC is working properly.
• The user's interruptions are clear, like "wait" or "no, not that".
• The backchannels are common words like "okay" and "yeah".

It fails when:

• AEC is weak, for example on a loud speakerphone, and the agent hears its own voice and stops itself.
• Another person in the room talks, and the agent thinks the user is interrupting.
• The user says "yeah" as a real answer, and the agent treats it as a backchannel and keeps talking.
• The minimum speech time is set too high, so the agent feels slow to stop, or too low, so it stops for every cough.
• The agent forgets to cut the history, and later talks about things the user never heard.

Now we must have understood how a voice agent handles the user and agent talking at the same time, what kinds of overlap exist, and how the agent decides whether to keep speaking or stop.

Frequently Asked Questions

Is "yeah" always a backchannel in a voice agent?

No. A "yeah" is not always a backchannel. If the agent just asked, "Is that correct?" and the user says "yeah", it is a real answer. This is why some voice agents use a small model that looks at what the agent was saying and what the user said. If the agent treats such a "yeah" as a backchannel, it keeps talking when it should stop.

Why should a backchannel not be added to the conversation history as a new user turn?

A backchannel should not be added as a new user turn because the LLM would then reply to it. When the user says "okay" or "uh-huh" while the agent is speaking, the user only means "I am following you, please continue". So, the agent keeps speaking and does not treat it as something to answer.

What is a soft stop in a voice agent?

A soft stop is when the agent lowers its volume or pauses as soon as the user starts speaking, instead of fully cancelling its reply. If the sound turns out to be a cough or a backchannel, it continues. If it is a real interruption, it cancels. This way, a false alarm does not throw away the reply, and the agent still reacts quickly.

How fast should a voice agent stop when the user interrupts?

A good voice agent stops within a few hundred milliseconds of the user starting to speak. To do this, it must stop the audio that is playing and also throw away the audio waiting in the buffer. If it stops the speaker but forgets to clear the buffer, the user hears a few more words after they interrupted.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.