How does a voice agent tell a real interruption from a cough or background noise?

A voice agent tells a real interruption from a cough or background noise by passing every sound through a series of checks, and it stops talking only when the sound passes all of them. It removes its own voice and the noise, then checks if the sound is human speech, lasts long enough, and contains real words.

In this blog, we will learn about how a voice agent tells a real interruption from a cough or background noise. We will also see why this is hard, how the agent cleans the sound before judging it, how it checks for human speech, length, and real words, how it pauses first and resumes when it was wrong, how a trained model helps, and where it works well and where it fails.

We will cover the following:

What is an interruption in a voice agent?
Why is it hard to tell a real interruption from noise?
Step 1: Remove the agent's own voice
Step 2: Remove the background noise
Step 3: Is it human speech?
Step 4: Did it last long enough?
Step 5: Did the user say real words?
Step 6: Pause first, then decide
Using a trained interruption model
Putting it all together
Where it works well and where it fails

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is an interruption in a voice agent?

A voice agent is a system that talks with us using voice, like a customer support agent on a phone call. We speak, it listens, and then it speaks back.

An interruption is when the user starts speaking while the agent is still talking, because the user wants the agent to stop and listen. This is also called barge-in.

Let's say the agent is reading out a long list of flight options. After the first option, we say, "Wait, I want the morning flight." The agent must stop talking right away and listen to us. This is a real interruption.

But, a phone call is never perfectly quiet. While the agent is talking, many other sounds reach the microphone:

The user coughs or sneezes.
A door closes or a dog barks.
A TV is playing in the next room.
A car horn sounds outside.
The user says "uh-huh" just to show that they are listening.

None of these mean "stop talking". When the agent stops for one of them anyway, it is called a false interruption.

So, the agent must answer one question again and again, many times every second: Is this sound a real interruption, or is it just noise?

Deciding when to stop and listen is one part of a bigger job. We have a detailed blog on turn-taking in a voice AI agent that explains how the agent and the user share the conversation.

Why is it hard to tell a real interruption from noise?

For us humans, this is easy. When our friend coughs in the middle of our sentence, we keep talking. When our friend says "Wait", we stop.

But, the agent does not hear sounds the way we do. The microphone gives it only a long stream of numbers. Each number tells how strongly the air was pushing on the microphone at that tiny moment. A cough, a dog bark, and the word "Wait" are all just numbers in this stream.

On top of this, the agent has to decide very fast. If it takes too long, the user keeps talking over the agent and feels that the agent is not listening. If it stops at the very first sound, it stops for every cough.

Means, the agent is stuck between two mistakes:

Stopping too easily: The agent stops for a cough or a door. The user is confused and has to say "Please continue".
Stopping too late: The agent keeps talking while the user is trying to say something important. The user feels ignored.

No single check solves this. So, the agent uses a series of checks, one after another. Each check removes some of the wrong sounds, and only the sounds that pass all the checks can stop the agent. Let's understand each check step by step.

Step 1: Remove the agent's own voice

Before judging any sound, the agent must handle one problem: it can hear itself.

When the agent's voice plays from the user's phone speaker, it can go right back into the same phone's microphone. If the agent does not handle this, it hears its own voice, thinks the user is speaking, and interrupts itself.

Here comes the Echo Cancellation into the picture.

The system knows exactly what sound it is playing through the speaker, because the agent created that sound. So, it can predict what that sound looks like when it comes back into the microphone, and subtract it from what the microphone hears. What remains is mostly the sound from the room and the user.

For the sake of understanding, consider this:

Microphone hears:   Agent's voice + User's voice + Noise
Subtract:           Agent's voice
What remains:       User's voice + Noise

Now, the agent is no longer confused by its own voice. Now, let's handle the noise.

Step 2: Remove the background noise

The next step is Noise Suppression. Today, it is usually a model that has learned what human speech sounds like and what common noises sound like, such as fans, traffic, typing, and wind. It keeps the speech and lowers everything else.

Let's say the user is calling from a busy street. Before noise suppression, the microphone has the user's voice mixed with car engines. After noise suppression, the car engines become very quiet and the user's voice stays clear.

This removes a large part of the problem. A steady fan or the sound of traffic does not reach the next checks.

Some systems go one step further with background voice cancellation. It tries to keep only the voice of the main speaker, the person closest to the microphone, and lowers the voices of other people, like a TV or someone talking in the next room. This matters because a voice on the TV is real human speech, but it is not our user.

But, noise suppression cannot remove everything. A cough is made by a human mouth, so it shares some sound qualities with speech. A sudden loud bang can also slip through. Hence, we need more checks.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

AI Engineer
Forward Deployed Engineer
AI Agent Engineer
Harness Engineer
Voice AI Engineer
AI Architect
LLM Engineer
LLM Inference Engineer
Gen AI Engineer
MLOps Engineer
ML Engineer
Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Step 3: Is it human speech?

Now, the agent asks: is this cleaned sound human speech at all?

Here comes the Voice Activity Detection (VAD) to the rescue. VAD is a small model that listens to tiny pieces of audio, often around 30 milliseconds long, and gives a score between 0 and 1 for each piece. The score tells how likely it is that the piece contains human speech.

A score close to 1 means "this is very likely speech".
A score close to 0 means "this is not speech".

The agent picks a cut-off, for example 0.5. Pieces above the cut-off are treated as speech.

The VAD model is trained on many hours of audio where people have marked which parts are speech and which parts are not. So, it learns the patterns of the human voice. A dog bark, a door closing, or music without singing usually gets a low score.

But, a cough, a laugh, or a loud "hmm" can still get a high score, because these sounds come from a human throat. VAD can tell a human voice from other sounds quite well, but it cannot reliably tell a cough from a word.

Step 4: Did it last long enough?

Now, the agent looks at time.

Let's compare a cough with a real interruption:

A cough is very short, usually a fraction of a second.
A real interruption, like "Wait, I want the morning flight", keeps going for longer.

So, the agent uses a rule: the speech must continue for a minimum amount of time before it counts as an interruption. Some voice agent frameworks use a default of about half a second (0.5 seconds), and we can change it based on our use case.

Let's see how this works with a timeline:

Time (seconds):     0.0   0.1   0.2   0.3   0.4   0.5   0.6
Cough:              ###########
User says "Wait":   #####################################
                                                  ↑
                                          0.5 second mark

Here, # marks the time when VAD hears speech. We can see that the cough ends before the 0.5 second mark, so it never reaches the minimum and the agent keeps talking. The real interruption keeps going past the mark, so it passes this check.

This check removes a large number of false interruptions, because most coughs, sneezes, claps, and bangs are short.

Now, there is a trade-off:

A longer minimum removes more coughs, but the agent takes longer to stop for a real interruption.
A shorter minimum makes the agent stop faster, but more coughs get through.

We must choose the value based on our use case. A noisy call center needs a longer minimum. A quiet app where users expect a quick response can use a shorter one.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

LLM
RAG
MCP
Agent
Fine-tuning
Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Step 5: Did the user say real words?

Time alone is still not enough. A long coughing fit or a burst of laughter can easily last more than half a second.

So, the agent uses its strongest proof: words.

Most voice agents already have a Speech-to-Text model. It listens to the audio and writes down the words that the user says. The agent can use this text to check the interruption:

A cough gives no words. The Speech-to-Text model writes nothing.
Background noise also gives no words, or gives random broken pieces.
A real interruption gives real words like "Wait" or "Stop" or "No, not that one".

So, the agent can add a rule: only stop when the Speech-to-Text model has produced at least a certain number of words. Some systems wait for one word, some wait for two or more.

Let's see an example. The agent is talking, and the user coughs for 0.7 seconds:

VAD:            speech detected
Duration:       0.7 seconds (passes the 0.5 second minimum)
Speech-to-Text: "" (no words)
Decision:       not an interruption, keep talking

Now, the user says "Wait":

VAD:            speech detected
Duration:       0.6 seconds
Speech-to-Text: "wait"
Decision:       real interruption, stop talking

Here, we can notice that the cough passed the time check, but it failed the words check. The real interruption passed both.

The words also help with one more case: backchannels. A backchannel is a short sound we make to show that we are listening, like "uh-huh", "okay", "right", or "mm-hmm". It is real speech, and the Speech-to-Text model writes it down as words, but it does not mean "stop talking". It means "go on, I am listening".

So, some agents keep a small list of such words. If the user only says "uh-huh" or "okay" while the agent is talking, the agent treats it as a backchannel and keeps talking. If the user says "okay, but wait", the agent stops.

But, the words check has one problem: time. The Speech-to-Text model needs a moment to produce the first word. If the agent waits for words before doing anything, the user hears the agent talking over them for a bit longer. Let's see how the next step solves this issue.

Step 6: Pause first, then decide

Instead of choosing between "stop" and "keep talking" right away, the agent can do something in between: pause.

The idea is like this:

As soon as VAD hears speech, the agent pauses its voice, or lowers it. It does not throw away what it was going to say.
Then, it waits for a short time to see what happens next.
If the speech lasts long enough and real words arrive, it is a real interruption. The agent stops fully and listens to the user.
If no words arrive and the user goes quiet, it was a false interruption. The agent resumes speaking from where it paused.

Let's see the cough case with this approach:

Agent:  "Your flight leaves at nine in the morning and..."
User:   *cough*
Agent:  (pauses)
        (waits, hears silence, no words)
Agent:  "...and arrives at noon."

Here, the cough caused only a short pause, and the agent continued its sentence. To the user, it feels like a speaker who paused for a moment after hearing a cough.

In some frameworks, this waiting time is about 2 seconds of silence by default. A shorter wait makes the pause less noticeable, but the agent can then resume while the user is still about to speak.

This way, the agent reacts fast, because it pauses on the very first sign of speech. And, it does not lose its sentence when the sound turns out to be a cough.

Using a trained interruption model

All the checks above are rules made by people: a minimum time, a minimum number of words, a list of backchannel words. Rules are easy to set up, but they have limits. A cough and the word "Wait" can both be short. Some users say "yeah" to agree, and some say "yeah" to start a new question.

So, some newer voice agents use an interruption model. It is a model trained on many real conversations, where each sound that the user made while the agent was talking is labeled as one of these:

A real interruption, where the user wanted the agent to stop.
A backchannel, where the user was only showing that they were listening.
Noise, like a cough, a laugh, or a background sound.

The model listens to the audio itself, not just the text. So, it can use the tone, the pitch, and the rhythm of the sound. A sharp, rising "Wait" sounds different from a soft, flat "mm-hmm", even though both are short. Over many examples, the model learns these patterns better than a fixed rule can.

To learn Multimodal AI and how to Design a Real-Time Voice AI Agent end to end, check out our AI and Machine Learning Program at Outcome School.

Putting it all together

Now, let's see the full journey of a sound while the agent is talking. Each "If no" line shows where a sound leaves the flow:

Sound reaches the microphone
              ↓
Echo Cancellation: remove the agent's own voice
              ↓
Noise Suppression: remove fans, traffic, other voices
              ↓
VAD: is it human speech?
If no: ignore it and keep talking
              ↓
Agent pauses its voice
              ↓
Time check: did it last about 0.5 seconds?
If no: resume talking
              ↓
Words check: real words, not just "uh-huh"?
If no: resume talking
              ↓
Real interruption: stop and listen

Now, let's run three sounds through it:

A dog bark: Noise suppression lowers it, and VAD gives it a low score. The agent never even pauses.
A cough: VAD may say "speech", so the agent pauses. But, it is too short, or it has no words. The agent resumes.
"Wait, I want the morning flight": It passes VAD, lasts long enough, and has real words. The agent stops and listens.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Where it works well and where it fails

It works well when:

The noise is steady, like a fan, traffic, or an air conditioner. Noise suppression removes it easily.
The noise is short, like a cough, a clap, or a door. The time check and the words check catch it.
The user interrupts with a clear phrase, like "Wait a second" or "No, not that one".

It struggles when:

Another person is talking near the user. Their voice is real speech with real words. Background voice cancellation helps, but it is not perfect.
The user interrupts with one very short word, like "No". It is close to the minimum time and looks like a backchannel, so the agent can take a moment longer to stop.
The user says "yeah" or "okay" to start a new point. It looks like a backchannel, so the agent can keep talking when it should stop.
The line is very noisy, like a poor phone connection. Speech-to-Text can produce wrong words from noise, which can trigger a false interruption.

So, every setting is a trade-off. We must test the agent on real call recordings, with real coughs, real TVs, and real users, and tune the minimum time, the minimum words, and the pause timing based on our use case.

Now we must have understood how a voice agent tells a real interruption from a cough or background noise, why it uses a series of checks, how each check works, and where it works well and where it fails.

Frequently Asked Questions

What is a false interruption in a voice agent?

A false interruption is when the voice agent stops talking for a sound that does not mean "stop talking". This can be a cough, a sneeze, a door, a dog bark, a TV in the next room, or a short "uh-huh". The user is then confused and has to say "Please continue".

Should a voice agent stop talking when the user says "uh-huh"?

No. Words like "uh-huh", "okay", "right", and "mm-hmm" are backchannels. They mean "go on, I am listening", not "stop talking". Some agents keep a small list of such words and keep talking when the user only says one of them, but they stop if the user says something like "okay, but wait".

Can noise suppression alone stop false interruptions?

No. Noise suppression removes steady noise like fans and traffic, but it cannot remove everything. A cough is made by a human mouth, so it shares some sound qualities with speech, and a sudden loud bang can also slip through. That is why the agent adds more checks for human speech, length, and real words.

How long should the user speak before the agent treats it as an interruption?

Some voice agent frameworks use a default of about half a second, and we can change it based on our use case. A longer minimum removes more coughs but makes the agent slower to stop. A noisy call center needs a longer minimum, while a quiet app where users expect a quick response can use a shorter one.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.