Outcome School

How does a voice agent recover from a false interruption?

Authors
How does a voice agent recover from a false interruption?

A voice agent recovers from a false interruption by pausing its speech instead of stopping it, waiting a short moment to check if the user really said something, and then continuing from where it paused if the user did not.

In this blog, we will learn about how a voice agent recovers from a false interruption. We will also see what a false interruption is, why pausing works better than stopping, how the agent decides that an interruption was false, how it continues speaking from the right place, how it keeps the conversation history correct, and how to tune the waiting time.

We will cover the following:

  • What is a false interruption?
  • What goes wrong without recovery
  • Pause, do not stop
  • How does the agent decide that an interruption was false?
  • How does the agent continue speaking?
  • When the interruption turns out to be real
  • Keeping the conversation history correct
  • A complete example from start to end
  • Tuning the waiting time

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is a false interruption?

Before jumping into a false interruption, we must know what a normal interruption is in a voice agent.

A voice agent is a program that talks with us using voice. We speak, it listens, it thinks using an LLM (Large Language Model), and it speaks back. While the agent is speaking, we can start talking over it, and the agent stops to listen to us. This is called barge-in. It is what makes a voice agent feel like a real conversation and not like a recorded message.

To support barge-in, the agent keeps listening even while it is speaking. The moment it hears something that sounds like the user's voice, it treats it as an interruption.

But, the agent does not hear only words. It hears every sound that reaches the microphone. In the first fraction of a second, a cough or a loud "hmm" can look a lot like the start of a sentence.

A false interruption is a moment when the agent thinks the user wants to take over the conversation, but the user did not really mean to interrupt.

Some common causes of a false interruption are:

  • The user coughs, sneezes, or clears their throat.
  • The user says a short "uh-huh", "okay", or "hmm" just to show that they are listening.
  • A door closes, a dog barks, or a car honks near the user.
  • Someone else in the room says something.
  • A TV or radio is playing in the background.
  • The agent's own voice comes back into the microphone from the speaker. This is called an echo.

In all these cases, the user still wants the agent to continue. But a simple agent hears a sound and stops.

What goes wrong without recovery

Let's say we have a voice agent for a bank. The user asks, "How can I reset my card PIN?"

The agent starts speaking: "Sure. First, open the app and go to the Cards section. Then, tap on Reset PIN and..."

At this moment, the user coughs.

A simple agent hears the cough, treats it as an interruption, and stops speaking right away. It also throws away the rest of its answer, because it thinks the user wants to say something new.

Now, the agent waits for the user to speak. But the user has nothing to say. The user was only coughing.

So, there is silence. After a few seconds, the user gets confused and says, "Hello? Are you there?" or "You stopped in the middle."

This creates the following problems:

  • The user never hears the rest of the steps.
  • There is an awkward silence that makes the agent feel broken.
  • The user has to ask the same question again.
  • The agent spends extra time and money to create a new answer.

This is why a voice agent needs a way to recover from a false interruption. Now, let's see how it does that.

Pause, do not stop

The best way to learn this is by taking an example.

Suppose we are reading a story aloud to a friend. In the middle, our friend sneezes. We do not throw away the book and wait for them to start a new topic. We pause for a moment, see that they are fine, and continue from the same line.

A voice agent can do the same thing. Let's see how the approaches improved one by one to reach this behavior.

Approach 1: Stop and throw away. The agent stops speaking on any sound and throws away the rest of its reply. This is what we saw above. The issue with this approach is that a single cough loses the whole answer. Let's see how the next approach solve this issue.

Approach 2: Stop, then create the answer again. The agent stops on any sound. If the user says nothing, the agent asks the LLM to create a new reply. The issue with this approach is that it is slow and the new reply often repeats the start of the old one, so the user hears the same lines again. Let's see how the next approach solve this issue.

Approach 3: Pause and hold. The agent pauses its audio the moment it hears a possible interruption, but it keeps the rest of the reply ready. If the interruption turns out to be false, it continues from where it paused. If the interruption turns out to be real, it stops for good and listens.

Approach 3 is how a voice agent recovers from a false interruption.

Here is the difference between pausing and stopping:

  • Stopping means the rest of the reply is gone.
  • Pausing means the rest of the reply is still there, waiting.

The pause gives the agent time to decide, without losing anything.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

How does the agent decide that an interruption was false?

Now, the question is: after pausing, how does the agent know that the sound was not a real interruption?

The answer is: it waits for a short time after the sound ends and checks what the user did in that time. This short time is called the false interruption timeout. For example, LiveKit Agents uses 2 seconds for this timeout by default.

During this waiting time, the agent checks the following things.

Check 1: Did the user keep speaking? A small model called VAD (Voice Activity Detection) tells the agent whether a piece of audio contains human speech, and for how long. If the speech lasts only a fraction of a second and then stops, it is often not a real interruption.

Check 2: Did the user say any real words? The agent keeps sending the user's audio to a speech-to-text model. If no words come back, the sound was most likely a cough, a bang, or background noise.

Check 3: Were the words only a filler? If the text is only "uh-huh", "okay", "yeah", or "hmm", the user was just showing that they are listening. These short sounds are called backchannels. They are not a request to take over. But, if the filler is followed by more words, like "okay, but what about my old card?", then it is a real interruption. This is one more reason why the agent waits for a short time instead of deciding on the first word.

Some newer systems also use a small audio model trained to tell a real barge-in apart from other sounds, like coughs and backchannels. It listens to the sound itself and gives a decision, without waiting for the full text.

If the user stays quiet, or says only a filler, until the timeout ends, the agent marks the interruption as false.

We have a detailed blog on how a voice agent tells a real interruption from a cough or background noise that explains these checks step by step.

How does the agent continue speaking?

Now that we have learned how the agent decides, it's time to learn how it continues speaking. Once the agent knows the interruption was false, it needs to continue speaking from the right place.

To do this, the agent must track how much of its reply has already been played to the user. Let's understand this with our bank example.

The agent's full reply is:

"Sure. First, open the app and go to the Cards section. Then, tap on Reset PIN and enter your new PIN twice."

The voice for this reply is created by a text-to-speech model and played in small pieces of audio. The agent keeps a count of how much audio has been played. Many text-to-speech models also tell the agent the time at which each word is spoken, so the agent can match the audio position to the exact word.

When the user coughed, the agent had played up to "Then, tap on Reset PIN and". The agent pauses exactly here and remembers this point.

After the timeout, the agent continues from that point: "...enter your new PIN twice."

Since the rest of the audio was already ready and only paused, nothing new has to be created. The user hears a short gap, and then the answer continues.

An agent can also step back a little and repeat the start of the current sentence, so that the user does not hear a word cut in half. Both ways work, and the choice depends on our use case.

Note: Resuming works only when the audio output supports pausing. If the audio output can only play or stop, for example in some talking-avatar video setups, the agent cannot freeze the audio in place, and it must fall back to stopping.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

When the interruption turns out to be real

Now, let's see the other case. Not every interruption is false. Many times, the user really wants to speak.

Let's take our bank example again. The agent has just said "Then, tap on Reset PIN and", and the user says, "Wait, I do not have the app."

Here, the speech-to-text model returns real words, and they are not a filler. So, the agent decides that the interruption is real. It does not need to wait for the timeout to end.

Now, the agent does the following:

  • It stops the paused audio for good and throws away the rest of the reply.
  • It cuts the conversation history at the point where it stopped.
  • It listens to the full message from the user.
  • It sends the message to the LLM to create a new reply.

So, if the interruption is real, the agent turns the pause into a stop. We will learn why cutting the history matters in the next section.

Keeping the conversation history correct

Now, there is one more thing to handle. The LLM keeps a history of the conversation, and it must match what the user actually heard.

Let's see both cases.

Case 1: The interruption was false. The agent continued and finished the full reply. So, the full reply goes into the history. The LLM correctly believes that the user heard everything.

Case 2: The interruption was real. The agent stopped after "Then, tap on Reset PIN and". The user never heard the rest. So, the agent must cut the reply in the history at the exact point where it stopped.

If the agent does not cut the history, the LLM believes the user heard "enter your new PIN twice", even though they did not. Then, later in the conversation, the LLM can say "As I mentioned, enter your PIN twice", and the user will be confused.

This is why tracking the played audio position is important. It is used both to continue speaking and to keep the history correct.

To learn Memory in Agents, Context Engineering, and how to Design a Real-Time Voice AI Agent from the ground up, check out our AI and Machine Learning Program at Outcome School.

A complete example from start to end

Now, let's put everything together and walk through one full example. For the sake of understanding, let say the false interruption timeout is 2 seconds, and it starts counting once the sound ends.

Step 1: The user asks, "How can I reset my card PIN?" The agent starts speaking the answer.

Step 2: At 0.0 seconds, while the agent is speaking, the user coughs. The VAD reports a sound that can be speech.

Step 3: Within a fraction of a second, the agent pauses the audio. It remembers that it has played up to "Then, tap on Reset PIN and".

Step 4: At 0.3 seconds, the cough ends. The agent starts the 2-second false interruption timeout and keeps listening.

Step 5: The speech-to-text model returns no words, and the user stays quiet.

Step 6: At 2.3 seconds, the timeout ends. The agent marks the interruption as false.

Step 7: Right after 2.3 seconds, the agent continues from the paused point: "...enter your new PIN twice."

Step 8: The full reply goes into the conversation history.

We can see this flow in a diagram as below:

+-----------------------------+
|      Agent is speaking      |
+-----------------------------+
               ↓
+-----------------------------+
|  VAD hears possible speech  |
+-----------------------------+
               ↓
+-----------------------------+
|  Pause audio, save position |
+-----------------------------+
               ↓
+-----------------------------+
|  Check the words until the  |
|         timeout ends        |
+-----------------------------+
               ↓
       +-------+-------+
       ↓               ↓
 Noise/filler     Real words
       ↓               ↓
+-------------+ +-------------+
| Resume from | |  Stop, cut  |
| saved point | |   history,  |
|             | |    listen   |
+-------------+ +-------------+

Here, we can see that every interruption first goes into a pause. Only after the check does the agent decide whether to resume or to stop.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Tuning the waiting time

Now, let's discuss the length of the false interruption timeout.

If the timeout is too short: The agent resumes too quickly. A user who takes a breath before their real sentence, or whose words take a moment to turn into text, gets talked over again. This feels rude.

If the timeout is too long: After a cough, the user hears a long silence before the agent continues, and starts wondering if the agent has frozen. This feels slow and broken.

So, we must pick a value that is long enough to catch real speech and short enough that a cough does not create an awkward gap.

A few other settings help along with the timeout:

  • A minimum speech length, so that very short sounds, like a tap on the table, do not even cause a pause.
  • A minimum number of words, so that a single filler word does not count as an interruption.
  • Echo cancellation, which removes the agent's own voice from the microphone audio, so that the echo does not trigger a pause.

We must test these settings with real users and real background noise, because a quiet office and a busy street behave very differently.

Now we must have understood how a voice agent recovers from a false interruption, how it pauses instead of stopping, how it decides that an interruption was false, how it continues speaking from the right place, and how it keeps the conversation history correct.

Frequently Asked Questions

What is the false interruption timeout?

The false interruption timeout is the short time a voice agent waits after a sound ends before it decides that the interruption was false. During this time, it checks whether the user kept speaking and said real words. For example, LiveKit Agents uses 2 seconds for this timeout by default.

Does the agent create a new reply after a false interruption?

No. When the agent pauses instead of stopping, the rest of its reply is still ready. Once it decides the interruption was false, it continues from the exact point where it paused, so nothing new has to be created. The user hears only a short gap, and then the answer continues.

Can every voice agent pause and resume its audio?

No. Resuming works only when the audio output supports pausing. If the audio output can only play or stop, for example in some talking-avatar video setups, the agent cannot freeze the audio in place. In that case, it must fall back to stopping.

How can we stop the agent's own voice from causing a false interruption?

We can use echo cancellation. The agent's own voice can come back into the microphone from the speaker, and this echo can look like the user speaking. Echo cancellation removes the agent's own voice from the microphone audio, so that the echo does not trigger a pause.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.