Outcome School

Why does a voice agent cut people off mid-sentence?

Authors
Why does a voice agent cut people off mid-sentence?

A voice agent cuts people off mid-sentence because it often decides that a person has finished talking just by noticing a short silence, and people pause in the middle of their sentences all the time. When the agent hears that pause, it wrongly thinks it is its turn to speak, and it starts replying while the person still has more to say.

In this blog, we will learn about why a voice agent cuts people off mid-sentence. We will also see how a voice agent listens and decides when to reply, why a simple silence timer gets it wrong, the common situations that cause it, why making the agent wait longer creates a new problem, how modern voice agents fix it by understanding the meaning of what we said, and how they recover when they still get it wrong.

We will cover the following:

  • How a voice agent decides it is time to reply
  • The silence timer
  • Why a pause is not the end of a sentence
  • Common reasons a voice agent cuts us off
  • Why not just wait longer?
  • How does semantic turn detection fix it?
  • Recovering when the agent still gets it wrong

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

How a voice agent decides it is time to reply

A voice agent is an AI system that we can talk to using our voice, and it replies to us with a voice of its own. We find them in customer support phone lines, in voice assistants, and in apps where we speak instead of typing.

Many voice agents work in a few steps, like below:

+-----------------------------+
|  User speaks into the mic   |
+-----------------------------+
              ↓
+-----------------------------+
|  Speech-to-Text (STT)       |
|  converts voice into text   |
+-----------------------------+
              ↓
+-----------------------------+
|  LLM reads the text and     |
|  writes a reply             |
+-----------------------------+
              ↓
+-----------------------------+
|  Text-to-Speech (TTS)       |
|  converts reply into voice  |
+-----------------------------+
              ↓
+-----------------------------+
|  Agent speaks the reply     |
+-----------------------------+

Here, we can notice that the LLM (Large Language Model) needs our full sentence before it can write a good reply, because half a sentence gives it only half the meaning. So, before the LLM starts writing, the voice agent must answer one question:

Has the user finished speaking, or are they still talking?

Deciding this is called turn detection, and the moment when it decides "the user is done" is called the end of turn. In a conversation, a turn is one person's chance to speak. When we finish our turn, the other person takes theirs.

Humans do this without thinking. We look at the face of the other person, we listen to their tone, and we understand the meaning of their words. A simple voice agent has none of this when it decides the end of turn. It only has a stream of sound coming from the microphone.

So, how does it decide? The answer is a silence timer.

The silence timer

The simplest way to detect the end of turn has two parts.

The first part is Voice Activity Detection (VAD). VAD is a small, fast model that listens to the audio and answers one question again and again, many times per second: "Is someone speaking right now, or is it silent?"

The second part is a timer. The rule is simple:

If VAD hears silence for longer than a fixed time, the user has finished speaking.

This fixed time is called the silence threshold. It is often set somewhere between a few hundred milliseconds and one second.

Let's say the silence threshold is 500 milliseconds (half a second). We can write the logic as below:

SILENCE_THRESHOLD_MS = 500
silence_ms = 0
user_is_talking = False

for chunk in audio_stream:          # each chunk is 20 ms of audio
    if vad.is_speech(chunk):
        user_is_talking = True
        silence_ms = 0              # user is talking, reset the timer
    elif user_is_talking:
        silence_ms += 20            # silence after speech, add to the timer

    if user_is_talking and silence_ms >= SILENCE_THRESHOLD_MS:
        end_of_turn()               # send the text to the LLM and reply
        user_is_talking = False
        silence_ms = 0

Here, we have:

  • audio_stream gives us small pieces of audio, 20 milliseconds each.
  • vad.is_speech(chunk) tells us if that small piece has a human voice in it.
  • Whenever the user speaks, we mark user_is_talking as True and reset silence_ms to 0.
  • Whenever it is silent after the user has spoken, we keep adding 20 to silence_ms.
  • Once the silence reaches 500 milliseconds, we call end_of_turn(), and the agent starts replying.

This approach is simple and fast. But it has one big problem. It only looks at the silence. It does not care about what we said.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Why a pause is not the end of a sentence

People do not speak in one smooth stream. We pause all the time, and that too in the middle of our sentences.

The best way to learn this is by taking an example.

Suppose we call a voice agent to book a table at a restaurant, and we say:

"I want to book a table for... [pause of 0.7 seconds] ...four people tomorrow evening."

Let's see what happens with a silence threshold of 500 milliseconds, step by step:

Step 1: We say "I want to book a table for". VAD hears speech, and the timer stays at 0.

Step 2: We pause to think. VAD hears silence. The timer goes 20, 40, 60, and so on.

Step 3: The timer reaches 500 milliseconds. Our pause is not over yet, because it is 700 milliseconds long. But the agent has already decided that we are done.

Step 4: The agent sends "I want to book a table for" to the LLM, and the agent starts saying "Sure. For how many people?"

Step 5: We continue with "...four people tomorrow evening." at the same time. Now, both of us are talking over each other.

This is exactly what it means when a voice agent cuts people off mid-sentence. The timer worked as designed, but the rule behind it is wrong. A pause of 500 milliseconds does not always mean the end of a sentence.

Now, the question is: what makes the agent hear a gap when we are not done yet? Let's see the most common reasons.

Common reasons a voice agent cuts us off

Thinking pauses: We pause while we think about what to say next. "My order number is... let me check... 4 5 7 2."

Filler words: We say "um", "uh", or "so..." and then stop for a moment. These words are usually a sign that we are not done, but a silence timer does not understand words.

Reading out numbers and lists: When we read a phone number, a card number, or an address, we naturally pause between the groups. "My number is 98765... 43210." The agent can jump in after the first group.

Slow speakers: Some people speak slowly, for example elderly people, or people who are speaking in a language that is not their first language. Their normal gaps between words can be longer than the threshold.

Background noise: Sometimes VAD gets confused by noise. A car horn, a fan, or another person talking can make it think speech has stopped or started at the wrong moment.

Network delays: On phone and internet calls, small pieces of audio can arrive late or get lost on the way. For a moment, the agent receives nothing, and it can look like silence even though we are still speaking.

Here, we can notice one common thing. In every case, the agent hears a gap, but the meaning is "I am not done yet." The silence timer cannot see the meaning.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Why not just wait longer?

The first idea that comes to mind is simple. Let's increase the silence threshold from 500 milliseconds to 2 seconds. Then, most of the thinking pauses will be shorter than the threshold, and the agent will stop cutting us off.

But this creates a new problem. Now, every time we really finish speaking, the agent waits 2 full seconds before it even starts preparing a reply. Then, the LLM needs time to write the reply, and the TTS needs time to turn it into voice. The total delay becomes long, and the conversation feels slow. We start wondering if the agent heard us at all, and some people say "Hello?" again, which confuses the agent further.

So, we are stuck between two bad options:

Silence thresholdWhat goes wellWhat goes wrong
Short (300 ms)Agent replies quicklyAgent cuts people off mid-sentence
Long (2 seconds)Agent rarely cuts people offAgent feels slow, awkward silences

The people who build voice agents want them to feel fast, so they often keep the threshold short, and a short threshold means more interruptions.

A single fixed number can never be right for every pause. The agent needs something smarter than a timer.

To learn how to balance fast replies and accurate turn detection when we Design a Real-Time Voice AI Agent, we can check out our AI and Machine Learning Program at Outcome School.

How does semantic turn detection fix it?

So, here comes the semantic turn detection to the rescue.

Semantic turn detection is a way of deciding the end of turn by looking at the meaning of what the user said, and not only at the silence.

The word semantic means "related to meaning". Means, instead of asking "How long has it been silent?", the agent asks "Does what the user said so far sound like a complete thought?"

Let's take our restaurant example again:

  • "I want to book a table for" sounds incomplete. A sentence like this does not end with "for". So, the agent must keep listening.
  • "I want to book a table for four people tomorrow evening." sounds complete. So, the agent can reply.

To do this, a small, fast model looks at what the user has said so far, and it gives a score for how likely it is that the user has finished. This is often called a turn detection model. Some turn detection models read the text from Speech-to-Text, and some listen to the audio itself, because the voice carries extra clues. For example, our pitch often drops at the end of a statement, and it often stays level when we are still in the middle of a thought.

Now, the agent combines both signals, the silence and the meaning, as below:

silence_ms = get_silence_duration()
done_score = turn_model.predict(transcript)   # 0.0 = not done, 1.0 = done

if done_score > 0.8 and silence_ms >= 300:
    end_of_turn()        # complete thought and a short pause, reply quickly
elif silence_ms >= 2000:
    end_of_turn()        # incomplete thought, but a very long pause
# otherwise, keep listening

Here, we have:

  • turn_model.predict(transcript) reads the text so far and gives a score between 0 and 1 for how complete the sentence is. We are using a text-based turn detection model here for the sake of understanding.
  • If the sentence looks complete and there is a short pause, the agent replies quickly.
  • If the sentence looks incomplete, the agent keeps waiting, even after the pause crosses 300 milliseconds.
  • If the pause becomes very long, the agent replies anyway, because the user has probably stopped.

The values 0.8, 300, and 2000 here are just for the sake of understanding. Real systems tune them based on their use case.

Let's see the restaurant example again with this approach:

Step 1: We say "I want to book a table for" and pause. The turn detection model gives a low score, because the sentence is incomplete.

Step 2: The silence crosses 300 milliseconds, but the score is low. So, the agent keeps listening.

Step 3: We say "four people tomorrow evening." and stop. The turn detection model gives a high score.

Step 4: The silence crosses 300 milliseconds, and the score is high. The agent replies quickly.

The agent waited during the thinking pause, and it replied fast when we were really done.

We have a detailed blog on how silence-based and model-based turn detection differ that explains this in depth.

There are a few more fixes that help along with it:

  • Context-aware waiting: If the agent has just asked "What is your phone number?", it can wait longer, because people pause while reading numbers.
  • Filler word awareness: If the last word is "um" or "uh", the agent can wait longer, because the user is still thinking.
  • Signals from Speech-to-Text: Some Speech-to-Text services send their own signal when they think the user has stopped speaking, and the agent can use it as one more clue.
  • Better VAD and noise removal: A cleaner audio signal means fewer fake silences and fewer false alarms of speech.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Recovering when the agent still gets it wrong

No turn detection is perfect. Sometimes, the agent will still start speaking too early. So, a good voice agent must also know how to recover.

Here comes the barge-in into the picture. Barge-in is the ability of the user to interrupt the agent while it is speaking. When the agent is speaking and it hears the user start talking again, it must do the following:

  • Stop speaking immediately.
  • Throw away the rest of the reply it was speaking, and remember only the part that the user really heard.
  • Keep listening to the user.
  • Join the new words with the old words, so that "I want to book a table for" and "four people tomorrow evening" become one sentence.
  • Write a fresh reply for the full sentence.

One more thing to notice is that not every sound from the user means "stop". When we say "uh-huh", "okay", or "right" while the agent is talking, we usually mean "I am listening, go on". These short sounds are called backchannels. If the agent stops every time it hears "uh-huh", it will keep stopping for no reason. So, a good voice agent must tell the difference between a real interruption and a backchannel.

Also, the agent must not mistake its own voice for the user. On a speakerphone, the voice of the agent comes out of the speaker and goes back into the microphone. Without echo cancellation, which removes the agent's own voice from the microphone audio, the agent can think the user is talking and interrupt itself.

Now we must have understood why a voice agent cuts people off mid-sentence, how the silence timer causes it, why waiting longer is not the answer, how semantic turn detection fixes it, and how barge-in handling helps the agent recover when it still gets it wrong.

Frequently Asked Questions

What is a silence threshold in a voice agent?

A silence threshold is the fixed time of silence after which a simple voice agent decides that the user has finished speaking. It is often set somewhere between a few hundred milliseconds and one second. If VAD hears silence for longer than this time, the agent ends the turn and starts replying.

What is Voice Activity Detection (VAD)?

Voice Activity Detection (VAD) is a small, fast model that listens to the audio and answers, many times per second, whether someone is speaking or it is silent. A simple voice agent pairs VAD with a timer to decide the end of turn. VAD can get confused by noise like a car horn, a fan, or another person talking.

What is a backchannel in a voice conversation?

A backchannel is a short sound like "uh-huh", "okay", or "right" that we make while the other person is talking, and it means "I am listening, go on". A good voice agent must tell the difference between a backchannel and a real interruption. Otherwise, it keeps stopping its reply for no reason.

Why does a voice agent sometimes interrupt itself on a speakerphone?

On a speakerphone, the voice of the agent comes out of the speaker and goes back into the microphone. Without echo cancellation, which removes the agent's own voice from the microphone audio, the agent can think that the user is talking. Then, it stops and interrupts itself.

Can a turn detection model listen to the audio instead of reading the text?

Yes. Some turn detection models read the text from Speech-to-Text, and some listen to the audio itself, because the voice carries extra clues. For example, our pitch often drops at the end of a statement, and it often stays level when we are still in the middle of a thought.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.