How does a voice agent handle long pauses when the user is thinking?

A voice agent handles long pauses by not treating every silence as the end of the user's turn. It checks the words, the voice, and the time together, and it waits longer when the user sounds like they are still thinking.

In this blog, we will learn about how a voice agent handles long pauses when the user is thinking. We will also see why silence is confusing for a voice agent, why a simple silence timer fails, how the agent uses the words and the voice to decide, how it changes its waiting time, what it does when it speaks too early or the user stays silent for very long, and where it works well and where it fails.

We will cover the following:

• What is a thinking pause?
• Why is a long pause a hard problem?
• How does a voice agent hear silence?
• Approach 1: A fixed silence timer
• Approach 2: A longer silence timer
• Approach 3: Looking at the words
• Approach 4: Listening to how the words are said
• Changing the waiting time dynamically
• What if the agent starts speaking too early?
• What if the user stays silent for a very long time?
• The complete flow
• Where it works well and where it fails

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is a thinking pause?

A voice agent is an AI system that we talk to with our voice, and it talks back to us with its voice. A customer support bot on a phone call is a good example.

When we talk, we do not speak in one long smooth line. We stop in the middle to think.

Let's say we call a pharmacy voice agent and it asks, "What is your date of birth?"

We reply like below:

"It is the 14th of... [silence for 2 seconds] ...March, 1990."

That 2-second silence in the middle is a thinking pause. The user has not finished. The user is remembering the month.

A thinking pause is a silence in which the user is still in the middle of their turn.

Now, compare it with this:

"It is the 14th of March, 1990." [silence for 2 seconds]

Here also, there is a 2-second silence. But this time, the user has finished. The user is now waiting for the agent to reply.

Both silences are of the same length. One means "wait for me", and the other means "your turn now".

Why is a long pause a hard problem?

A conversation works by taking turns. One person speaks, then the other person speaks. This is called turn-taking.

A voice agent must decide, again and again, "Has the user finished speaking, or not?" This decision is called end-of-turn detection, and it is also known as endpointing.

If the agent gets this decision wrong, one of two bad things happens:

• Speaking too early: The agent cuts in while the user is still thinking. The user feels interrupted, and the agent replies to half a sentence. In our example, it hears only "It is the 14th of" and replies to an incomplete date.
• Speaking too late: The agent waits too long after the user has finished. The user hears dead silence and wonders if the call has dropped.

So, the agent is always balancing between two things: not interrupting, and not being slow.

How does a voice agent hear silence?

Before jumping into the approaches, we must know how a voice agent knows that the user is silent.

A voice agent uses a small model called Voice Activity Detection, or VAD. It listens to the incoming audio in very small pieces, often 10 to 32 milliseconds each, and for each piece it answers one question: "Is a human speaking in this piece, or not?"

So, the VAD gives us a simple signal like below:

speech speech speech silence silence silence silence speech speech

From this signal, the agent knows two things:

• When the user started speaking.
• How long the user has been silent.

Note: VAD only tells us whether there is speech. It does not tell us whether the user has finished. That decision is what the rest of this blog is about.

Now, let's see the approaches one by one.

Approach 1: A fixed silence timer

The simplest approach is a timer.

The rule is: "If the user is silent for more than X milliseconds, the turn is over."

Let's say we pick 500 milliseconds, which is half a second.

User: "It is the 14th of..."
       [silence starts]
       [500 ms passes]
Agent: decides the turn is over and starts replying

This works well for short and quick answers like "Yes", "No", or "Book it".

People who are thinking often pause for more than half a second. In our date of birth example, the user paused for 2 seconds. The agent cut in after half a second.

The issue with this approach is that it interrupts the user every time they stop to think. Let's see how the next approach solve this issue.

Approach 2: A longer silence timer

If half a second is too short, let's make the timer longer.

Let's say we pick 3 seconds.

User: "It is the 14th of..."
       [silence for 2 seconds]
User: "...March, 1990."
       [silence starts]
       [3 seconds pass]
Agent: starts replying

Now, the user is not interrupted during the thinking pause.

But this creates a new problem. Now, every single reply is slow. When the user says just "Yes", the agent still waits for 3 full seconds before it replies. The conversation feels like talking to someone on a very bad phone line.

The issue with this approach is that one fixed number cannot be right for every situation. A short timer interrupts thinkers, a long timer makes everyone wait, and a pause longer than 3 seconds still gets cut off. Let's see how the next approach solve this issue.

Approach 3: Looking at the words

Humans do not decide only by the length of the silence. We also listen to what was said.

If someone says "It is the 14th of" and stops, we know they are not done. The sentence is clearly incomplete. "of" needs something after it.

So, the voice agent can do the same. Here comes the semantic turn detection into the picture.

The agent already converts the user's speech into text using Speech-to-Text, also called ASR (Automatic Speech Recognition). It then gives this text to a small language model that is trained to answer one question: "Given what the user has said so far, how likely is it that they have finished their turn?" The model is kept small so that it answers very quickly on every pause.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

The model gives back a score between 0 and 1. Let's see a few examples:

Note: These scores are only for the sake of understanding. A real model gives its own numbers.

Here, we can notice that this model catches many thinking signals:

• Incomplete sentences: "for", "of", "and", "but" at the end mean more words are coming.
• Filler words: "um", "uh", "hmm", and "so" often mean the user is still thinking.
• Thinking phrases: "let me think", "give me a second", "let me check" are direct signals.

The model also looks at the conversation so far. If the agent asked "What is your date of birth?", then "14th of March, 1990" is a complete answer, and "14th of March" alone is less likely to be complete, because the year is still missing.

But the user can say a complete sentence and still not be done:

"I want to book a table for two." [pause] "Actually, make it four."

The first part is a full sentence, so the words alone say "finished". But a human listener hears that the user said "two" slowly, with a flat voice, and knows that the user is still deciding. The issue with this approach is that the text loses how the words were spoken. Let's see how the next approach solve this issue.

Approach 4: Listening to how the words are said

When we speak, our voice carries signals beyond the words. This is called prosody. Prosody means the pitch, the speed, the loudness, and the length of the sounds in our speech.

The best way to learn this is by taking an example. Let's say the word "Okay" in two ways:

• "Okay." with the pitch going down at the end. This sounds finished.
• "Okay..." with the pitch staying flat or going up, and the sound stretched out. This sounds like more is coming.

A few common prosody signals:

• Falling pitch at the end often means the user is done.
• Flat or slightly rising pitch in the middle of a statement often means the user is going to continue. Many questions in English also end with a rising pitch, so the model must learn the difference from many examples.
• Stretched sounds, like "sooooo" or "and uhhhh", often mean the user is thinking.

So, some turn detection models listen to the raw audio directly, not just the text. They look at the last few seconds of the user's audio and give a score, just like the text model. They learn these voice patterns from many hours of recorded human speech.

Now, the agent has three sources of information:

• Silence length from the VAD.
• Meaning of the words from the text model.
• Tone of the voice from the audio model.

Good systems combine them. A long silence after "for" with a flat pitch is very likely a thinking pause. A short silence after "Yes." with a falling pitch is very likely the end of the turn.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Changing the waiting time dynamically

Now, the question is: how does the agent turn all this information into an action?

The answer is a dynamic waiting time. Instead of one fixed timer, the agent picks how long to wait based on how sure it is.

The agent keeps two limits:

• Minimum wait: the shortest time the agent waits, even when it is very sure the user is done. This gives the user a small chance to continue, and gives the Speech-to-Text time to finish the last words.
• Maximum wait: the longest time the agent waits, even when it thinks the user is still thinking. This avoids waiting forever.

Then, it uses the score from the models to pick how long to wait within these two limits like below:

If the model is very sure the user is done:
    wait close to the minimum (fast reply)

If the model thinks the user is still thinking:
    wait close to the maximum (patient)

Let's see this with our two examples. Assume that the minimum wait is 0.5 seconds and the maximum wait is 4 seconds, just for the sake of example.

Example 1: The user says "Yes." with a falling pitch. The model is very sure the turn is over. The agent waits 0.5 seconds and replies.

Example 2: The user says "It is the 14th of..." and stops. The model is very sure the user is not done. The agent stretches its wait toward 4 seconds. The user says "...March, 1990." within 2 seconds, so the agent does not interrupt. Now, the sentence is complete, so the agent waits close to 0.5 seconds and replies.

This is how the agent is fast when the user is clearly done, and patient when the user is clearly thinking.

One more thing to notice. While waiting, the agent keeps checking. If the user speaks again, the timer resets, and the new words are added to the same turn.

If we want to go deep into how to Design a Real-Time Voice AI Agent, including SLMs and Model Distillation for fast turn detection models, we can check out our AI and Machine Learning Program at Outcome School.

What if the agent starts speaking too early?

No detection is perfect. Sometimes the agent still guesses wrong and starts replying while the user was only thinking.

So, a good voice agent also handles this mistake. This is called barge-in or interruption handling.

The flow is as below:

Agent starts speaking
        ↓
VAD detects the user speaking again
        ↓
Agent stops its own speech immediately
        ↓
Agent throws away the rest of its reply
        ↓
Agent listens to the rest of the user's turn

Here, we can see that the agent lets the user continue. The user's new words are joined with the earlier words, and the agent creates a fresh reply based on the full sentence. The agent also keeps in its conversation history only the part of its reply that the user actually heard, so that it does not assume the user heard the rest.

Note: The agent must not treat every small sound as an interruption. A cough, a short "mm-hmm", or background noise must not stop the agent. So, many systems wait for real words or a minimum length of speech before they stop.

Some agents also save time by preparing a reply early. They start creating the answer during the pause, but they do not speak it until the turn is confirmed. If the user continues, they throw that draft away and create a new one. The cost is some wasted work every time the user keeps talking.

What if the user stays silent for a very long time?

Sometimes the user pauses for much longer. They can be searching for a credit card, reading a long ID number, or talking to someone else in the room.

If the silence goes past the maximum wait and the words still look incomplete, the agent must not just reply to half a sentence. It also must not stay silent forever.

So, the agent uses a gentle check-in. Let's see the steps:

Step 1: After a long silence, the agent says something short and kind, like "Take your time." or "No problem, I am here when you are ready."

Step 2: If the silence continues, the agent asks, "Are you still there?"

Step 3: If there is still no answer after a few tries, the agent politely ends the call or offers to call back.

The user can also ask for time directly. If the user says "Hold on, let me find my card", the agent treats this as a request to wait. It replies with "Sure, take your time.", and our code raises the silence limit for a while, so the agent waits much longer than usual before checking in.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

The complete flow

Now, let's put everything together in one diagram.

User is speaking
        ↓
VAD detects silence
        ↓
Text model and audio model check: "Is the user done?"
        ↓
Agent picks a waiting time between the minimum and the maximum
        ↓
Does the user speak again during the wait?
  Yes: add the new words to the same turn and keep listening
  No:  move to the next check
        ↓
Do the words and the voice say the turn is complete?
  Yes: the agent replies (and stops if the user cuts in)
  No:  the agent checks in gently ("Take your time.")

Where it works well and where it fails

It works well when:

• The user leaves clear signals like filler words, thinking phrases, incomplete sentences, or a flat pitch.
• The agent asked a clear question, so the model knows what a complete answer looks like.

It fails when:

• The user pauses after a complete-sounding sentence and then adds more. The words and the tone both say "done", so the agent cuts in.
• The background is noisy, so the VAD can mistake the noise for speech and never see the silence.
• The user speaks a language or accent that the turn detection model was not trained on, so its signals are less reliable.
• The user is reading out a long number, like a card number, with long gaps between groups of digits. Each group can look like a complete answer.

For the last case, some agents use a special rule: when they ask for a number, they wait longer between digit groups, or they wait until they have the expected count of digits.

Now we must have understood how a voice agent handles long pauses when the user is thinking, and how the words, the voice, and a flexible waiting time work together.

Frequently Asked Questions

How long should a voice agent wait before replying?

There is no single right number. A short fixed timer interrupts users who stop to think, and a long fixed timer makes every reply slow. A good voice agent picks its waiting time between a minimum wait and a maximum wait, based on how sure its turn detection models are that the user is done.

Can Voice Activity Detection (VAD) alone tell when the user has finished speaking?

No. VAD only tells us whether a human is speaking in each small piece of audio. It does not tell us whether the user has finished their turn. For that, the voice agent also looks at the meaning of the words and the tone of the voice.

Do filler words like "um" and "uh" help a voice agent detect a thinking pause?

Yes. Filler words like "um", "uh", "hmm", and "so" often mean the user is still thinking. The turn detection model gives a low score to text that ends with them, so the voice agent waits longer instead of cutting in.

What should a voice agent do when the user says "hold on"?

It should treat it as a request to wait. The agent replies with something short like "Sure, take your time.", and the silence limit is raised for a while. This way, the agent waits much longer than usual before it checks in with the user.

Why is reading out a card number hard for a voice agent?

Because the user leaves long gaps between groups of digits, and each group can look like a complete answer. To handle this, some agents wait longer between digit groups when they ask for a number, or they wait until they have the expected count of digits.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.