How does endpointing decide when the user has finished speaking?

Endpointing is the part of a voice assistant that decides when the user has finished speaking, so that the assistant knows when to stop listening and start replying. It does this by watching for silence, listening to how the user speaks, and reading what the user has said so far.

In this blog, we will learn about how endpointing decides when the user has finished speaking. We will also see why this decision is so hard, how a simple silence timer works, how voice activity detection finds speech and silence, how the words and the tone of the voice help, how all these signals come together, and how a real voice assistant balances speed with not cutting off the user.

We will cover the following:

• What is Endpointing?
• Why is Endpointing Hard?
• Approach 1: Fixed Silence Timer
• How Voice Activity Detection Works
• Approach 2: Reading the Words
• Approach 3: Listening to the Voice
• Approach 4: Combining All the Signals
• Waiting Too Long vs Cutting Off Too Early
• Extra Steps Around the Endpoint

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is Endpointing?

Endpointing is the process of detecting the point in time where the user has finished their turn of speaking.

A voice assistant listens to us through the microphone. At some moment, it has to say to itself: "The user is done. Now it is my turn to speak."

That moment is the endpoint. Endpointing is also called end-of-turn detection or turn detection.

Endpointing is one part of a bigger job called turn-taking. We have a detailed blog on turn-taking in a voice AI agent that explains how the agent decides whose turn it is to speak.

Let's say we ask a voice assistant: "What is the weather in Delhi today?"

The flow looks like below:

User speaks: "What is the weather in Delhi today?"
                       ↓
Endpointing decides: the user has finished
                       ↓
Speech recognition gives the final text
                       ↓
The AI model writes the answer
                       ↓
Text-to-speech speaks the answer

Here, we can see that the assistant normally does not start its reply until the endpoint is decided. If endpointing is slow, the whole reply is slow. If endpointing is wrong, the assistant replies to half a sentence.

Why is Endpointing Hard?

When we talk to another person, we usually know when they have finished, often even before they stop. Studies of human conversation show that the gap between two speakers is often only around 200 milliseconds. For a machine, matching this is not easy.

The main problem is that silence does not always mean the end.

Humans pause in the middle of a sentence all the time. Let's say a user says:

"I want to book a flight to... umm... Mumbai."

Here, there is a long pause after "to". If the assistant treats this pause as the end, it will reply to "I want to book a flight to", which is an incomplete request. The user gets cut off.

Now, consider the opposite case. The user says:

"Yes."

And then stops. Here, the user has clearly finished. If the assistant keeps waiting for more, the user feels that the assistant is slow.

So, endpointing has to solve two problems at the same time:

• Do not cut off the user in the middle of a thought.
• Do not make the user wait after they have finished.

These two goals pull in opposite directions. The best way to learn how endpointing handles this is by looking at the approaches one by one, from the simplest to the most advanced.

Approach 1: Fixed Silence Timer

The simplest way to do endpointing is to wait for silence.

The rule is like below:

If the user has been silent for 700 milliseconds,
then the user has finished speaking.

Here, 700 milliseconds is just an example value for the sake of understanding. Real systems choose their own value, and it is often somewhere between a few hundred milliseconds and around one second.

Let's see how it works step by step for "What is the weather in Delhi today?":

Step 1: The user starts speaking. The timer is not running.

Step 2: The user says "today" and stops. The silence timer starts at 0 milliseconds.

Step 3: The silence continues, and the timer reaches 700 milliseconds. The assistant decides that the user has finished.

If the user starts speaking again before the timer reaches 700 milliseconds, the timer resets to 0.

Advantage:

• It is simple to build and fast to run.

Disadvantage:

• One fixed number cannot fit every situation. A short value cuts off users who pause to think. A long value makes every reply feel slow, even for a simple "Yes."
• Every reply is delayed by at least the full wait time, because the assistant must wait 700 milliseconds even when the user has clearly finished.

The issue with this approach is that it only listens to silence, and silence alone does not tell us whether the user is done. Let's see how the next approach solve this issue.

But before jumping into the next approach, we must know how the machine even knows whether there is silence or speech. Here comes the Voice Activity Detection into the picture.

How Voice Activity Detection Works

Voice Activity Detection (VAD) is a small detector, today usually a small neural network, that listens to the audio and tells us whether a human is speaking or not at each moment.

The microphone gives us a continuous stream of audio. VAD cuts this stream into small pieces called frames. A frame is usually a few tens of milliseconds long, for example 20 or 30 milliseconds.

For each frame, VAD gives a score between 0 and 1:

• A score close to 1 means "someone is speaking".
• A score close to 0 means "no one is speaking".

The system picks a threshold, for example 0.5. A score above it counts as speech, and a score below it counts as silence.

The output looks like below:

Frame   Score   Label
1       0.95    speech
2       0.97    speech
3       0.92    speech
4       0.10    silence
5       0.05    silence
6       0.04    silence

Here, we can see that the user was speaking in frames 1 to 3, and then the silence started at frame 4.

Note: Silence here does not mean zero sound. A room almost always has some noise, like a fan or traffic. A good VAD is trained to tell the difference between human speech and background noise. Simple older methods only checked how loud the audio was, which made them fail often in noisy rooms.

Now, the silence timer from Approach 1 counts how long the VAD has been saying "silence". With 30 millisecond frames, 700 milliseconds of silence is about 23 silent frames in a row. This is how VAD and the silence timer work together.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

But silence alone is still not enough. Let's see how the meaning of the words helps.

Approach 2: Reading the Words

When we listen to someone, we use the meaning of their words to know if they are done. Let's take two sentences:

• "What is the weather in Delhi today?"
• "What is the weather in"

Even without hearing any pause, we can tell that the first sentence is complete and the second is not. The second one is clearly waiting for a place name.

A voice assistant can do the same thing. While the user is speaking, the speech recognition model is already turning the audio into text, piece by piece. This is called streaming speech recognition. So, at any moment, the assistant has the text of what the user has said so far.

Now, we can give this text to a small model that answers one question:

"Is this sentence complete, or is the user going to say more?"

This model is often called a turn detector or an end-of-turn model. It is trained on many real conversations, where each piece of text is labeled as "finished" or "not finished". It gives a probability. The numbers below are only for the sake of understanding:

Here, we can see that the model gives a low probability to the incomplete sentences and a high probability to the complete ones.

Many turn detectors also read the last few messages of the conversation, not just the current sentence. Let's say the assistant asked: "What is your phone number?" The user says "98765" and pauses. With the question in front of it, the model can tell that a phone number usually has more digits, so the user is not done yet.

Now, the assistant can change how long it waits based on this probability:

• If the probability is high, wait a short time, for example 200 milliseconds.
• If the probability is low, wait much longer, for example 2 seconds.

Let's go back to "I want to book a flight to... umm... Mumbai." When the user pauses after "to", the turn detector says the sentence is incomplete. So, the assistant waits longer. The user gets time to think and says "Mumbai". Now, the sentence is complete, and the assistant replies quickly.

Advantage:

• It understands when a sentence is clearly unfinished, so it cuts off users less often.

Disadvantage:

• It depends on the text from speech recognition. If the text is wrong or arrives late, the decision also becomes wrong or late.
• Text loses the way something was said. "You are coming." and "You are coming?" look the same when speech recognition gives text without punctuation, but they sound very different.

The issue with this approach is that it ignores the voice itself. Let's see how the next approach solve this issue.

Approach 3: Listening to the Voice

When humans finish a sentence, the voice itself gives clues. These clues are called prosody. Prosody is the melody and rhythm of speech. It covers how the voice goes up and down, how fast we speak, and how we stretch the sounds.

Some common clues are:

• Falling pitch: At the end of a statement, the voice usually goes down. "I am going home." ends low.
• Rising pitch: At the end of a question, the voice often goes up. "Are you going home?" ends high.
• Slowing down: People often stretch the last word a little when they finish.
• Filler sounds: Sounds like "umm" and "uhh" usually mean the user is still thinking and is not done. Many speech recognition models drop these sounds from the text, so only the audio keeps them.
• Breathing in: A quick breath often means the user is about to continue.

An audio-based turn detector listens directly to the audio, not the text. It learns these clues from many recorded conversations, where each moment is labeled as "the turn ended here" or "the turn did not end here".

Let's say the user says "I need a taxi" and then pauses, because they also want to add "and a hotel room". The text "I need a taxi" looks like a complete request. But the voice stayed level at the end instead of falling, and this tells the audio model that the user is likely to continue.

Advantage:

• It catches clues that text cannot capture, like the tone of a question or a thinking pause.
• It does not have to wait for speech recognition to finish.

Disadvantage:

• Prosody changes from person to person, from language to language, and from accent to accent. A model trained on one group of speakers can do worse on others.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Now, it's time to learn how a real system uses all three signals together.

Approach 4: Combining All the Signals

Each signal is good at something and weak at something:

So, many modern voice assistants combine them. A common way to do this is like below:

Audio from microphone
          ↓
VAD: is the user speaking or silent?
          ↓
Silence detected, timer starts
          ↓
Turn detector checks words, voice, or both
          ↓
Probability that the user is done
          ↓
High probability: short wait
Low probability: long wait
          ↓
Timer reaches the wait time
          ↓
Endpoint decided, assistant replies

Here, we can notice a few things:

• VAD is always running because it is cheap. It decides when to ask the turn detector.
• In many systems, the turn detector runs only when silence starts, so the heavier model does not run all the time.
• The silence timer still exists, but its length is no longer fixed. It changes based on how confident the turn detector is.
• If the user speaks again during the wait, VAD detects speech and the timer resets.
• There is usually a maximum wait time. Even if the turn detector says "not done", the assistant does not wait forever. If the user stays silent for a long time, the assistant takes its turn.

Let's walk through one example from start to end. The user says: "Can you set an alarm for... umm... seven in the morning?"

Step 1: The user says "Can you set an alarm for... umm..." and pauses. VAD detects silence. The timer starts.

Step 2: The turn detector reads "Can you set an alarm for" and hears the "umm". It gives a low probability. So, the wait time becomes long.

Step 3: Before the long wait ends, the user starts speaking again. VAD detects speech, and the timer resets. The user says "seven in the morning".

Step 4: The user stops. VAD detects silence. The timer starts again.

Step 5: The turn detector reads the full question and hears the rising pitch at the end of the question. It gives a high probability. So, the wait time becomes short.

Step 6: The short wait ends. The endpoint is decided. The assistant replies: "Sure, I have set an alarm for 7 AM."

The user was not cut off at the pause, and the reply came quickly at the real end.

To learn how VAD, turn detection, and the rest of the pipeline fit together when we Design a Real-Time Voice AI Agent, check out our AI and Machine Learning Program at Outcome School.

Waiting Too Long vs Cutting Off Too Early

Every endpointing system has to balance two kinds of mistakes:

• Early endpoint (cutting off): The assistant thinks the user is done, but the user was only pausing. The assistant replies to half a request. This annoys the user and often gives wrong answers.
• Late endpoint (waiting too long): The user is done, but the assistant keeps waiting. This feels slow and unnatural.

If we make the wait shorter, we get fewer late endpoints but more early endpoints. If we make the wait longer, we get the opposite. This is a trade-off.

We have a detailed blog on how to build a latency budget for a voice AI agent that covers how this wait fits with every other stage of the reply.

So, teams measure both:

• Latency: How much time passes between the user really finishing and the assistant deciding that the user has finished.
• Cut-off rate: How often the assistant decides too early and interrupts the user.

Then, they tune the wait times and the probability thresholds based on their use case. For example, a voice assistant for quick commands like "Turn off the lights" can use shorter waits. A voice assistant for customer support, where users explain long problems, needs longer waits.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Extra Steps Around the Endpoint

Even with the best endpointing, mistakes will happen. So, voice assistants add a few extra steps around the endpoint.

The user starts speaking again. Let's say the assistant decided the endpoint too early, and the user continues speaking. A good assistant notices that VAD has detected speech again, stops its reply, and keeps listening to the rest of the request. This is called barge-in, which means the user can interrupt the assistant at any time.

Starting the work early. Some systems start preparing the answer at the first sign of an endpoint, before they are fully sure. If the user continues, the system throws away that work, which costs extra computing. If the user is really done, the answer is already partly ready, and this hides some of the waiting time from the user.

Now we must have understood how endpointing decides when the user has finished speaking, why it is hard, and how a real voice assistant balances speed with not cutting off the user.

Frequently Asked Questions

Is endpointing the same as voice activity detection?

No. Voice activity detection only tells us whether a human is speaking in each small frame of audio. Endpointing uses that silence signal, and often the words and the voice too, to decide that the user has finished their turn. A pause that VAD marks as silence is not always the end of the turn.

How long should a voice assistant wait in silence before replying?

There is no single right value. A fixed silence timer is often set somewhere between a few hundred milliseconds and around one second. With a turn detector, the wait changes: a short wait, for example 200 milliseconds, when the user is likely done, and a much longer one, for example 2 seconds, when the sentence looks unfinished.

What happens if the assistant decides the endpoint too early?

The assistant replies to half a request, which annoys the user and often gives wrong answers. A good assistant handles this with barge-in. When VAD detects that the user has started speaking again, the assistant stops its reply and keeps listening to the rest of the request.

Why do filler sounds like "umm" matter for endpointing?

Filler sounds like "umm" and "uhh" usually mean that the user is still thinking and is not done. Many speech recognition models drop these sounds from the text, so a text-based turn detector cannot see them. An audio-based turn detector listens to the voice directly, so it can still hear them.

Does the turn detector run all the time?

Not in many systems. VAD is always running because it is cheap, and the turn detector often runs only when silence starts, so the heavier model does not run all the time. There is also usually a maximum wait time, so the assistant takes its turn if the user stays silent for a long time.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.