How does dynamic endpointing adapt to how fast a user talks?

Dynamic endpointing adapts to how fast a user talks by learning the length of that user's normal pauses and then changing how long the voice assistant waits in silence before it decides that the user has finished speaking. A fast talker gets a short wait, and a slow talker gets a long wait.

In this blog, we will learn about how dynamic endpointing adapts to how fast a user talks. We will also see what endpointing is, why a fixed silence timer fails for fast and slow talkers, how the system learns the pauses of a user and turns them into a waiting time, how the meaning of the sentence changes that waiting time, and where it works well and where it fails.

We will cover the following:

• What is endpointing?
• The problem with a fixed silence timer
• What is dynamic endpointing?
• How does the system measure how fast a user talks?
• Turning pauses into a waiting time
• A simple dynamic endpointer in code
• A step-by-step walkthrough
• Using the meaning of the sentence as well
• Where it works well and where it fails

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is endpointing?

When we talk to a voice assistant, the assistant has to answer one question again and again: "Has the user finished speaking, or is the user only taking a short break?"

Endpointing is the process of deciding the point in time where the user has finished speaking, so that the voice assistant can start its reply.

Once the system finds this end point, it sends what the user said to the LLM (Large Language Model), and the assistant prepares its reply.

A turn is one complete piece of speech by one speaker before the other speaker replies. So, endpointing means finding the end of the turn of the user.

Let's say we are talking to a voice assistant like below:

"Book a table for two... at 8 PM."

Here, there is a small silence after "two". The assistant has to decide whether "Book a table for two" is the full request or whether more words are coming. If it decides too early, it cuts us off. If it decides too late, we sit in silence waiting for a reply.

Endpointing is a balance between two mistakes:

• Cutting the user off: The assistant replies while the user is still in the middle of a sentence.
• Replying too slowly: The user has finished, but the assistant keeps waiting, and the conversation feels slow.

We have a detailed blog on how to build a latency budget for a voice AI agent that explains how this wait fits with every other stage of the reply.

The problem with a fixed silence timer

The simplest way to do endpointing is a fixed silence timer.

The system uses a component called a Voice Activity Detector (VAD). A VAD listens to the audio and tells us, for every small slice of time, whether someone is speaking or not. When the VAD says "no speech", a timer starts. If the silence lasts longer than a fixed limit, for example, 800 milliseconds, the system decides that the user has finished.

But the following issues arise because people do not talk at the same speed.

Suppose we have two users, with numbers just for the sake of example:

• User A is a fast talker. The normal pauses between the words of User A are around 200 milliseconds.
• User B is a slow talker. The normal pauses between the words of User B are around 700 milliseconds, and sometimes 900 milliseconds when User B is thinking.

Now, let's apply the same fixed limit of 800 milliseconds to both users.

For User A, the assistant waits 800 milliseconds after every request, even though a silence of 400 milliseconds is already very long for User A. The assistant feels slow to User A.

For User B, a normal thinking pause of 900 milliseconds crosses the limit of 800 milliseconds. The assistant decides that User B has finished and replies in the middle of the sentence. User B gets cut off.

So, one fixed number cannot be right for both users. If we lower the limit, User B gets cut off more often. If we raise the limit, User A waits even longer. The issue with this approach is that it treats every user in the same way. Let's see how the next approach solve this issue.

What is dynamic endpointing?

Here comes the dynamic endpointing into the picture.

Dynamic endpointing is a way of doing endpointing where the waiting time is not fixed. The system changes the waiting time based on how the current user is speaking and what the user has said so far.

The word dynamic means changing. In fixed endpointing, the waiting time is the same for everyone. In dynamic endpointing, the waiting time changes from user to user, and even from moment to moment for the same user.

The central idea is this: a silence is "long" or "short" only when compared with the normal pauses of that user. For User A, a silence of 500 milliseconds is very long. For User B, the same silence of 500 milliseconds is a normal pause between two words.

So, instead of asking "Has the silence crossed 800 milliseconds?", the system asks "Has the silence become much longer than the normal pauses of this user?"

We will learn about the speed part first, and then the meaning part.

How does the system measure how fast a user talks?

Before jumping into the waiting time, we must know how the system measures the speed of a user.

There are two common signals:

• Pause length: The length of the small silences between words while the user is still speaking.
• Speaking rate: How many words or syllables the user says in one second.

Both signals are connected. A fast talker says more words per second and leaves short pauses. A slow talker says fewer words per second and leaves longer pauses.

In this blog, we will focus on pause length, because it connects directly to the silence timer. The silence timer measures silence, and the pause length is also silence. So, we compare silence with silence. Speaking rate can be used in the same way: a higher speaking rate gives a shorter waiting time, and a lower speaking rate gives a longer waiting time.

The system collects pauses like below:

• The VAD marks every slice of the audio as speech or no speech.
• Every time the user stops for a short moment and then continues speaking, the system notes the length of that silence.
• These are the in-sentence pauses. They are the pauses that did not end the turn, because the user continued speaking after them.

Here, we can notice that these pauses show what a normal "I am not done yet" pause looks like for this user.

The system keeps only the most recent pauses, for example, the last 5 pauses. This way, if the user speeds up or slows down during the conversation, the system follows that change.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Turning pauses into a waiting time

Now, we have a list of recent pauses for the user. Next, we need to turn this list into a waiting time.

We can do it in three steps.

Step 1: We find the average of the recent pauses. This is the normal pause of the user.

Step 2: We multiply the average by a number, for example, 2. This gives the waiting time. The idea is that the user has finished only when the silence is clearly longer than a normal pause, here, two times longer.

Step 3: We keep the waiting time inside a minimum and a maximum limit, for example, between 300 milliseconds and 2000 milliseconds.

The formula is as below:

waiting_time = clamp(average_pause * 2, 300, 2000)

Here, clamp means "keep the value inside the limits". If the value is below 300, we use 300. If the value is above 2000, we use 2000. Otherwise, we use the value itself.

Why do we need these limits?

• The minimum limit protects us from replying too fast. Even for a very fast talker, a silence of 100 milliseconds is too short to decide.
• The maximum limit protects us from waiting forever. Even for a very slow talker, the assistant must reply at some point.

Instead of a simple average, some systems use a moving average that gives more weight to the most recent pauses, and some set the waiting time a little above the longest recent pause. The idea stays the same: the waiting time grows and shrinks with the normal pauses of the user.

There is one more question. What happens at the very start of the conversation, when the system has not seen any pause yet? The system starts with a default waiting time, for example, 800 milliseconds, and switches to the learned waiting time once it has collected a few pauses.

A simple dynamic endpointer in code

Let's see the code for a simple dynamic endpointer in Python. This is just for the sake of understanding, so it is kept small.

We can write the code as below:

from collections import deque

class DynamicEndpointer:
    def __init__(self):
        self.pauses = deque(maxlen=5)  # keep only the last 5 pauses
        self.default_wait = 800        # milliseconds
        self.multiplier = 2
        self.min_wait = 300
        self.max_wait = 2000

    def add_pause(self, pause_ms):
        # called when the user paused and then continued speaking
        self.pauses.append(pause_ms)

    def waiting_time(self):
        if len(self.pauses) < 3:
            return self.default_wait  # not enough data yet
        average = sum(self.pauses) / len(self.pauses)
        wait = average * self.multiplier
        return max(self.min_wait, min(wait, self.max_wait))

    def is_turn_over(self, silence_ms):
        return silence_ms >= self.waiting_time()

Here, we have:

• pauses keeps the last 5 in-sentence pauses. When a sixth pause comes, the oldest one is removed automatically.
• add_pause is called every time the user stops for a short moment and then continues speaking.
• waiting_time returns the default of 800 milliseconds until we have at least 3 pauses. After that, it takes the average, multiplies it by 2, and keeps it between 300 and 2000.
• is_turn_over is called while the user is silent. It returns True when the current silence reaches the waiting time.

Now, let's use it for User A and User B.

user_a = DynamicEndpointer()
for pause in [150, 200, 250]:
    user_a.add_pause(pause)
print(user_a.waiting_time())

user_b = DynamicEndpointer()
for pause in [500, 700, 900]:
    user_b.add_pause(pause)
print(user_b.waiting_time())

It will print the following:

400.0
1400.0

Here, we can see that the fast talker, User A, gets a waiting time of 400 milliseconds, and the slow talker, User B, gets a waiting time of 1400 milliseconds. The same code gives two different waiting times because the two users talk at different speeds.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

A step-by-step walkthrough

Now, let's see the values at each step.

For User A, the pauses are 150, 200, and 250 milliseconds.

Step 1: The average is (150 + 200 + 250) / 3 = 600 / 3 = 200 milliseconds.

Step 2: We multiply by 2. So, 200 * 2 = 400 milliseconds.

Step 3: 400 is between 300 and 2000, so the waiting time stays 400 milliseconds.

For User B, the pauses are 500, 700, and 900 milliseconds.

Step 1: The average is (500 + 700 + 900) / 3 = 2100 / 3 = 700 milliseconds.

Step 2: We multiply by 2. So, 700 * 2 = 1400 milliseconds.

Step 3: 1400 is between 300 and 2000, so the waiting time stays 1400 milliseconds.

Now, let's see what happens when User B takes a thinking pause of 900 milliseconds. With the fixed timer, 900 is more than 800, so User B gets cut off. With dynamic endpointing, 900 is less than 1400, so the assistant keeps listening.

Let me tabulate the difference between the fixed timer and dynamic endpointing for both users for your better understanding.

The flow of the whole system looks like the diagram below:

+-----------------------------+
|  User speaks into the mic   |
+-----------------------------+
              ↓
+-----------------------------+
|  VAD marks speech / silence |
+-----------------------------+
              ↓
+-----------------------------+
|  Store in-sentence pauses   |
|  (last 5 pauses)            |
+-----------------------------+
              ↓
+-----------------------------+
|  wait = clamp(avg * 2,      |
|         300, 2000)          |
+-----------------------------+
              ↓
+-----------------------------+
|  Silence >= wait?           |
|  Yes: turn is over, reply   |
|  No: keep listening         |
+-----------------------------+

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Using the meaning of the sentence as well

So far, our system uses only the speed of the user. But speed alone is not enough.

Consider these two sentences, spoken by the same user with the same pause at the end:

• "What is the weather in Delhi?"
• "What is the weather in..."

The first sentence is complete. The second sentence is not complete. A good assistant must reply quickly to the first one and keep waiting for the second one, even if the silence is the same.

So, many systems also look at the words. A small model looks at what the user has said so far and gives a score: how likely it is that the user has finished the thought. This is often called a turn detection model or an end-of-turn model.

The system then combines both signals:

• If the sentence looks complete, the system lowers the waiting time.
• If the sentence looks incomplete, for example, it ends with "and", "to", "the", or "in", the system raises the waiting time.

We can add this to our formula, just for the sake of understanding, like below:

• Sentence looks complete: waiting_time = clamp(average_pause * 1.5, 300, 2000)
• Sentence looks incomplete: waiting_time = clamp(average_pause * 3, 300, 2000)

When the model is not sure, the system uses the normal multiplier of 2.

For User A, with an average pause of 200 milliseconds, a complete sentence gives 200 * 1.5 = 300 milliseconds, and an incomplete sentence gives 200 * 3 = 600 milliseconds.

For User B, with an average pause of 700 milliseconds, an incomplete sentence gives 700 * 3 = 2100 milliseconds. This is above the maximum limit, so the waiting time becomes 2000 milliseconds.

Some systems also listen to how the user says the words. The pattern of pitch, loudness, and rhythm in the voice is called prosody. For example, the pitch of the voice often falls at the end of a statement, and it often stays level when the user is going to continue. These sound patterns give one more hint about whether the user has finished.

This is how dynamic endpointing combines the speed of the user and the meaning of the words to decide the right waiting time.

To learn Multimodal AI, SLMs and Model Distillation, and how to Design a Real-Time Voice AI Agent, check out our AI and Machine Learning Program at Outcome School.

Where it works well and where it fails

Dynamic endpointing works well in the following cases:

• Fast talkers: They get quick replies, and the conversation feels natural.
• Slow talkers: People speaking in a second language and people who think while they speak get cut off less often.
• A change in speed during the conversation: Because the system keeps only the recent pauses, it follows the user when the user speeds up or slows down.

But it also fails in some cases:

• The start of the conversation: The system has no pauses yet, so it uses the default waiting time until it learns the pauses of the user.
• Noisy rooms: If the VAD treats background noise as speech, the pause lengths become wrong, and so the waiting time also becomes wrong.
• One very long thinking pause: A user who normally talks fast can suddenly stop for two seconds to think. The learned waiting time is short, so the assistant replies too early. The meaning of the sentence helps here, but it does not fix every case.

So, dynamic endpointing does not remove every mistake. But it reduces both kinds of mistakes for users who talk faster or slower than average, because it waits for each user in the way that user talks.

Now we must have understood how dynamic endpointing adapts to how fast a user talks, how it learns the pauses of a user, how it turns them into a waiting time, and how the meaning of the sentence improves that waiting time.

Frequently Asked Questions

What does dynamic endpointing do at the start of a conversation?

It uses a default waiting time, for example, 800 milliseconds. At the very start of the conversation, the system has not seen any pause yet, so it cannot learn a waiting time. Once it has collected a few in-sentence pauses, for example at least 3, it switches to the waiting time learned from the pauses of that user.

Why does dynamic endpointing keep only the most recent pauses?

Dynamic endpointing keeps only the most recent pauses, for example the last 5, so that it can follow the user when the user speeds up or slows down during the conversation. Older pauses are dropped, and the waiting time grows and shrinks with how the user is talking right now.

Can dynamic endpointing use speaking rate instead of pause length?

Yes. Speaking rate, which is how many words or syllables the user says in one second, can be used in the same way. A higher speaking rate gives a shorter waiting time, and a lower speaking rate gives a longer waiting time. Pause length connects directly to the silence timer, because both measure silence, so we compare silence with silence.

Does background noise affect dynamic endpointing?

Yes. If the Voice Activity Detector treats background noise as speech, the measured pause lengths become wrong, and so the waiting time also becomes wrong. This is one of the cases where dynamic endpointing fails, along with the start of the conversation and one very long thinking pause from a user who normally talks fast.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.