How do silence-based and model-based turn detection differ?

Silence-based turn detection decides that a person has finished speaking when it hears a long enough pause. Model-based turn detection uses a trained model that looks at what was said and how it was said to decide whether the person is done.

In this blog, we will learn about how silence-based and model-based turn detection differ. We will also see what turn detection is, how silence-based turn detection works and why a fixed pause fails, how a model predicts the end of a turn from words and from sound, how both are combined in a voice pipeline, and when to use which one.

We will cover the following:

• What is Turn Detection?
• How Silence-Based Turn Detection Works
• The Problems with Silence-Based Turn Detection
• How Model-Based Turn Detection Works
• Text-Based and Audio-Based Turn Detection Models
• How Both Work Together in a Voice Pipeline
• Silence-Based vs Model-Based Turn Detection
• When to Use Which One

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is Turn Detection?

When two people talk, they take turns. One person speaks, then stops, and then the other person replies. We do this without thinking. We just know when the other person has finished.

A voice assistant has to do the same thing. It must decide when the user has finished speaking, so that it can start replying.

Turn detection is the job of deciding when the user has finished their turn in a conversation, so that the voice assistant knows when to start speaking.

The point where the user finishes is called the end of turn. Hence, turn detection is also called end-of-turn detection or endpointing.

Turn detection is one part of a bigger job called turn-taking. We have a detailed blog on turn-taking in a voice AI agent that covers how the agent decides whose turn it is to speak.

Let's say we are talking to a voice assistant to book a table at a restaurant. We say:

"I want to book a table for four people."

The assistant must decide: is the user done, or is the user going to say more? If it decides too early, it cuts us off. If it decides too late, we sit in silence and wonder if it heard us.

So, turn detection has two ways to fail:

• Too early: The assistant interrupts the user in the middle of a sentence.
• Too late: The assistant waits too long, and the conversation feels slow.

There are two main ways to solve this problem:

• Silence-based turn detection: Wait for a pause of a fixed length.
• Model-based turn detection: Use a trained model to predict whether the user is done.

Let's understand each of them one by one.

How Silence-Based Turn Detection Works

The idea of silence-based turn detection is simple. If the user stops making speech sounds for long enough, we assume the user is done.

In order to know whether the user is speaking or silent, we use a Voice Activity Detector, also called a VAD.

A Voice Activity Detector is a small tool, usually a tiny model, that listens to short pieces of audio and tells us whether each piece contains human speech or not.

The audio from the microphone comes in small pieces, for example, one piece every 30 milliseconds. For each piece, the VAD says either "speech" or "no speech".

Then, we add one rule on top of it:

If the VAD says "no speech" continuously for a fixed amount of time, the user has finished their turn.

This fixed amount of time is called the silence timeout. For the sake of example, let's say the silence timeout is 700 milliseconds.

Let's see this step by step.

Step 1: The user says "I want to book a table". The VAD says "speech" for every piece.

Step 2: The user stops. The VAD starts saying "no speech". A timer starts counting.

Step 3: The timer reaches 200 milliseconds. The user is still silent, so the system keeps waiting.

Step 4: The timer reaches 700 milliseconds. The user is still silent, so the silence timeout is reached.

Step 5: The system declares the end of turn and sends the speech to the next part of the assistant to generate a reply.

If the user starts speaking again before the timer reaches 700 milliseconds, the timer resets to zero, and the turn continues.

We can write the logic in code as below:

SILENCE_TIMEOUT_MS = 700
CHUNK_MS = 30

silence_ms = 0
user_spoke = False

for chunk in microphone_chunks():
    if vad.is_speech(chunk):
        user_spoke = True
        silence_ms = 0  # user is speaking, reset the timer
    elif user_spoke:
        silence_ms += CHUNK_MS
        if silence_ms >= SILENCE_TIMEOUT_MS:
            end_of_turn()  # user is done, start replying
            user_spoke = False
            silence_ms = 0

Here, we have a counter silence_ms that keeps track of how long the user has been silent. Every time the VAD hears speech, we reset the counter to zero. Every time the VAD hears no speech after the user has spoken, we add 30 milliseconds to the counter. When the counter reaches the silence timeout, we call end_of_turn().

The flag user_spoke makes sure that we count silence only after the user has said something. Without it, a quiet room would keep ending empty turns again and again.

This is how silence-based turn detection works.

Advantages:

• It is simple to build.
• It is fast and cheap, because the VAD is a tiny model.
• It works for almost any language, because it does not care about the words.
• It is easy to tune, because there is only one main number to change.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

The Problems with Silence-Based Turn Detection

Silence-based turn detection only looks at one thing: the length of the pause. But humans do not only pause at the end of a turn. We pause in the middle of sentences all the time.

Let's say the user says:

"I want to book a table for... um... four people."

The user paused after "for" to think. If that pause is longer than 700 milliseconds, the assistant decides the user is done. It replies to "I want to book a table for", which is an incomplete sentence. The user gets interrupted.

This is the first problem: the pause length does not tell us whether the sentence is complete.

Now, let's say we increase the silence timeout to 2 seconds to stop the interruptions. Then, every time the user finishes a complete sentence, the assistant waits 2 full seconds before replying. The conversation feels slow.

So, we are stuck with a trade-off:

• Short timeout: Fast replies, but the assistant interrupts users who pause to think.
• Long timeout: Fewer interruptions, but every reply feels slow.

No single number works well for every user and every sentence.

Let's see a few more cases where a fixed pause fails:

• Reading numbers: "My phone number is 98765... 43210." People pause between groups of digits.
• Thinking aloud: "Book it for, hmm, let me check, Friday."
• Short complete answers: The assistant asks "Shall I confirm?" and the user says "Yes." This is clearly complete, but the system still waits for the full timeout.
• Background noise: A noisy room can make the VAD think the user is still speaking, so the end of turn is detected late.

The issue with this approach is that it listens only to silence, not to meaning. Let's see how the next approach solve this issue.

How Model-Based Turn Detection Works

Humans do not decide the end of turn by counting milliseconds. We use clues. We listen to the words. We notice if the sentence sounds complete. We hear if the voice goes down at the end like a full stop, or stays up like the person has more to say.

This is the idea behind model-based turn detection.

Model-based turn detection uses a trained model that looks at the conversation and predicts how likely it is that the user has finished their turn.

Instead of a simple "speech or no speech" answer, the model gives a probability, which is a number between 0 and 1. For the sake of example, let's say:

• "I want to book a table for" gives a probability of 0.05 that the turn is over. It is very likely incomplete.
• "I want to book a table for four people." gives a probability of 0.92 that the turn is over. It is very likely complete.

The model learns this by training on a large number of conversation examples, where each example is labeled as "turn complete" or "turn not complete". During training, it learns the patterns that humans use to signal the end of a turn.

Now, with this probability, the system can make a better decision:

• If the probability is high, the user is likely done. The assistant can reply after a short pause.
• If the probability is low, the user is likely still thinking. The assistant waits longer, even if there is silence.

Let's go back to our example:

"I want to book a table for... um... four people."

At the pause after "for", the model sees an incomplete sentence. The probability of end of turn is low. So, the assistant keeps waiting. When the user says "four people." and stops, the probability is high, and the assistant replies quickly.

The assistant does not interrupt, and it does not feel slow.

Advantages:

• It interrupts users much less when they pause to think.
• It replies faster when the sentence is clearly complete.
• It uses the meaning of the words and the tone of the voice, not just the silence.

Disadvantages:

• It needs an extra model, which uses more computing power.
• It works well only for the languages it was trained on.
• It can be wrong, so we still keep a silence timeout as a backup.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Text-Based and Audio-Based Turn Detection Models

Now, the next big question is: what does the model look at? There are two main types of turn detection models.

Text-Based Turn Detection

A text-based model looks at the words. First, a speech recognition model turns the user's speech into text. This text is called the transcript. Then, the turn detection model, which is often a small language model, reads the transcript, and usually the last few messages of the conversation too, and predicts whether the user's sentence is complete.

Let's say the assistant asked "What is your phone number?" and the user said "98765". A text-based model can use the question to understand that a phone number usually has more digits, so the user is likely not done.

Advantage: It understands meaning and the context of the conversation well.

Disadvantage: It cannot hear the tone of the voice. It also has to wait for the speech recognition model to produce the text, which adds a little delay. Mistakes in the transcript also become mistakes in the turn decision.

Audio-Based Turn Detection

An audio-based model listens directly to the sound of the user's voice. It does not need the text.

From the sound, it can pick up clues like:

• Pitch: Does the voice go down at the end, like a finished statement, or stay up, like an unfinished thought?
• Speed: Does the user slow down at the end of a sentence?
• Filler sounds: Sounds like "um" and "uh" usually mean the user is still thinking.

Advantage: It hears how something is said, not just what is said, and it does not depend on the transcript.

Disadvantage: It has less understanding of the meaning and the earlier conversation than a text-based model.

Some systems combine both the words and the sound in one model, so that they get the meaning from the words and the tone from the sound.

If we want to go deep into SLMs and Model Distillation, Multimodal AI, and how to Design a Real-Time Voice AI Agent, we can check out our AI and Machine Learning Program at Outcome School.

How Both Work Together in a Voice Pipeline

Model-based turn detection usually does not replace the VAD. It works on top of it.

The VAD is cheap, so it runs all the time. The turn detection model is bigger, so we do not want to run it on every 30 millisecond piece of audio. Hence, we run the model only when the VAD detects a short pause, for example, 200 milliseconds (ms) of silence.

The flow looks like the diagram below:

        User speaks
             ↓
+-------------------------+
|  VAD: speech or not?    |
+-------------------------+
             ↓
     200 ms of silence
             ↓
+-------------------------+
|  Turn detection model:  |
|  is the user done?      |
+-------------------------+
             ↓
   Probability above 0.8
             ↓
+-------------------------+
|  End turn and reply     |
+-------------------------+

When the model says the user is likely not done, the system waits for a longer silence before replying, just in case the model is wrong.

We can write the logic in code as below:

SHORT_PAUSE_MS = 200
MAX_WAIT_MS = 3000
THRESHOLD = 0.8

def on_pause(silence_ms, conversation, audio):
    if silence_ms < SHORT_PAUSE_MS:
        return  # too short, keep listening

    prob_done = turn_model.predict(conversation, audio)

    if prob_done >= THRESHOLD:
        end_of_turn()  # likely done, reply quickly
    elif silence_ms >= MAX_WAIT_MS:
        end_of_turn()  # long silence, reply anyway

Here, we can see that:

• The function on_pause is called for every piece of audio while the user is silent, and silence_ms tells us how long the user has been silent.
• After a short pause of 200 milliseconds, we ask the turn detection model for the probability that the user is done.
• If the probability is at or above the threshold of 0.8, we end the turn quickly.
• If the probability is low, we keep waiting. But if the silence reaches 3 seconds, we end the turn anyway, so the assistant never gets stuck.

Note: This code calls the model on every check to keep it short. In a real system, we run the model once per pause and reuse the result until the user speaks again.

Now, let's run our restaurant example through this pipeline.

Step 1: The user says "I want to book a table for" and pauses to think.

Step 2: After 200 milliseconds of silence, the model gives a probability of 0.05. This is below 0.8, so the assistant keeps waiting.

Step 3: The user says "um... four people." The VAD hears speech, and the silence counter resets.

Step 4: The user stops. After 200 milliseconds of silence, the model gives a probability of 0.92. This is above 0.8, so the assistant ends the turn and starts replying.

With a 700 millisecond silence timeout, the assistant would interrupt at Step 2 if the thinking pause was longer than 700 milliseconds, and it would wait 700 milliseconds at Step 4. Here, it waits through the thinking pause and replies after 200 milliseconds of silence, plus the time the model takes to run.

We have a detailed blog on how speculative LLM generation starts replying before the user finishes that explains how to hide even this wait.

Silence-Based vs Model-Based Turn Detection

Let me tabulate the differences between silence-based and model-based turn detection for your better understanding.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

When to Use Which One

We must choose based on our use case.

Use silence-based turn detection when:

• We are building a prototype and want something working quickly.
• The users give short, simple commands, like "turn on the lights".
• The device has limited computing power.
• We need to support many languages and do not have a turn detection model for them.

Use model-based turn detection when:

• We are building a natural, human-like voice assistant.
• Users speak in long sentences, pause to think, or give numbers and addresses.
• Both interruptions and slow replies hurt the experience, like in customer support calls.

Many real voice assistants keep the VAD for detecting speech and silence, and add a turn detection model on top of it to decide whether the pause is the real end of the turn, with a long silence timeout as a backup.

Now we must have understood how silence-based and model-based turn detection differ, how each one decides the end of a turn, and when to use which one.

Frequently Asked Questions

Is turn detection the same as endpointing?

Yes. Turn detection is also called end-of-turn detection or endpointing. All three names mean the same job: deciding when the user has finished their turn in a conversation, so that the voice assistant knows when to start speaking.

What does a Voice Activity Detector (VAD) do in turn detection?

A VAD tells us whether each short piece of audio contains human speech or not. It is a small tool, usually a tiny model, so it is cheap enough to run all the time. Silence-based turn detection counts the silence it reports, and model-based turn detection uses it to decide when to run the bigger model.

Is there one silence timeout that works for everyone?

No. A short timeout gives fast replies but interrupts users who pause to think. A long timeout causes fewer interruptions but makes every reply feel slow. No single number works well for every user and every sentence, which is why model-based turn detection changes how long it waits based on whether the sentence looks complete.

Does model-based turn detection work for every language?

No. A turn detection model works well only for the languages it was trained on. Silence-based turn detection works for almost any language, because it does not care about the words. So, when we need many languages and do not have a turn detection model for them, silence-based turn detection is the safer choice.

Why do we still keep a silence timeout with model-based turn detection?

We keep it because the model can be wrong. If the model keeps saying the user is not done, a long silence timeout, for example 3 seconds, ends the turn anyway. This makes sure the assistant never gets stuck waiting.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.