How does a voice agent know a phone number or email is not finished yet?

A voice agent knows that a phone number or email is not finished yet by combining a few signals: how long the person has been silent, what question the agent just asked, and whether the words heard so far look like a complete phone number or email. If the value still looks incomplete, the agent keeps listening instead of replying.

In this blog, we will learn about how a voice agent knows a phone number or email is not finished yet. We will also see why people pause in the middle of numbers and emails, why waiting for silence alone fails, how the agent uses the question it just asked, how it checks whether the value looks complete, how a turn detection model reads the words, and where this works well and where it fails.

We will cover the following:

• The problem: people pause in the middle
• What is endpointing?
• Approach 1: Wait for a fixed silence
• Approach 2: Wait longer for everyone
• Approach 3: Use the question the agent just asked
• Approach 4: Check if the value looks complete
• Approach 5: A turn detection model that reads the words
• How all the signals work together
• What the agent does when it is still unsure
• Where it works well and where it fails

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

The problem: people pause in the middle

The best way to learn this is by taking an example. Let's say we call a pizza shop, and a voice agent answers the phone. The agent asks:

"Can I have your phone number, please?"

Most of us do not say all ten digits in one breath. We say it in small groups, like below:

"five five five"   (pause)   "one two three"   (pause)   "four five six seven"

Each pause can last close to one second. We pause because we are reading the number from a card, or remembering it, or breathing.

The same thing happens with an email address:

"john"   (pause)   "dot smith"   (pause)   "at gmail"   (pause)   "dot com"

Now, the question is: how does the agent know that "five five five" is only the beginning and not the full answer?

If the agent gets this wrong, it jumps in after the first pause and says something like "Sorry, that does not look like a valid phone number." The person gets interrupted, gets annoyed, and has to start again. This is a common problem in voice agents.

To solve this, we must first understand how a voice agent decides that a person has stopped talking.

What is endpointing?

Endpointing is the job of deciding the exact moment when a person has finished their turn of speaking, so that the agent can start replying.

The agent is always asking itself one question: "Is the person done, or should I keep listening?" This job is also called turn detection.

A typical voice agent works in a flow like below:

Person speaks
      ↓
Speech-to-Text (audio to text)
      ↓
Endpointing (is the person done?)
      ↓
LLM (decides what to reply)
      ↓
Text-to-Speech (text to audio)
      ↓
Agent speaks

Here, the LLM (Large Language Model) writes the reply, and the endpointing step sits between hearing and replying. If it says "done" too early, the agent interrupts. If it says "done" too late, the agent feels slow, and the person can think that the call has dropped.

The phone number and email problem is a special case of endpointing. Now, let's see the different approaches one by one, from the simplest to the best.

Approach 1: Wait for a fixed silence

The simplest way is to use a Voice Activity Detector, called VAD in short. A VAD listens to the audio and tells us one thing: is someone speaking right now, or is it silence?

So, the rule becomes:

"If the person has been silent for more than a fixed time, they are done."

This fixed time is often around half a second to one second. We can write the logic as below:

SILENCE_LIMIT = 0.6  # seconds

def is_turn_finished(silence_duration):
    return silence_duration > SILENCE_LIMIT

Here, we are checking only how long the person has been quiet. We are not looking at what they said.

This works well for normal answers like "Yes", "No", or "I want a large pizza." But for a phone number, the person says "five five five" and pauses for 0.8 seconds. The rule says the person is done. The agent replies too early.

The issue with this approach is that a pause in the middle of a number looks exactly like a pause at the end of a sentence. Silence alone cannot tell the difference. Let's see how the next approach solve this issue.

Approach 2: Wait longer for everyone

The first fix that comes to mind is to make the silence limit longer, say 2 seconds.

SILENCE_LIMIT = 2.0  # seconds

Now, the person can pause between "five five five" and "one two three", and the agent keeps listening.

But, every answer now has a 2-second wait at the end. When the person says "Yes", the agent waits 2 full seconds before replying. The whole conversation feels slow, and people start saying "Hello? Are you there?"

The issue with this approach is that we made every turn slow just to fix a few special turns. Let's see how the next approach solve this issue.

Approach 3: Use the question the agent just asked

The agent knows what it just asked.

If the agent just asked "Can I have your phone number?", then it already knows that the next answer is a phone number. We also know that people pause while saying phone numbers. So, the agent can wait longer only for this one answer.

This is called context-aware endpointing. The silence limit changes based on what the agent is expecting.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

We can write the logic as below:

SILENCE_LIMITS = {
    "normal": 0.6,
    "phone_number": 1.5,
    "email": 2.0,
}

def is_turn_finished(silence_duration, expected_answer):
    limit = SILENCE_LIMITS.get(expected_answer, SILENCE_LIMITS["normal"])
    return silence_duration > limit

Here, we have:

• expected_answer: what kind of answer the agent is waiting for. Our code sets this at the moment the agent asks the question.
• SILENCE_LIMITS: a different waiting time for each kind of answer.
• For a normal answer, or any kind of answer not in the list, the agent waits 0.6 seconds.
• For a phone number, it waits longer, and for an email, even longer, because people often spell parts of an email letter by letter.

The numbers here are just for the sake of understanding. Every team tunes them for their own users.

Now, "Yes" gets a quick reply, and the phone number gets more patience.

The issue with this approach is that it still only looks at silence. If the person says all ten digits quickly and stops, the agent still waits the full 1.5 seconds for nothing. And if the person pauses for 2 seconds while searching for their card, the agent still cuts in. Let's see how the next approach solve this issue.

Approach 4: Check if the value looks complete

Now, let's use the words themselves. Speech-to-Text gives us the text while the person is still speaking. These early pieces of text are called partial transcripts. So, at every pause, we can look at the text heard so far and ask: "Does this look like a full phone number yet?"

For a phone number, the check is simple. We count the digits.

Let's say we expect a 10-digit phone number. We can write the code as below:

def phone_looks_complete(text):
    digits = [ch for ch in text if ch.isdigit()]
    return len(digits) >= 10

Here, we are taking only the digits from the text and counting them. Speech-to-Text often turns spoken "five five five" into the digits "555", so counting works. If it gives the words instead, we first turn the words into digits. For a country where phone numbers have a different length, we change the number 10 in the code.

Let's see a step-by-step walkthrough of the phone number from our example:

Step 1: The person says "five five five" and pauses. Text so far: "555". Digits: 3. Not complete, so the agent keeps listening.

Step 2: The person says "one two three" and pauses. Text so far: "555 123". Digits: 6. Not complete, so the agent keeps listening.

Step 3: The person says "four five six seven" and pauses. Text so far: "555 123 4567". Digits: 10. Complete, so the agent can reply after a short silence.

For an email, we look for the shape of an email. A complete email has a name part, then "at", then a domain, then "dot", then an ending like "com" or "org".

We can write the code as below:

import re

EMAIL_PATTERN = re.compile(r"^[\w.+-]+@[\w-]+(\.[\w-]+)*\.[a-z]{2,}$")

def email_looks_complete(text):
    email = (
        text.lower()
        .replace(" at ", "@")
        .replace(" dot ", ".")
        .replace(" ", "")
    )
    return bool(EMAIL_PATTERN.match(email))

Here, we have:

• First, we turn the spoken words "at" and "dot" into the symbols @ and ..
• Then, we remove the spaces.
• Finally, we check the result against a pattern that describes the shape of an email. The pattern says: some letters or numbers, then @, then a domain name, then a dot and an ending of at least two letters.

Let's see the walkthrough for "john dot smith at gmail dot com":

Step 1: Text so far: "john dot smith". After the conversion: "john.smith". There is no @. Not complete.

Step 2: Text so far: "john dot smith at gmail". After the conversion: "john.smith@gmail". There is no ending like ".com". Not complete.

Step 3: Text so far: "john dot smith at gmail dot com". After the conversion: "john.smith@gmail.com". It matches the pattern. Complete.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Now, we combine this with Approach 3. The final rule becomes:

• If the value looks incomplete, wait longer, even beyond the normal limit for this kind of answer.
• If the value looks complete, reply after a short silence.

We can write the combined logic as below:

def is_turn_finished(silence_duration, expected_answer, text):
    if expected_answer == "phone_number":
        if phone_looks_complete(text):
            return silence_duration > 0.5
        return silence_duration > 3.0
    if expected_answer == "email":
        if email_looks_complete(text):
            return silence_duration > 0.7
        return silence_duration > 4.0
    return silence_duration > 0.6

Here, we can notice that the agent is now both faster and more patient. It is faster when the value is complete, and more patient when the value is clearly not finished. Now, if the person has said 6 digits and pauses for 2 seconds to look at their card, the agent does not cut in, because it waits up to 3 seconds for an incomplete number.

Note: We still keep a maximum waiting time, which is 3 seconds for an incomplete phone number in our code. The agent must not wait forever. If the person stays silent for that long with only 6 digits, the agent replies and asks for the rest of the number.

The issue with this approach is that rules are rigid. Phone numbers have different lengths in different countries. People add extensions. People say "double five" instead of "five five". People correct themselves, like "five five... no wait, five six". Writing a rule for every case is hard. Let's see how the next approach solve this issue.

Approach 5: A turn detection model that reads the words

A turn detection model is a small AI model that reads the conversation so far and predicts how likely it is that the person has finished their turn.

Instead of writing rules by hand, we train the model on many real conversations. It learns patterns like:

• "My number is five five five" usually means the person is not done.
• "My email is john at" usually means the person is not done.
• "My number is five five five, one two three, four five six seven" usually means the person is done.

The model reads the conversation, including the question the agent asked, and gives back a number between 0 and 1. A number close to 1 means "the person is done". A number close to 0 means "the person is still talking".

Some turn detection models read only the text from Speech-to-Text. Others listen to the audio as well, because the voice carries extra clues. For example, when we pause in the middle of a number, our voice often stays flat or goes up, as if to say "more is coming". When we finish, our voice often drops down. This rise and fall of the voice is called intonation.

We have a detailed blog on how silence-based and model-based turn detection differ that explains this in depth.

We can use the model like below:

def is_turn_finished(silence_duration, conversation):
    done_probability = turn_model.predict(conversation)

    if done_probability > 0.8:
        return silence_duration > 0.3
    if done_probability > 0.4:
        return silence_duration > 1.0
    return silence_duration > 3.0

Here, we have:

• turn_model: the trained turn detection model.
• turn_model.predict(conversation): the model reads the conversation and gives back the chance that the person is done.
• If the model is very sure the person is done, the agent replies after a very short silence.
• If the model is not sure, the agent waits a medium time.
• If the model is sure the person is still talking, the agent waits a long time.

The best part is that the model can handle messy cases that rules miss, like "double five", extensions, and self-corrections, as long as it has seen enough such examples while training.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

How all the signals work together

In a real voice agent, these approaches are often stacked together. The flow looks like the diagram below:

Person pauses while speaking
      ↓
VAD detects silence and starts a timer
      ↓
Check what the agent asked last
(phone number, email, or normal answer)
      ↓
Check if the value looks complete
(digit count, email pattern)
      ↓
Turn detection model gives a score
      ↓
Decide how long to wait
      ↓
Silence longer than the wait time?
      ↓
Yes: agent replies
No: agent keeps listening

Here, each signal covers a weakness of the others:

• The VAD knows when there is silence, but not why.
• The question the agent asked tells us what kind of answer is coming.
• The completeness check tells us if the value has all its parts.
• The turn detection model can handle the messy cases that rules do not cover.

To learn how to Design a Real-Time Voice AI Agent, including SLMs and Model Distillation for small and fast turn detection models, we can check out our AI and Machine Learning Program at Outcome School.

What the agent does when it is still unsure

Even with all these signals, the agent will sometimes be wrong. So, good voice agents add a safety step after the person finishes: they read the value back.

For example:

"Thanks. I have your number as five five five, one two three, four five six seven. Is that correct?"

This gives the person a chance to fix any mistake. For emails, the agent can also spell the tricky part letter by letter, because "m" and "n", or "b" and "d", sound very similar on a phone line.

Another useful trick is to let the person interrupt. If the agent starts replying too early and the person keeps speaking, the agent must stop talking right away and keep listening. This is called barge-in. With barge-in, even an early reply does not ruin the call, because the person can continue.

Where it works well and where it fails

Where it works well:

• Phone numbers with a known, fixed length, because counting digits is simple and reliable.
• Emails with common endings like ".com" or ".org", because the pattern clearly shows when the email is complete.
• Calls where the agent asks one clear question at a time, because the agent knows exactly what kind of answer to expect.

Where it fails:

• When the person adds something after a complete-looking value, like "five five five, one two three, four five six seven, extension two". The value looks complete after "seven", so the agent replies too early.
• When the person gives a number of an unexpected length, like an international number with a country code.
• When Speech-to-Text makes a mistake, like hearing "for" instead of "four", so the digit count is wrong.
• When an email ending looks complete but is not, like "john at company dot co" when the person was about to add "dot uk".
• When there is a lot of background noise, because the VAD can mistake the noise for speech, so the silence timer does not start at the right time.

The read-back step and barge-in catch the cases that endpointing gets wrong.

Now we must have understood how a voice agent knows that a phone number or email is not finished yet.

Frequently Asked Questions

What is context-aware endpointing in a voice agent?

Context-aware endpointing means the silence limit changes based on what kind of answer the agent is expecting. If the agent just asked for a phone number or an email, it waits longer before replying. For a normal answer like "Yes", it still replies quickly.

What are partial transcripts in a voice agent?

Partial transcripts are the early pieces of text that Speech-to-Text gives us while the person is still speaking. At every pause, the voice agent can look at the text heard so far and check whether it already looks like a full phone number or email.

Can a voice agent use the tone of the voice to tell that a number is not finished?

Yes. Some turn detection models listen to the audio as well as reading the text. When we pause in the middle of a number, our voice often stays flat or goes up, as if more is coming. When we finish, our voice often drops down. This rise and fall of the voice is called intonation.

Should a voice agent wait forever for an incomplete phone number?

No. The agent still keeps a maximum waiting time, even when the value looks incomplete. If the person stays silent for that long with only part of the number, the agent replies and asks for the rest of the number.

Why is barge-in useful when a voice agent collects a phone number?

Because the agent will sometimes start replying too early. With barge-in, if the person keeps speaking, the agent stops talking right away and keeps listening. So, an early reply does not ruin the call, because the person can continue with the rest of the number.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.