How does a voice agent confirm and correct misheard information?

Authors
How does a voice agent confirm and correct misheard information?

A voice agent confirms misheard information by reading the important details back to the caller and asking "Is that correct?", and it corrects them by listening for a "no", finding which part was wrong, and fixing only that part.

In this blog, we will learn about how a voice agent confirms and corrects misheard information. We will also see why a voice agent mishears in the first place, how it uses a confidence score to decide when to confirm, the difference between explicit and implicit confirmation, how it reads numbers and spellings back clearly, how it fixes only the wrong part when the caller says "no", and when to confirm and when to skip it.

We will cover the following:

  • Why does a voice agent mishear?
  • What does it mean to confirm and correct?
  • What is a confidence score?
  • Explicit confirmation vs Implicit confirmation
  • Reading back numbers and spellings
  • Correcting only the wrong part
  • Checking the value with simple rules
  • Putting it all together in code
  • When to confirm and when to skip

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

Why does a voice agent mishear?

A voice agent is a system that talks to us over a call. It listens to our voice, understands what we said, and replies with a voice.

Many voice agents work in three steps:

  • Speech-to-Text (STT): It converts our voice into text.
  • LLM (Large Language Model): It reads that text and decides what to say.
  • Text-to-Speech (TTS): It converts the reply text back into voice.

Most mishearing happens in the first step. The STT model hears sound, not letters, and it has to guess which words made that sound. Most of the time, the guess is right. But, sometimes it is wrong.

Let's say the caller says:

"My name is Amit, A-M-I-T."

The STT model can return:

"My name is Amit, A-N-I-T."

The letters "M" and "N" sound close, especially on a phone line. Similarly:

  • "fifteen" and "fifty", or "thirteen" and "thirty", sound alike.
  • "B", "D", "P", and "T" sound alike.
  • Background noise, a weak network, and accents make it harder.

If the agent books an appointment for "Anit" or charges 50 dollars instead of 15 dollars, it causes a real problem for a real person. And in this pipeline, the LLM cannot know that the text is wrong, because the LLM only sees the text, not the voice.

So, the agent needs a way to catch these mistakes before it acts on them. Here comes confirmation into the picture.

What does it mean to confirm and correct?

Confirmation is the step where the voice agent repeats the important information back to the caller, so that the caller can say whether it is right or wrong.

Correction is the step where the voice agent takes the caller's "no", finds which part was wrong, and fixes it.

The best way to learn this is by taking an example.

Think about how we give our phone number to a person at a pizza shop over a call. We say the number. The person says it back to us, "9 8 7 6 5, 4 3 2 1 9, right?". They got one digit wrong, so we say, "No, the last digit is 0, not 9." They fix only that one digit.

A voice agent does the same thing. The full flow looks like below:

Caller says the information
            ↓
STT converts it into text
            ↓
Agent pulls out the important value
            ↓
Agent reads the value back to the caller
            ↓
Caller says "yes" or "no"
            ↓
"yes": agent saves it and moves on
"no":  agent fixes only the wrong part

Now, the next big question is: should the agent confirm everything? The answer is no. If it confirms every sentence, the call becomes slow and annoying. This is where the confidence score comes into the picture.

What is a confidence score?

A confidence score is a number between 0 and 1 that tells how sure the STT model is about its guess.

Many STT models return this score along with the text. Many of them give it for every single word, not only for the full sentence.

Let's say the caller says their account number. The STT output looks like below:

Word:        4      7      1      9      2      5
Confidence:  0.98   0.97   0.55   0.96   0.99   0.62

Here, we can see that the model is very sure about most of the digits. But it is not sure about the third digit "1" and the last digit "5". The sound of these two digits was unclear, so the model only gave its best guess.

Now, the agent can use a simple rule:

  • If the confidence is high, for example above 0.9, the agent can move ahead and repeat the value lightly inside its next sentence.
  • If the confidence is low, the agent must confirm clearly.
  • If the confidence is very low, for example below 0.4, the agent must ask the caller to say it again.

Note: These numbers are only for the sake of example. Every STT model scores differently, so the team must pick these limits by testing with real calls. Also, a high score does not guarantee that the word is right. The model can be confident and still be wrong.

Some STT models can also return more than one guess for the same audio. These are called N-best results, which means the top few guesses in order. For example:

Guess 1: "fifteen"   (0.51)
Guess 2: "fifty"     (0.47)

When the top two guesses are this close, the agent knows it must confirm. It can even use both guesses in the question: "Did you say fifteen, one five, or fifty, five zero?"

The confidence score is the first reason to confirm. The second reason is the cost of a mistake. A wrong name in a greeting is a small problem, but a wrong amount in a payment is a big problem. So, the agent confirms based on two things:

  • How unsure the STT model is.
  • How costly a mistake would be.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Now, let's learn how the agent confirms.

Explicit confirmation vs Implicit confirmation

There are two main ways to confirm: explicit and implicit.

Explicit confirmation

Explicit confirmation is when the agent reads the value back and directly asks a yes or no question.

Example:

Caller: I want to send 500 dollars to Rahul.
Agent:  Just to confirm, you want to send 500 dollars to Rahul. Is that correct?
Caller: Yes.

Here, the agent does not move ahead until the caller says "yes".

Advantage: It is safe. The caller gets a clear chance to catch the mistake.

Disadvantage: It adds one extra turn, which means one more back-and-forth, to the call. If the agent does this for every small detail, the call becomes long and tiring.

So, we use explicit confirmation for things that are costly to get wrong, like money, appointment dates, addresses, and account changes.

Implicit confirmation

Implicit confirmation is when the agent repeats the value inside its next question, without asking a separate yes or no question.

Example:

Caller: I want to book a table for Thursday.
Agent:  Sure, a table for Tuesday. How many people will be there?
Caller: Not Tuesday, Thursday. And four people.

Here, the STT model misheard "Thursday" as "Tuesday", and the agent said "Tuesday" inside its next question. So, the caller caught the mistake and corrected it in the same turn. If the agent had heard it right, the caller would simply answer the next question, and the call would move fast.

Advantage: It saves time. There is no extra yes or no turn.

Disadvantage: The caller can miss it. If the caller is busy thinking about the next question, they do not notice that the agent said "Tuesday" instead of "Thursday".

So, we use implicit confirmation for things that are less costly, and only when the confidence score is high.

Let me tabulate the differences between Explicit confirmation and Implicit confirmation for your better understanding.

PointExplicit confirmationImplicit confirmation
How it worksReads back and asks "Is that correct?"Repeats the value inside the next question
Extra turnYesNo
SafetyHighMedium
Speed of the callSlowerFaster
Best forMoney, dates, addresses, account changesLow-risk details, high confidence

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Now, it's time to learn how the agent reads back numbers and spellings, because these are the values that get misheard the most.

Reading back numbers and spellings

If the agent reads back a phone number as "nine billion eight hundred seventy six million...", the caller cannot check it. So, the agent must read back values the way a human would.

Rule 1: Read digits one by one, in small groups.

Instead of "9876543210", the agent says "9 8 7 6 5, 4 3 2 1 0". Short groups with a small pause between them are easy for the caller to follow and check.

Rule 2: Say tricky numbers in two ways.

For "15", the agent can say "fifteen, that is one five". Now, the caller can easily tell "fifteen" apart from "fifty".

Rule 3: Spell names and emails with a clear word for each letter.

Letters like "M" and "N", or "B" and "D", sound alike. So, the agent can spell with a word for each letter, like below:

"A as in Apple, M as in Mango, I as in India, T as in Tiger."

The agent can also ask the caller to spell the same way.

Rule 4: Write the read-back text for the voice, not for the screen.

The LLM writes text, and the TTS speaks it. If the LLM writes "amit.s@gmail.com", the TTS can read it in a way that is hard to follow. So, the agent turns it into spoken words first: "amit dot s at gmail dot com".

We have a detailed blog on how a voice agent handles structured inputs like dates, numbers, and addresses that explains this in depth.

Correcting only the wrong part

Now, let's say the caller says "no". What should the agent do?

The naive way is to ask for everything again: "Okay, please tell me the full phone number again." The issue with this approach is that the STT model can make a new mistake in another digit, and the caller gets frustrated because they have to repeat everything. Let's see how the better approach solves this issue.

The better way is to fix only the wrong part. Let's say the agent read back the number and the caller replied:

Agent:  I have 9 8 7 6 5, 4 3 2 1 9. Is that correct?
Caller: No, the last digit is zero.

Here, the caller has already told us which part is wrong and the right value. So, the agent must:

  • Understand that the caller is correcting, not giving a new number.
  • Find the part that changed: the last digit.
  • Replace only that part: 9 becomes 0.
  • Read back the fixed value once more: "Got it, 9 8 7 6 5, 4 3 2 1 0. Is that correct now?"

The LLM handles this step well, because it is trained to work with everyday language. It reads the old value and the caller's correction, and returns the new value. We can give the LLM an instruction like below:

The current phone number is 9876543219.
The caller said: "No, the last digit is zero."
Return the corrected phone number only.

The LLM returns 9876543210.

Callers correct in many ways, and the agent must handle all of them:

  • "No, it is Thursday, not Tuesday." (gives the right value directly)
  • "No, the last digit is zero." (points to the wrong part)
  • "Not M, it is N." (swaps one letter)
  • "No." (says nothing more)

For a plain "No.", the agent does not know which part is wrong. So, it asks a narrow question: "Sorry about that. Which part is wrong?" or it breaks the value into parts and confirms each part one by one.

Note: If the same value fails two or three times, the agent must stop trying with voice. It can ask the caller to type the number on the phone keypad, send a text message link, or transfer the call to a human. Asking again and again in a loop only frustrates the caller.

To master Prompt Engineering, Tool use in Agents, and how to Design a Real-Time Voice AI Agent from the ground up, check out our AI and Machine Learning Program at Outcome School.

Checking the value with simple rules

The agent does not depend only on the caller to catch mistakes. Many values follow fixed rules, and the agent can check them in code before it reads back the value. For example:

  • In many countries, a mobile number has a fixed number of digits. If the STT gave 9 digits where 10 are needed, something was missed.
  • The last digit of a credit card number is a check digit, calculated from all the other digits using a simple math rule called the Luhn check. If the number fails this check, at least one digit is wrong.
  • A date like "February 30" does not exist.
  • An email must have exactly one "@" and a domain after it.
  • An order ID or account number can be looked up in the company database. If it does not exist, it was most likely misheard.

When a rule fails, the agent knows the value is wrong even if the confidence score was high. So, it asks again right away and saves a wasted read-back turn.

Putting it all together in code

Now, let's see a simple version of this logic in code, just for the sake of understanding.

First, we decide how to confirm, based on the confidence and the risk. We can write the code as below:

def choose_confirmation(confidence, is_high_risk):
    if confidence < 0.4:
        return "ask_again"
    if is_high_risk or confidence < 0.9:
        return "explicit"
    return "implicit"

Here, we have three results:

  • ask_again when the STT model is very unsure. Reading back a bad guess only wastes a turn.
  • explicit when the value is high-risk, or when the confidence is not high enough.
  • implicit when the value is low-risk and the confidence is high.

Then, we turn a phone number into a voice-friendly read-back. We can write the code as below:

def speak_digits(number):
    groups = [number[i:i + 5] for i in range(0, len(number), 5)]
    return ", ".join(" ".join(group) for group in groups)

Here, we are splitting the number into groups of 5 digits, and we are putting a space between the digits so that the TTS reads each digit on its own. Most TTS models take a small pause at a comma, so the two groups are easy to hear.

If we call speak_digits("9876543210"), it will return the following:

9 8 7 6 5, 4 3 2 1 0

After that, we handle the caller's reply in a loop. We can write the code as below:

def confirm_value(value, max_tries=3):
    for _ in range(max_tries):
        reply = ask(f"I have {speak_digits(value)}. Is that correct?")

        if is_yes(reply):
            return value

        value = llm_correct(value, reply)

        if not is_valid_phone(value):
            value = ask_value_again()

    return transfer_to_human()

Here, ask, is_yes, llm_correct, is_valid_phone, ask_value_again, and transfer_to_human are helper functions that our agent already has. We can see that:

  • The agent reads back the value and asks "Is that correct?".
  • If the caller says "yes", the agent saves the value and moves on.
  • Otherwise, the LLM applies the caller's correction to the old value, so only the wrong part changes.
  • The agent checks the new value with a simple rule. If it fails, it asks for the value again.
  • After 3 failed tries, the agent stops looping and transfers the call to a human.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

When to confirm and when to skip

Confirmation makes the agent safer, but too much of it makes the call slow. So, we must find the right balance based on our use case.

Confirm explicitly when any one of these is true:

  • The value involves money, like a payment or a refund.
  • The value is hard to hear, like names, emails, codes, and long numbers.
  • The action cannot be undone easily, like cancelling an account.
  • The confidence score is low.

Use implicit confirmation when the value is low-risk and the confidence score is high. For example, the number of people at a table is low-risk, because it can be changed easily later.

Skip confirmation when the caller is just asking a question, like "What are your opening hours?", and nothing is being saved or changed.

One more thing to notice is that the agent can confirm several values together at the end of a booking, instead of one by one. For example: "So, a table for 4 people, on Thursday, at 7 PM, under the name Amit. Is that all correct?". This saves turns, and if one part is wrong, the agent fixes only that part.

We have a detailed blog on how to write a system prompt for a voice agent that covers how to tell the agent which details to confirm.

Now we must have understood how a voice agent confirms and corrects misheard information, how it decides when to confirm, and how it fixes only the wrong part.

Frequently Asked Questions

Why can't the LLM catch a misheard word on its own?

Because the LLM only sees the text, not the voice. The STT model turns the voice into text first, so if it hears "fifty" instead of "fifteen", the LLM gets the wrong text and has no way to know it is wrong. This is why the agent reads the important values back to the caller.

Does a high confidence score mean the word is right?

No. A high confidence score does not guarantee that the word is right, because the STT model can be confident and still be wrong. So, the agent also checks values with simple rules, like the number of digits in a mobile number or the Luhn check for a card number. When a rule fails, the agent knows the value is wrong even if the score was high.

What are N-best results in speech recognition?

N-best results are the top few guesses that an STT model returns for the same audio, in order. For example, it can return "fifteen" with 0.51 and "fifty" with 0.47. When the top two guesses are this close, the agent knows it must confirm, and it can even use both guesses in the question.

What should a voice agent do when the same value keeps failing?

It should stop trying with voice after two or three failed tries. It can ask the caller to type the number on the phone keypad, send a text message link, or transfer the call to a human. Asking again and again in a loop only frustrates the caller.

Can a voice agent confirm several details at once?

Yes. The agent can confirm several values together at the end of a booking, instead of one by one. For example: "So, a table for 4 people, on Thursday, at 7 PM, under the name Amit. Is that all correct?" This saves turns, and if one part is wrong, the agent fixes only that part.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.