How does a voice agent add its own backchannels like "okay" and "got it"?

A voice agent adds its own backchannels like "okay" and "got it" by running a fast, separate system next to the main conversation, which predicts the short moments where a listener would say "mm-hmm" and plays a short pre-recorded sound without taking the turn. It also says a quick "got it" right after the user finishes, so that the user knows they were heard while the real answer is still being prepared.

In this blog, we will learn about how a voice agent adds its own backchannels like "okay" and "got it". We will also see why a silent agent feels broken, the two kinds of backchannels an agent uses, how the agent decides the right moment and the right sound, how it plays the sound without interrupting itself, how full-duplex models do it, and where it works well and where it fails.

We will cover the following:

• What is a backchannel?
• Why does a silent voice agent feel broken?
• The two kinds of backchannels a voice agent uses
• The parts of a voice agent
• How the agent decides when to say "mm-hmm"
• How the agent picks the right sound
• How the agent plays the sound without stopping its listening
• How the agent avoids interrupting itself
• How the agent says "got it" after the user finishes
• How full-duplex models do it in a different way
• Where it works well and where it fails

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is a backchannel?

A backchannel is a short sound or word that a listener says while the other person is still talking, to show that they are listening, without trying to take the turn.

Some common backchannels are "mm-hmm", "uh-huh", "yeah", "okay", "right", and "got it".

Let's say we are telling a friend about our weekend on a phone call. While we are talking, our friend says "mm-hmm" and "yeah" every few seconds. Our friend is not trying to take over the conversation. Our friend is only telling us: "I am here, I am following you, please continue."

A backchannel is not a turn. After a backchannel, the speaker keeps talking. The listener does not start a full reply.

We have a detailed blog on backchannels and why "mm-hmm" should not stop the agent that explains how a voice agent handles the backchannels that the user says.

Why does a silent voice agent feel broken?

Let's say we call a voice agent and explain a long problem:

"So, I ordered a phone last week, and the box arrived yesterday, but when I opened it, the screen was cracked, and I also noticed that the charger was missing..."

We talk for 15 seconds, and during these 15 seconds, the agent says nothing.

We start to wonder whether the call dropped, and we often stop in the middle and ask "Hello? Are you there?"

A phone call has no face and no nodding, so the only way to show listening is with small sounds. Without them, the silence feels like a dead line.

So, here comes the backchannel to the rescue. If the agent says a soft "mm-hmm" after "the box arrived yesterday" and an "okay" after "the screen was cracked", we feel heard, and we keep talking naturally.

The two kinds of backchannels a voice agent uses

A voice agent uses backchannels at two different moments, and they are built in different ways.

Kind 1: While the user is still talking. These are sounds like "mm-hmm", "uh-huh", and "okay" in the middle of the user's speech. They mean "I am listening, please continue." Researchers call sounds like "mm-hmm" and "uh-huh" continuers, because they ask the speaker to continue.

Kind 2: Right after the user finishes. This is a short acknowledgment like "Got it." or "Okay, let me check that." at the start of the agent's reply. It means "I heard you, my answer is coming." Strictly speaking, this is an acknowledgment and not a pure backchannel, because the agent takes the turn.

Let me tabulate the differences between the two kinds for your better understanding.

We will learn how the agent builds each of them in detail.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

The parts of a voice agent

Before jumping into how the agent adds backchannels, we must know the basic parts of a voice agent.

Many voice agents follow a chain of steps like below:

User speaks
     ↓
Speech-to-Text (turns audio into text)
     ↓
LLM (decides what to reply)
     ↓
Text-to-Speech (turns reply text into audio)
     ↓
Agent speaks

Here, we can see that the main answer has to travel through all the steps, and each step takes some time. But, a backchannel cannot wait for this full chain. A "mm-hmm" that comes two seconds late sounds strange. Hence, the agent adds a fast, separate path only for backchannels. This path runs next to the main chain, all the time, while the user is talking.

The backchannel path has three jobs:

• Decide when to say something.
• Decide what to say.
• Play it without disturbing the main conversation.

Now, let's understand each job one by one.

How the agent decides when to say "mm-hmm"

The first job is timing. The agent must find the small openings in the user's speech where a listener would naturally say "mm-hmm".

When we finish a small piece of information, our voice gives clues without us thinking about it:

• Our pitch drops a little at the end of a phrase.
• We take a very short pause, often shorter than half a second.
• We finish a complete piece of meaning, like "and the box arrived yesterday".

The agent uses a small model called a backchannel predictor to catch these clues. This model listens to the user's audio continuously, and for every small slice of time, it gives a score between 0 and 1. The score answers one question: "Is this a good moment for a backchannel?"

Let's say the user says: "I ordered a phone last week, and the box arrived yesterday, but..."

The predictor gives scores like below (the numbers are just for the sake of understanding):

"I ordered a phone"        → score 0.05
"last week,"               → score 0.40
"and the box"              → score 0.08
"arrived yesterday,"       → score 0.82
"but"                      → score 0.03

Here, we can notice that the score jumps to 0.82 right after "arrived yesterday", where the pitch drops and a small pause appears. The agent has a threshold, for example 0.7. When the score crosses the threshold, the agent decides: "Now is a good moment."

The predictor learns this from many recorded human conversations, where we already know exactly where real listeners said "mm-hmm". It learns to connect the voice clues just before those moments with the backchannel that followed. Some models are trained to predict who will be speaking in the next couple of seconds, and a short burst of listener speech in that prediction means a backchannel moment.

But, a high score alone is not enough. The agent also follows a few simple rules on top of the score:

• Do not repeat too often. If the agent said "mm-hmm" in the last few seconds, it waits.
• Do not backchannel too early. If the user has spoken only for one or two seconds, the agent stays quiet.
• Do not backchannel at the end of a question. If the user asks "Can you check my order?", the user expects a real answer, not a "mm-hmm".

How the agent picks the right sound

The second job is choosing the sound. Not every backchannel fits every moment.

Let's see a few examples:

• The user is telling a long story: "mm-hmm" or "uh-huh" fits well, because it only means "continue".
• The user gives an important fact like "my order number is 4 5 7 2": "okay" or "got it" fits better, because it means "I have noted it".
• The user shares bad news like "the gift is ruined": a soft "oh, I see" fits better than a cheerful "okay".

So, the agent picks the sound based on what the user just said. A simple way is to look at the latest words from Speech-to-Text and follow rules. For example, when the user just said a number or a name, pick "got it". Otherwise, pick "mm-hmm". Some systems use a small classifier model that looks at the recent words and the tone of voice, and picks a category like "continue", "noted", or "sympathy". Some research models even predict the timing and the type together, at every small slice of time.

Now, the next big question is: where does the sound itself come from?

The agent does not send "mm-hmm" through the full LLM and Text-to-Speech chain, because that is too slow. Instead, the agent keeps a small set of pre-recorded clips ready, generated earlier with the same Text-to-Speech voice that the agent uses for its replies. For example:

continue  → "mm-hmm" (3 versions), "uh-huh" (2 versions)
noted     → "okay" (3 versions), "got it" (2 versions)
sympathy  → "oh, I see" (2 versions)

Here, we have more than one version of each sound, so that the agent does not play the same clip again and again. These clips are very short and already in memory, so the agent can play one instantly.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

How the agent plays the sound without stopping its listening

The third job is playing the sound, and here a backchannel is very different from a normal reply.

When the agent gives a normal reply, the turn switches: the user stops, and the agent speaks. But, when the agent says "mm-hmm", the turn must not switch. The user keeps the turn. The agent keeps listening.

So, the agent plays the backchannel clip with a few special rules:

• Listening never stops. Speech-to-Text and the backchannel predictor keep running while the clip plays.
• The turn does not change. The agent's main logic still believes "the user is speaking". The clip does not start a reply.
• The volume is low. The clip is often played a little softer than the normal voice, so that it sits under the user's speech instead of covering it.
• It is usually not added to the LLM conversation. Adding every "mm-hmm" would only fill the conversation history with noise.

How the agent avoids interrupting itself

Many voice agents support barge-in. Barge-in means that if the user starts speaking while the agent is talking, the agent stops talking and listens. To do this, the agent watches the incoming audio for speech whenever it plays any audio, and that includes a backchannel clip.

But, the agent's own "mm-hmm" also comes out of the user's speaker. If the user is on a speakerphone, the microphone picks up that "mm-hmm" again. Now, the agent hears a voice and can wrongly think that the user has started speaking.

So, how can we solve this? The answer is a mix of a few tools:

• Echo cancellation. This removes the agent's own audio from the microphone signal. The agent knows exactly what sound it just played, so it can subtract that sound from what the microphone hears.
• Ignoring its own clips. The agent marks the time window when a backchannel clip is playing. During that window, the agent raises the bar for what counts as new user speech, so that its own clip does not trigger barge-in.
• Not counting the clip as speech in the transcript. If Speech-to-Text writes "mm-hmm" into the user's text at the exact time the same clip was playing, the agent removes it, because the user did not say it.

To master Multimodal AI and how to Design a Real-Time Voice AI Agent, we can explore our AI and Machine Learning Program at Outcome School.

How the agent says "got it" after the user finishes

Now, it's time to learn about the second kind: the quick acknowledgment after the user finishes.

Let's say the user says: "Can you check the status of my order 4572?"

Now, the agent must turn the speech into text, send it to the LLM, look up the order in the order system, and then turn the reply into speech. This can take two or three seconds, or even more. That much silence after a question feels long on a phone call.

We have a detailed blog on how filler phrases hide latency in a voice agent that covers this end to end.

So, the agent says "Got it, let me check that for you." almost immediately. Then, while this short line is playing, the real work continues in the background. When the result is ready, the agent continues: "Your order was shipped yesterday and will reach you on Friday."

There are two common ways to build this:

Way 1: A fixed phrase from a small list. As soon as the system detects that the user has finished their turn, it plays a short pre-recorded phrase like "Okay." or "Got it." Later, when the LLM recommends an order lookup and the agent starts running it, the agent can play another short phrase like "Let me check that." to cover the waiting time of the lookup.

Way 2: Ask the LLM to start with a short acknowledgment. We can instruct the LLM to begin its reply with a short phrase. Since the LLM produces its reply token by token, and Text-to-Speech can start speaking as soon as the first few words arrive, the user hears "Sure, let me check." while the rest of the reply is still being generated.

The acknowledgment must never promise something that is not true. If the agent says "Got it" but then asks the user to repeat the order number, the "Got it" was wrong. Hence, it is safer to keep the acknowledgment neutral, like "Okay, one moment", when the agent is not yet sure that it understood the request.

How full-duplex models do it in a different way

Everything we discussed so far is for the chain of steps. In this setup, backchannels need a separate path, because the main chain handles one turn at a time.

There is another kind of voice model called a full-duplex model. Full-duplex means both sides can speak and listen at the same time, just like a normal phone call between two humans.

A full-duplex model works directly on audio. It listens to the user's audio stream and produces its own audio stream at the same time, in small time steps. For example, Moshi, a full-duplex model from Kyutai, models the user's audio and its own audio as two parallel streams.

In such a model, a backchannel is not a separate feature. At every small time step, the model decides what to put in its own audio stream: silence, a soft "mm-hmm", or real speech. Since it is trained on many real two-person conversations, where listeners say "mm-hmm" naturally, it learns to produce backchannels on its own, without a separate predictor.

Advantage: The timing and tone of the backchannel come from the same model that understands the conversation, so they often sound more natural.

Disadvantage: It is harder to control. With a separate backchannel path, we can set clear rules, like "not more than once every 4 seconds". With a full-duplex model, the behavior comes from training, and changing it often means more training.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Where it works well and where it fails

Where it works well:

• Long user explanations, where soft "mm-hmm" sounds keep the user talking with confidence.
• Collecting details like order numbers and addresses, where an "okay" after each part shows that the agent is keeping up.
• Slow requests that need a lookup, where a quick "Got it, let me check." fills the silence.

Where it fails:

• Wrong timing. If the predictor fires in the middle of a sentence, the "mm-hmm" sounds like an interruption, and the user often stops talking.
• Too many backchannels. An agent that says "mm-hmm" every two seconds sounds fake.
• Wrong sound for the moment. A cheerful "okay" after bad news sounds rude.
• Echo problems. Without good echo cancellation, the agent hears its own "mm-hmm" and treats it as the user's speech.
• Thinking pauses. When the user pauses to think, both a separate predictor and a full-duplex model can wrongly backchannel or even take the turn too early.
• False acknowledgments. Saying "Got it" and then asking the user to repeat everything makes the user lose trust.

Now we must have understood how a voice agent adds its own backchannels like "okay" and "got it", how it decides the right moment and the right sound, how it plays them without stopping its listening, and where it works well and where it fails.

Frequently Asked Questions

Does a voice agent send its "mm-hmm" through the LLM?

No. A "mm-hmm" sent through the full LLM and Text-to-Speech chain would come too late and sound strange. Instead, the agent keeps a small set of pre-recorded clips, generated earlier with the same Text-to-Speech voice, and plays one instantly when the backchannel predictor finds a good moment.

Should a voice agent's backchannels be added to the conversation history?

No. The agent's own "mm-hmm" and "okay" are usually not added to the LLM conversation. They do not change the turn, and the user keeps speaking. Adding every backchannel to the history would only fill it with noise.

How often should a voice agent say "mm-hmm"?

Not too often. If the agent said "mm-hmm" in the last few seconds, it waits, and it also stays quiet when the user has spoken for only one or two seconds. An agent that says "mm-hmm" every two seconds sounds fake.

Why should a voice agent say "Okay, one moment" instead of "Got it"?

Because the acknowledgment must never promise something that is not true. If the agent says "Got it" but then asks the user to repeat the order number, the "Got it" was wrong. A neutral line like "Okay, one moment" is safer when the agent is not yet sure that it understood the request.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.