What are backchannels, and why should "mm-hmm" not stop the agent?

Backchannels are the short sounds like "mm-hmm", "yeah", and "okay" that a listener makes to tell the speaker "I am following you, please continue". A voice agent must not stop for them, because the person is asking the agent to keep going, not asking to speak.

In this blog, we will learn about what backchannels are and why "mm-hmm" should not stop the agent. We will also see how backchannels work in a normal human conversation, what goes wrong when a voice agent treats them as interruptions, how an agent tells a backchannel apart from a real interruption, what the agent does while it is still unsure, and where this works well and where it fails.

We will cover the following:

• What is a backchannel?
• Backchannels in a normal conversation
• What goes wrong when "mm-hmm" stops the agent
• Backchannel vs real interruption
• How a voice agent tells them apart
• What the agent does while it is unsure
• A complete example
• Where it works well and where it fails

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is a backchannel?

A backchannel is a short sound or word that a listener makes while someone else is speaking, to show that they are listening and that the speaker can continue.

Common examples are:

• "mm-hmm"
• "uh-huh"
• "yeah"
• "okay"
• "right"
• "I see"
• "wow"

The term was introduced by the linguist Victor Yngve in 1970. He noticed that in a conversation, there are two channels at the same time. The main channel belongs to the person who holds the turn and is speaking. The back channel belongs to the listener, who keeps sending small signals back to the speaker without taking the turn. So, a backchannel is a small message that travels back from the listener to the speaker, while the main message is still going on.

A backchannel is not an attempt to take the turn. The listener is not asking to speak. The listener is saying, "Go on, I am with you."

We have a detailed blog on turn-taking in a voice AI agent that explains how the speaker and the listener share the conversation.

Backchannels in a normal conversation

Let's say our friend is telling us about a trip.

Friend: So we reached the station at 6 in the morning,
Us:     mm-hmm
Friend: and the train was already full,
Us:     oh no
Friend: so we had to wait for the next one.

Here, we spoke twice while our friend was talking. But our friend never stopped. Our friend did not say, "Yes? Do you want to say something?" Our friend just continued the story.

This is how humans talk. We send backchannels all the time, and that too without even thinking about it. If we stay completely silent on a phone call, the other person often asks, "Hello? Are you there?" Means, backchannels are a part of what makes a conversation feel natural.

Backchannels are also not the same in every language and culture. In Japanese, they have an everyday name, aizuchi, and listeners use them much more often than in English. So, a voice agent that talks to people around the world will hear a lot of them.

Now, let's see what happens when a voice agent hears these small sounds.

What goes wrong when "mm-hmm" stops the agent

Before jumping into the problem, we must know how a voice agent listens while it speaks.

A voice agent has three main parts. Speech-to-Text (STT) converts our voice into text, an LLM (Large Language Model) reads that text and writes a reply, and Text-to-Speech (TTS) converts the reply into voice.

A voice agent keeps its microphone on even while it is talking, so that we can interrupt it. When we start speaking in the middle of the agent's reply, it is called barge-in. The agent detects this and stops talking, so that it can listen to us.

The simplest way to detect barge-in is to use Voice Activity Detection (VAD). VAD is a component, usually a small model, that listens to the audio and answers one question: "Is a human voice present right now, yes or no?" Before VAD runs, the agent removes its own voice from the microphone sound using echo cancellation, so that it does not hear itself.

So, the simplest rule is:

If VAD hears a human voice while the agent is speaking
    ↓
Stop the agent immediately

This rule looks fine. But, VAD only knows that someone made a voice sound. It does not know what that sound means. "Stop" and "mm-hmm" look the same to VAD. Both are a human voice.

Let's say we call a voice agent to know the steps to reset our internet router.

Agent: First, unplug the router from the power socket.
Us:    mm-hmm
Agent: (stops talking)
Agent: Sorry, did you want to say something?
Us:    No, please continue.
Agent: Sure. First, unplug the router from the power socket...

Here, we just said "mm-hmm" to show that we were following. But the agent treated it as an interruption. It stopped, asked a question, and then started again from the beginning.

The following issues arise:

• The conversation keeps breaking, again and again.
• The agent repeats what it has already said, which wastes time.
• We start to stay silent while the agent speaks, which feels unnatural.
• In the worst case, the agent throws away the rest of its answer and replies to "mm-hmm" as if it were a new question.

This mistake is called a false barge-in, which means the agent stops for something that was not a real interruption. Backchannels are a common cause of false barge-ins. We have a detailed blog on how a voice agent tells a real interruption from a cough or background noise that covers the other causes step by step.

So, the agent must learn one thing: not every voice in the middle of its reply is an interruption.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Backchannel vs real interruption

Now, the question is: how is "mm-hmm" different from a real interruption?

Let's look at a few things the user can say while the agent is speaking:

Here, we can see a clear pattern. A backchannel is short, it carries no new information, and it does not ask the agent to do anything. A real interruption carries a new meaning: a correction, a question, or a request to stop.

But, the same word can also mean two different things. Consider the word "okay":

• "okay" said softly in the middle of the agent's sentence is a backchannel. It means "go on".
• "okay, okay, I got it" said loudly and quickly is a request to stop. It means "enough, you can stop now".
• "okay, but what about the price?" starts like a backchannel and then becomes a real question.

So, the agent cannot decide only by looking at one word. It must look at a few signals together.

How a voice agent tells them apart

A voice agent uses a few signals to tell a backchannel apart from a real interruption. Let's see them one by one.

Signal 1: How long the sound is

A backchannel is usually very short, often less than a second. A real interruption is usually longer, because the person is saying a full sentence.

So, the agent can wait for a minimum amount of speech before it decides to stop. Let's say the agent waits until the user has been speaking for at least half a second.

Advantage: It is simple, and it ignores short sounds like a cough or a quick "mm".

Disadvantage: A real "stop" or "wait" is also very short, so it can get ignored. And a backchannel like "yeah, yeah, okay" can be longer than the limit.

So, time alone is not enough.

Signal 2: What the words are

As we saw earlier, the agent already uses Speech-to-Text to convert the user's voice into text. So, once the agent has the words, it can check them.

• If the words are only from a list like "mm-hmm", "uh-huh", "yeah", "okay", "right", it is most likely a backchannel.
• If the words contain "stop", "wait", "no", "hold on", or a new question, it is most likely a real interruption.

A simple version of this check looks like below:

BACKCHANNELS = {"mm-hmm", "mhm", "uh-huh", "yeah", "yes", "okay", "ok", "right", "sure"}

def is_backchannel(text):
    for mark in ",.!?":
        text = text.replace(mark, " ")
    words = text.lower().split()
    if len(words) == 0 or len(words) > 3:
        return False
    return all(word in BACKCHANNELS for word in words)

Here, we have a small set of common backchannel words. The function first removes the punctuation marks, converts the text to lowercase, and splits it into words. If the user said more than three words, it is treated as a real interruption, because a backchannel is short. Then, it checks that every word is a backchannel word. So, "yeah okay" returns True, but "okay but what about the price" returns False.

Note: Real STT systems can write the same sound in different ways, such as "mm hmm", "mmhmm", or "uh huh". So, in practice, the list must cover these spellings too.

Advantage: It understands the actual words, so "wait" and "mm-hmm" are no longer the same.

Disadvantage: STT needs a little time to produce the words. And a fixed list cannot understand meaning. "Right" can be a backchannel, but "right, but that is not my question" is not.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Signal 3: How it is said

Humans do not only listen to words. We also listen to the tone. A backchannel is usually soft and low in volume. A real interruption is often louder, faster, and has more energy, because the person wants the agent to stop.

Some voice agents use an audio model that listens to these features of the voice, such as volume, pitch, and speed. This helps in cases where the words are the same but the intention is different, like the soft "okay" and the loud "okay, okay, I got it" that we saw earlier.

Signal 4: When it is said

Humans do not send backchannels at random moments. We usually send them at small natural pauses, for example at the end of a phrase, when the speaker takes a short breath.

In our trip example, we said "mm-hmm" right after "at 6 in the morning". That is a natural place for a backchannel.

So, if a short sound arrives exactly at a small pause in the agent's speech, it is more likely a backchannel. The agent knows where these pauses are, because it created that speech itself. If the user starts speaking in the middle of the agent's sentence, and keeps speaking, it is more likely a real interruption.

Signal 5: A model that looks at everything together

Instead of writing separate rules for each signal, we can train a small model on real conversations. This model looks at the audio, the words, and the timing together, and gives a single answer: backchannel or real interruption.

For the sake of understanding, we can think of it like below:

User audio + words + timing
    ↓
Interruption model
    ↓
"backchannel" or "interruption"

The model learns from many examples of real conversations, where people have marked which sounds were backchannels and which were real interruptions. This usually handles the tricky cases better than a fixed list.

Newer full-duplex speech models go one step further. Full-duplex means the model can listen and speak at the same time, just like two people on a phone call. Such a model can learn from conversation data when to keep talking, when to stop, and when a sound is just a backchannel, without a separate set of rules.

To learn Multimodal AI and how to Design a Real-Time Voice AI Agent end to end, we can check out our AI and Machine Learning Program at Outcome School.

What the agent does while it is unsure

There is still one problem. All the signals above need a little time. VAD hears the voice almost instantly, but the words from Speech-to-Text come a bit later.

So, what must the agent do in that small gap?

If it stops immediately, it can stop for a backchannel. If it waits too long, it keeps talking over a person who really wants to speak.

Here comes the soft pause into the picture.

When VAD first hears the user's voice, the agent does not stop fully. Instead, it lowers its volume or pauses for a moment, and keeps its place in the sentence. Then:

• If the signals say it was a backchannel, the agent continues from the same place.
• If the signals say it was a real interruption, the agent stops fully and listens.

User makes a sound while the agent speaks
    ↓
Agent lowers its volume and keeps its place
    ↓
Check the length, words, tone, and timing
    ↓
Is it a backchannel?
    ↓
Yes: continue from the same place
No:  stop fully and listen

This way, the agent reacts fast, and it still does not throw away its answer by mistake.

One more thing to notice: a backchannel must not be sent to the LLM as a new message. If the agent sends "mm-hmm" to the LLM as if the user asked something, the LLM often tries to reply to it, with something like "Great, is there anything else I can help you with?" So, when a backchannel is detected, the agent does not treat it as a new turn and keeps going.

A complete example

Let's take the router example again, now with backchannel handling.

Agent: First, unplug the router from the power socket.
       Then, wait for thirty seconds.
Us:    mm-hmm
Agent: After that, plug it back in and wait for the lights to turn green.
Us:    okay
Agent: Once the lights are green, try opening a website.
Us:    wait, which light should be green?
Agent: (stops) The light with the internet symbol on it.

Let's see what happened at each step:

• Step 1: We said "mm-hmm". VAD heard a voice, so the agent lowered its volume. STT gave the text "mm-hmm". It is short, it is in the backchannel list, and it came at a natural pause. So, the agent continued from the same place.
• Step 2: We said "okay". The same checks passed. It was soft and short. The agent continued.
• Step 3: We said "wait, which light should be green?". STT gave a longer text with the word "wait" and a question. This is a real interruption. The agent stopped fully, and the LLM got our question as a new message.

The agent never broke the flow for the backchannels, and it still stopped quickly when we needed it to. This is how a voice agent handles backchannels.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Where it works well and where it fails

It works well:

• For common backchannels like "mm-hmm", "yeah", "okay", and "right", which are short and easy to recognize.
• When the user speaks clearly and the audio is clean.
• When the agent uses a soft pause, so that it does not lose its place.
• When it combines more than one signal instead of depending only on VAD.

It fails or struggles:

• When the same word can mean two things, like "okay" or "right", and the tone is not clear.
• When a backchannel turns into a real sentence, like "yeah, yeah, but...", because the agent must change its decision in the middle.
• With noisy audio, where it is hard to hear the length and the tone.
• With different languages and cultures, where backchannels sound different and come more often.
• When the Speech-to-Text is slow, because the agent stays unsure for a longer time.

Now we must have understood what backchannels are, why "mm-hmm" should not stop the agent, and how a voice agent tells a backchannel apart from a real interruption.

Frequently Asked Questions

Who introduced the term backchannel?

The linguist Victor Yngve introduced the term in 1970. He noticed that a conversation has two channels at the same time. The main channel belongs to the person who holds the turn and is speaking, and the back channel belongs to the listener, who keeps sending small signals back without taking the turn.

Is "okay" always a backchannel?

No. The same word can mean different things. A soft "okay" in the middle of the agent's sentence means "go on". A loud and quick "okay, okay, I got it" means "you can stop now". And "okay, but what about the price?" starts like a backchannel and becomes a real question. So, the agent must look at a few signals together.

Should a voice agent send "mm-hmm" to the LLM as a new message?

No. If the agent sends "mm-hmm" to the LLM as if the user asked something, the LLM often tries to reply to it, with something like "Great, is there anything else I can help you with?" So, when a backchannel is detected, the agent does not treat it as a new turn and keeps going.

Are backchannels the same in every language?

No. Backchannels are not the same in every language and culture. In Japanese, they have an everyday name, aizuchi, and listeners use them much more often than in English. So, a voice agent that talks to people around the world will hear a lot of them, and they can sound different and come more often.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.