How do filler phrases hide latency in a voice agent?
- Authors
- Name
- Amit Shekhar
- Published on
Filler phrases hide latency in a voice agent by filling the silence with a short, natural line like "Let me check that for you" while the agent is still working on the real answer, so the user hears a reply right away instead of waiting in silence.
In this blog, we will learn about how filler phrases hide latency in a voice agent. We will also see why silence feels so bad on a voice call, what a filler phrase is, when the agent decides to say one, the different ways to produce it, how it fits into the conversation memory, what to do when the user interrupts it, and where it helps and where it hurts.
We will cover the following:
- The problem: silence on a voice call
- What is a filler phrase?
- How a filler phrase hides latency
- When should the agent say a filler phrase?
- Ways to produce a filler phrase
- Adding the filler phrase to the conversation memory
- Handling an interruption during a filler phrase
- Where filler phrases help and where they hurt
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning at Outcome School.
Let's get started.
The problem: silence on a voice call
Before jumping into filler phrases, we must understand the problem they solve.
A voice agent is an AI agent that we talk to with our voice, and it talks back with a voice. Most voice agents work as a pipeline of three steps:
- Speech-to-Text (STT): It turns what the user said into text.
- LLM (Large Language Model): It reads that text and decides the reply.
- Text-to-Speech (TTS): It turns the reply text back into a voice.
Each step takes some time. The waiting time between the moment the user stops speaking and the moment the agent starts speaking is called latency.
In a normal human conversation, the gap between one person finishing and the other person starting is very short. Research on conversations across many languages has found that this gap is often around 200 milliseconds, which is one-fifth of a second.
Now, let's say the user asks:
"What is the status of my order number 4521?"
To answer this, the agent cannot reply from memory. It has to call a tool, which means our code has to look up the order in a database or call another service. The LLM recommends the tool and its input, and our code runs it. This lookup can take a second, two seconds, or even more.
So, the timeline looks like below:
User stops speaking
↓
STT finishes the text
↓
LLM decides to look up the order
↓
Tool call runs (about 2 seconds)
↓
LLM writes the reply
↓
TTS speaks the reply
During all of this, the user hears nothing. On a phone call, two or three seconds of silence feels very long. The user starts thinking:
- "Did it hear me?"
- "Has the call dropped?"
- "Should I say it again?"
Then, the user says "Hello?" just as the agent starts speaking. Now, both are talking over each other, and the conversation breaks.
This silence is the real problem. We cannot always remove the waiting time, because the tool call needs its own time to finish. But we can change what the user hears during that time.
Here comes the filler phrase into the picture.
What is a filler phrase?
A filler phrase is a short, natural line that a voice agent speaks while it is still preparing the real answer, so that the user is not left in silence.
Some common examples:
- "Let me check that for you."
- "One moment, please."
- "Sure, let me pull up your order."
- "Give me a second while I look that up."
Let's say we call a customer care center and ask a human agent about our order. The human agent does not stay silent while typing into the computer. They say "Sure, let me check that for you," and then they type. We are happy to wait, because we know someone is working on our request.
A voice agent does the same thing with a filler phrase.
How a filler phrase hides latency
The best way to learn this is by taking an example.
Assume that the full reply for the order question takes 3,000 milliseconds to start without any filler phrase.
Without a filler phrase:
0 ms User stops speaking
(silence)
3,000 ms Agent: "Your order 4521 is out for delivery."
The user sits in silence for 3 seconds.
With a filler phrase:
0 ms User stops speaking
800 ms Agent: "Sure, let me check that for you."
(tool call keeps running in the background)
3,000 ms Agent: "Your order 4521 is out for delivery."
Here, we can notice that the real answer still arrives at 3,000 milliseconds. The total work did not get faster at all. But the user hears the agent start replying at 800 milliseconds. For the sake of example, these 800 milliseconds cover three things. The agent makes sure that the user has finished speaking, STT turns the speech into text, and the LLM decides to look up the order.
Here, we must understand that there are two kinds of latency:
- Actual latency: the time until the real answer is ready. The filler phrase does not change this.
- Perceived latency: the time until the user hears something that shows the agent is working. The filler phrase cuts this down a lot.
The filler phrase also takes time to speak. "Sure, let me check that for you" takes around 2 seconds to say out loud at a normal speaking speed. So, while the agent is speaking it, the tool call is running at the same time.
In our example, the filler phrase plays from about 800 milliseconds to about 2,800 milliseconds, and the real answer starts at 3,000 milliseconds.
Means, the time spent speaking the filler phrase overlaps with the time spent on the tool call, and the gap of silence after the filler phrase shrinks to only about 200 milliseconds.
This is how a filler phrase hides latency. It does not remove the wait. It fills the wait with speech.
Planning your move into AI?
No matter which tech domain you work in today, with your effort, you can transition to these roles:
- AI Engineer
- Forward Deployed Engineer
- AI Agent Engineer
- Harness Engineer
- Voice AI Engineer
- AI Architect
- LLM Engineer
- LLM Inference Engineer
- Gen AI Engineer
- MLOps Engineer
- ML Engineer
- Deep Learning Engineer
Our AI and Machine Learning Program at Outcome School takes you there.
When should the agent say a filler phrase?
Now, the next big question is: should the agent say a filler phrase before every reply?
The answer is no.
If the user says "Hi" and the agent says "One moment, please" before saying "Hello", it sounds strange. For a simple question, the reply is fast anyway, and a filler phrase only adds extra talking.
So, we need a rule for when to say it. There are two common triggers.
Trigger 1: Say it when a tool call starts.
We know that a tool call is often slow. So, the moment the LLM recommends a tool, like get_order_status, our code plays a filler phrase and runs the tool at the same time.
We can write the logic as below:
import asyncio
async def on_tool_call(tool_name, tool_args):
# play the filler phrase and run the tool at the same time
filler_task = asyncio.create_task(speak("Let me check that for you."))
result = await run_tool(tool_name, tool_args)
await filler_task # let the filler phrase finish first
return result
Here, speak sends a line to TTS and plays it, and run_tool runs the tool. We have started speaking the filler phrase as a separate task, so it does not block the tool call. Then, we run the tool. Before we return the result, we wait for the filler phrase to finish, so that the real answer does not cut it off in the middle.
In a real agent, we play a filler phrase only for the tools that we know are slow. A tool that answers in 100 milliseconds does not need one.
Trigger 2: Say it when the wait crosses a time limit.
Sometimes, we do not know in advance if the reply will be slow. So, we start a timer when the user stops speaking. If the first part of the real answer has not arrived within a set time, for example 1,000 milliseconds, we play a filler phrase.
We can write the logic as below:
async def reply(user_text):
answer_task = asyncio.create_task(generate_answer(user_text))
try:
# wait up to 1 second for the real answer
answer = await asyncio.wait_for(asyncio.shield(answer_task), timeout=1.0)
except asyncio.TimeoutError:
await speak("One moment, please.")
answer = await answer_task
await speak(answer)
Here, we can see that if the answer comes back within 1 second, the user never hears a filler phrase. The filler phrase plays only when the answer is slow. We have used asyncio.shield so that the timeout does not cancel the real answer while it is still being prepared.
Few important points:
- For the sake of understanding, this code waits for the full answer. In a real agent, the answer is streamed, and the timer only waits for the first part of it.
- The time limit must be picked carefully. If it is too short, the agent says a filler phrase even for fast replies. If it is too long, the user already sits in silence before the filler phrase comes.
We can also use both triggers together: a filler phrase when a slow tool call starts, and a timer as a safety net for everything else.
A quick note for you
No matter which tech domain you work in, get familiar with these topics:
- LLM
- RAG
- MCP
- Agent
- Fine-tuning
- Quantization
We put it all together in one video:
AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization
No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.
Now, let's get back to the topic.
Ways to produce a filler phrase
Now, let's understand where the filler phrase audio comes from. There are three common options.
Option 1: Pre-recorded audio.
We generate the audio for a small set of filler phrases once, before any call starts, and save it. During the call, we play the saved audio.
- Advantage: It plays almost instantly, because there is no TTS step at that moment.
- Disadvantage: It always sounds the same, and it cannot mention anything specific to the user's question.
Note: The pre-recorded audio must use the same voice as the agent. If the filler phrase comes in one voice and the answer in another, the user notices it immediately.
Option 2: Generated by TTS from a fixed list.
We keep a list of filler phrases as text, pick one, and send it to TTS during the call.
- Advantage: We can change the list easily, and the voice always matches.
- Disadvantage: It adds a small TTS delay before the filler phrase starts.
Option 3: Generated by the LLM itself.
We tell the LLM in its instructions to say a short line before it calls a tool. For example: "Before calling a tool, tell the user in one short sentence what you are about to do." This short line before a tool call is often called a preamble.
We must also tell the LLM to describe the action it is taking, to vary the wording across turns, and to skip the preamble for direct answers and quick tool calls.
So, for the order question, the LLM itself first writes "Sure, let me look up order 4521 for you," and then it recommends the tool.
- Advantage: The filler phrase fits the question exactly, so it sounds more natural.
- Disadvantage: It is slower to start than pre-recorded audio, because it waits for both the LLM and TTS. Also, the LLM does not always follow the instruction.
Let me tabulate the differences between these three options for your better understanding so that you can decide which one to use based on your use case.
| Option | Speed to start | Sounds specific | Control |
|---|---|---|---|
| Pre-recorded audio | Fastest | No | Full |
| TTS from a fixed list | Fast | No | Full |
| Generated by the LLM | Slower | Yes | Partial |
One more thing to notice: with Option 1 and Option 2, we must keep a few different filler phrases and rotate them. If the agent says "Let me check that for you" five times in one call, it sounds like a machine. A simple rule is to never repeat the same filler phrase twice in a row.
Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.
Adding the filler phrase to the conversation memory
Now, let's understand why the agent must remember its own filler phrase.
Our code keeps a history of the conversation, which means a list of everything the user said and everything the agent said. We send this history to the LLM every time we need the next reply.
Now, suppose the agent played "Let me check that for you" from pre-recorded audio, but our code never added it to the history. The LLM does not know that it already said this. So, its next reply can start with "Let me check that for you" again. The user hears the same line twice:
"Let me check that for you. ... Let me check that for you. Your order is out for delivery."
To solve this, we must add the filler phrase to the history as an agent message, like below:
history.append({ "role": "assistant", "content": "Let me check that for you." })
Here, we are telling the LLM that the agent already said this line. Now, the LLM continues from there and gives the real answer without repeating the filler phrase.
When the LLM itself writes the preamble, as in Option 3, the line is already part of its own output. So, this step matters mainly for Option 1 and Option 2.
To learn Memory in Agents, Tool use in Agents, and how to Design a Real-Time Voice AI Agent end to end, check out our AI and Machine Learning Program at Outcome School.
Handling an interruption during a filler phrase
Now, let's say the agent starts saying "Let me check that for you," and the user cuts in with "Actually, it's order 4522."
The agent must treat the filler phrase like any other agent speech:
- Stop speaking the filler phrase right away.
- Cancel the tool call that is running for the wrong order, if it is not finished yet.
- Keep only the part of the filler phrase the user actually heard in the history.
- Listen to the new request and start again.
Where filler phrases help and where they hurt
Filler phrases help a lot, but they also have a cost. Let's see both sides.
Where they help:
- Tool calls, like looking up an order, booking a slot, or checking an account balance.
- Long thinking steps, where the LLM needs more time before it can start the answer.
- Phone calls, where silence makes users think the call has dropped.
Where they hurt:
- Fast replies. A filler phrase before a quick answer only adds delay to the real answer.
- Too many of them. If every reply starts with "One moment", the agent sounds slow, even when it is not.
- Wrong promises. If the filler phrase says "Let me book that for you" and the booking then fails, the user feels misled. A neutral line like "Let me check that for you" is safer.
- Very long waits. A filler phrase covers a wait of a few seconds well. If the tool takes ten seconds, one filler phrase is not enough. The agent must say another short update, like "This is taking a little longer than usual, thank you for waiting."
Note: A filler phrase is not a replacement for making the agent fast. We must still cut the real latency wherever we can, using streaming, faster models, and faster tools. The filler phrase covers only the part of the wait that we cannot remove.
We have a detailed blog on how speculative LLM generation starts replying before the user finishes that covers another way to cut this wait.
Now we must have understood how filler phrases hide latency in a voice agent, when to use them, how to produce them, how to handle an interruption, and where they help and where they hurt.
Frequently Asked Questions
Does a filler phrase make a voice agent answer faster?
No. A filler phrase does not make the real answer arrive any sooner, because the tool call still needs its own time to finish. It only cuts the perceived latency, which is the time until the user hears something that shows the agent is working. The actual latency, the time until the real answer is ready, stays the same.
What is a preamble in a voice agent?
A preamble is a short line that the LLM itself says before it calls a tool, like "Sure, let me look up order 4521 for you." We get it by telling the LLM in its instructions to say one short sentence about what it is about to do. It fits the question exactly, but it is slower to start than pre-recorded audio.
Why should a voice agent not repeat the same filler phrase?
If the agent says the same line, like "Let me check that for you", again and again in one call, it sounds like a machine. So, we keep a few different filler phrases and rotate them. A simple rule is to never repeat the same filler phrase twice in a row.
What should a voice agent say when a tool call takes very long?
It should say another short update. One filler phrase covers a wait of a few seconds well, but if the tool takes ten seconds, it is not enough. The agent can say something like "This is taking a little longer than usual, thank you for waiting" so that the user does not sit in silence.
Should a filler phrase promise a specific action?
No. It is safer to use a neutral line like "Let me check that for you." If the filler phrase says "Let me book that for you" and the booking then fails, the user feels misled.
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
You can connect with me on:
Follow Outcome School on: