Outcome School

How does a voice agent hide the delay of a slow tool call?

Authors
How does a voice agent hide the delay of a slow tool call?

A voice agent hides the delay of a slow tool call by never letting the line go silent. It speaks a short line while the tool runs, gives progress updates during a long wait, keeps the conversation going while the tool works in the background, and starts the tool early so that there is less delay to hide.

In this blog, we will learn about how a voice agent hides the delay of a slow tool call. We will also see why a slow tool call hurts so much on a voice call, how the agent speaks a short line right away, how it gives progress updates during a long wait, how it keeps talking while the tool runs in the background, how it starts the tool early to cut the delay itself, and what it says when the tool takes too long.

We will cover the following:

  • The problem: a slow tool call on a voice call
  • What is a tool call in a voice agent?
  • Technique 1: Speak a short line right away
  • Technique 2: Progress updates during a long wait
  • Technique 3: Keep talking while the tool runs in the background
  • Technique 4: Start the tool early
  • Technique 5: Make the tool itself faster
  • What if the tool takes too long?
  • Putting it all together in code

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

The problem: a slow tool call on a voice call

A voice agent is an AI agent that we talk to with our voice, and it talks back with a voice. Many voice agents work as a pipeline of three steps:

  • Speech-to-Text (STT): It turns what the user said into text.
  • LLM (Large Language Model): It reads that text and decides the reply.
  • Text-to-Speech (TTS): It turns the reply text back into a voice.

A good voice agent often aims to reply in around one second. People are used to quick replies in a conversation, so every extra second of waiting is easy to notice.

Now, let's say the user asks:

"Can you check if my flight AI 101 to Delhi is on time?"

The agent cannot answer this from memory. It has to ask the airline's system. This lookup can take two, three, or even ten seconds.

On a chat screen, a few seconds of waiting is fine, because we can see a loading spinner. On a voice call, there is no spinner. There is only silence.

And silence on a phone call means something to us. After two or three seconds of silence, the user starts to think:

  • "Did the call drop?"
  • "Did it hear me?"

So, the user says "Hello?" or repeats the question. Now, the agent hears this new speech and gets confused, and the whole conversation breaks.

So, the real problem is not the delay itself. The real problem is the silence that the delay creates.

What is a tool call in a voice agent?

Before jumping into the techniques, we must know what a tool call is.

A tool call is when the agent runs some code outside the LLM, like looking up a database or calling another service, to get information or to do an action.

The LLM never runs the tool itself. The LLM only recommends which tool to use and with what input. Our code, which we call the agent, runs the tool and gives the result back to the LLM. Then, the LLM uses that result to write the final reply.

For our flight example, the flow looks like below:

User: "Is my flight AI 101 to Delhi on time?"
        ↓
LLM recommends: check_flight_status(flight="AI 101")
        ↓
Agent runs the tool (this takes 3 seconds)
        ↓
Tool result: { "status": "delayed", "new_time": "6:40 PM" }
        ↓
LLM writes: "Your flight is delayed. It now leaves at 6:40 PM."
        ↓
TTS speaks the reply

Here, we can see that the 3 seconds of the tool are added on top of the normal delay of STT, LLM, and TTS. This is the delay that we need to hide.

Now, let's learn the techniques one by one.

Technique 1: Speak a short line right away

The first and the simplest technique is to say something as soon as the agent knows it needs a tool.

Let's take the example of a person at a help desk. When we ask them about our flight, they do not stare at us silently while typing. They say, "Sure, let me check that for you," and then they start typing. Those few words tell us that they heard us and that they are working on it.

A voice agent can do exactly the same. There are two common ways to do it.

Way 1: The LLM writes a short line along with the tool call.

We tell the LLM in its instructions: "Before you call any tool, first say one short sentence to the user." So, in the same reply, the LLM produces a short sentence like "Sure, let me check your flight." and also the tool call. The agent sends the sentence to TTS right away and runs the tool at the same time.

Advantage: The line fits the request. For a flight, it says "let me check your flight". For a refund, it says "let me look at your refund".

Disadvantage: We hear this line only after the LLM starts writing and TTS turns it into a voice, so it arrives a little later than a ready-made line.

Way 2: The agent plays a ready-made line.

The agent keeps a small list of ready-made lines like "One moment, please." or "Let me look that up." The audio for these lines is already made, so the agent can play one instantly as soon as the LLM recommends a tool.

Advantage: It is the fastest, because no extra LLM or TTS work is needed at that moment.

Disadvantage: The same few lines repeat, and they can sound robotic after some time. So, we must rotate between several lines.

Either way, the user hears a voice quickly, and the silence is gone.

Note: If a tool usually returns in a fraction of a second, the agent can skip the short line. Otherwise, the user hears "One moment, please." followed at once by the answer, which sounds odd.

We have a detailed blog on how filler phrases hide latency in a voice agent that explains this in depth.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Technique 2: Progress updates during a long wait

Technique 1 works well when the tool takes 2 or 3 seconds. But some tools take much longer. For example, booking a ticket or checking a long list of records can take 10 seconds or more.

The issue with only one short line is that after "Let me check that for you," there can still be 8 seconds of silence. The silence problem comes back.

So, here comes the progress update to the rescue.

A progress update is a short line the agent speaks at fixed points during a long wait, to tell the user that the work is still going on.

The agent uses a simple timer. The idea is like below:

Time since the tool startedWhat the agent says
0 seconds"Sure, let me check your flight."
4 seconds"I am still checking, it is taking a moment."
8 seconds"Almost there, thanks for waiting."

Here, we can notice that each line is different. If the agent says the same line again and again, it sounds repetitive.

If the tool finishes at 5 seconds, the agent stops the timer and speaks the real answer. The later lines are never spoken.

Some voice agents also play a soft background sound during the wait, like gentle keyboard typing. This works like hold music. It tells the user that the line is still open. But it must be quiet and simple, otherwise it becomes annoying.

Note: If the agent knows in advance that a tool is always slow, it can say so at the start: "This usually takes about 10 seconds." When the user knows how long to wait, the wait often feels shorter.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Technique 3: Keep talking while the tool runs in the background

In the first two techniques, the agent speaks while it waits, but the conversation is still on hold. The user cannot ask anything else until the tool finishes.

Now, let's understand a better approach. In this approach, the agent starts the tool in the background and continues to talk with the user. When the result arrives, the agent brings it into the conversation.

Let's see an example:

User:  "Book a table for two at 8 PM."
Agent: "Sure, I am booking your table now."
        ↓
(booking tool starts in the background, it takes 10 seconds)
        ↓
Agent: "While I do that, should I text you the confirmation?"
User:  "Yes, please."
Agent: "Sure, I will text it to you."
        ↓
(booking tool finishes)
        ↓
Agent: "Your table for two at 8 PM is booked. The text is on its way."

Here, we can see that the user never waits in silence. The 10 seconds of booking are filled with a useful question and answer, not with waiting.

To make this work, the agent must do two things:

  • It must not block the conversation while the tool runs. Our code starts the tool as a separate task and lets the LLM keep replying to the user.
  • When the result arrives, it must add the result to the conversation. Then, the LLM tells the user about it at the right moment, after the agent finishes its current sentence.

Some real-time voice model APIs support this directly. The model can keep talking with the user while a tool call is still waiting for its result.

Advantage: It is the most natural experience, because the conversation never stops.

Disadvantage: It is harder to build. The agent has to remember which tools are still running, and the LLM must not make up a result that has not arrived yet. If the user asks about it early, the LLM must say that it is still waiting.

Technique 4: Start the tool early

So far, all the techniques hide the delay. Now, let's see how we can make the delay itself smaller.

Normally, the agent waits for the user to finish speaking. Then, STT gives the full text, the LLM reads it, and only then the tool starts.

But often, it is clear which tool is needed long before the user finishes.

Let's say the user says:

"Can you check my flight AI 101... I think it leaves around six..."

STT gives the agent the text bit by bit while the user is still speaking. The agent can show this partial text to the LLM. By the time the user has said "check my flight AI 101", the LLM already knows that it needs the flight status tool with the input "AI 101". The rest of the sentence does not change the tool.

So, the agent can start the tool early, while the user is still speaking. This is called speculative execution, which means doing the work before we are fully sure we need it, because we are very likely to need it.

The flow looks like below:

User is still speaking: "check my flight AI 101..."
        ↓
Agent starts check_flight_status(flight="AI 101") early
        ↓
User finishes speaking
        ↓
Tool result is already ready or almost ready
        ↓
Agent replies with almost no wait

Here, the tool runs during the time the user is still talking. So, much of the delay is gone. And if the full sentence turns out to need a different tool, the agent throws the early result away.

Very important: We must only start a tool early when running it for nothing does no harm. Reading data is safe. Checking a flight status is safe, because if we do not need the result, we just throw it away. But booking a ticket, sending money, or cancelling an order is not safe. If the user says "book a ticket... oh no, wait, do not book it," we must never have booked it already. So, we use speculative execution only for tools that read data and do not change anything.

If we want to go deep into Tool use in Agents and how to Design a Real-Time Voice AI Agent, and to build an AI Coding Agent from scratch, we have it all in our AI and Machine Learning Program at Outcome School.

Technique 5: Make the tool itself faster

We must also make the tool itself faster wherever we can. Few common ways:

  • Cache the results: If many users ask for the same thing, like today's store timings, the agent keeps the answer for some time and reuses it instead of calling the tool again.
  • Load data at the start of the call: If the user is calling about their account, the agent can load the account details as soon as the call starts, before the user even asks.
  • Run tools in parallel: If the agent needs two tools, like the flight status and the weather at the destination, it runs both at the same time instead of one after the other.

Each of these removes some seconds from the wait, so there is less to hide.

What if the tool takes too long?

Sometimes, the tool is so slow that progress updates are not enough.

So, the agent must have a timeout, which is a time limit after which it stops waiting.

When the timeout is reached, the agent must not just go silent or say "Error". It must tell the user honestly and give a way forward. For example:

  • "The airline system is not responding right now. Can I send you the status by text message as soon as I get it?"
  • "This is taking longer than usual. Would you like me to keep trying, or can I help you with something else?"

Here, we can see that the agent turns a failure into a clear next step, so the user knows what is going on.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Putting it all together in code

Now, let's see a simple version of Technique 1, Technique 2, and the timeout together in Python. For the sake of understanding, we will keep it small.

import asyncio
import random

QUICK_LINES = ["Sure, let me check that.", "One moment, please.", "Let me look that up."]
PROGRESS_LINES = ["I am still checking.", "Almost there, thanks for waiting."]

async def run_tool_with_cover(tool, args, speak):
    # start the tool as a separate task, so we do not block
    task = asyncio.create_task(tool(**args))

    # Technique 1: speak a short line right away
    await speak(random.choice(QUICK_LINES))

    # Technique 2: progress updates every 4 seconds
    for line in PROGRESS_LINES:
        done, _ = await asyncio.wait({task}, timeout=4)
        if done:
            return task.result()
        await speak(line)

    # timeout: give the tool 4 more seconds, then stop waiting
    done, _ = await asyncio.wait({task}, timeout=4)
    if done:
        return task.result()
    task.cancel()
    return None

Here, we have:

  • QUICK_LINES and PROGRESS_LINES, which are the ready-made lines that the agent can speak.
  • speak, which is a function that sends a line to TTS and plays it to the user.
  • asyncio.create_task(...), which starts the tool as a separate task. We start it first, so the tool runs while the agent is speaking the short line.
  • asyncio.wait({task}, timeout=4), which waits for the tool for at most 4 seconds. If the tool finishes, we return the result right away. If not, we speak the next progress line.
  • At the end, if the tool still has not finished after about 12 seconds of waiting, we cancel it and return None.

Now, let's see how the agent uses it:

result = await run_tool_with_cover(check_flight_status, {"flight": "AI 101"}, speak)

if result is None:
    await speak("The airline system is not responding right now. "
                "Would you like me to try again?")
else:
    # give the result to the LLM, and it writes the final reply
    reply = await llm_reply_with_tool_result(result)
    await speak(reply)

Here, we can see that the agent always says something. If the tool is quick, the user hears one short line and then the answer. If the tool is slow, the user hears progress updates. If the tool is too slow, the user hears an honest message with a way forward.

Now we must have understood how a voice agent hides the delay of a slow tool call, and which technique fits which kind of tool.

Frequently Asked Questions

Should a voice agent speak a short line before every tool call?

No. If a tool usually returns in a fraction of a second, the agent can skip the short line. Otherwise, the user hears "One moment, please." followed at once by the answer, which sounds odd. The short line is useful when the tool takes a few seconds and the line would otherwise go silent.

Does speaking while the tool runs make the tool call faster?

No. A short line, progress updates, and talking while the tool runs in the background only hide the delay. They do not make it smaller. To cut the delay itself, the agent can start the tool early, cache the results, load data at the start of the call, and run tools in parallel.

Which tools are safe to start early in a voice agent?

Only the tools that read data and do not change anything, like checking a flight status. If the early result is not needed, the agent just throws it away. Tools that book a ticket, send money, or cancel an order must never start early, because the user can still change their mind before finishing the sentence.

What should a voice agent say if the user asks about a result that has not arrived yet?

It must say that it is still waiting. When the tool runs in the background, the LLM must not make up a result that has not arrived yet. The agent keeps track of which tools are still running, and it tells the user about the result only after the result is added to the conversation.

Why should the progress updates of a voice agent use different lines?

If the agent says the same line again and again during a long wait, it sounds repetitive. So, each progress update uses a different line, like "I am still checking" and then "Almost there, thanks for waiting." If a tool is always slow, the agent can also say at the start how long it usually takes.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.