How does a voice agent truncate its memory to only what the user actually heard?

A voice agent truncates its memory by finding out how much of its reply was actually played through the speaker before the user interrupted, and then cutting its saved reply down to only those words, so that its memory matches what the user really heard.

In this blog, we will learn about how a voice agent truncates its memory to only what the user actually heard. We will also see what the memory of a voice agent is, why the agent prepares its reply faster than it speaks it, what goes wrong when the memory is not cut, how the agent measures what was played and turns that time into words, how a speech-to-speech API does the same thing, and which edge cases the agent must handle.

We will cover the following:

• What is the memory of a voice agent?
• The problem: the agent remembers more than the user heard
• Why the reply is ready before it is spoken
• What goes wrong if we do not truncate
• How the agent knows what the user actually heard
• Turning played time into words
• Truncating the memory step by step
• A complete walkthrough
• How a speech-to-speech API does this
• Edge cases

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is the memory of a voice agent?

A voice agent is a system that talks with us using voice. We speak, it listens, it thinks, and it speaks back.

Many voice agents work as a pipeline of three parts:

• Speech-to-Text (STT): It turns the user's voice into text.
• LLM (Large Language Model): It reads the text and writes a reply.
• Text-to-Speech (TTS): It turns the reply text into voice.

We have a detailed blog on how to design a real-time voice AI agent that explains this pipeline end to end.

Now, the LLM does not remember anything on its own. Every time it writes a reply, we must send it the full conversation so far. This saved conversation is the memory of the voice agent. Many people also call it the conversation history or the context.

The memory is a list of turns, like below:

User:  Hi, I want to book a table for tonight.
Agent: Sure. For how many people?
User:  Four people.

Each time the user speaks, the agent adds the user's words to this list. Each time the agent replies, it adds its own reply to this list. Then, for the next reply, the LLM reads the whole list.

So, the memory is the agent's record of what has been said so far. And the LLM writes the next reply based on this record. If the record is wrong, the next reply is wrong too.

The problem: the agent remembers more than the user heard

The best way to learn this is by taking an example.

Let's say the agent replies to "Four people." with a long reply:

"We have three tables free tonight. One at 7 PM, one at 8 PM, and one at 9 PM. Which one would you like?"

The agent starts speaking. The user hears "We have three tables free tonight. One at 7 PM," and then the user cuts in and says:

"7 PM is perfect."

This is called barge-in. The user interrupts, and the agent stops speaking right away.

The LLM had already written the full reply. And a simple agent saves the full reply in its memory. So, the memory now looks like below:

Agent: We have three tables free tonight. One at 7 PM, one at 8 PM,
       and one at 9 PM. Which one would you like?
User:  7 PM is perfect.

The user never heard "one at 8 PM, and one at 9 PM. Which one would you like?" But the agent believes the user heard all of it.

Now, the memory of the agent and the experience of the user do not match. This mismatch is the problem we must solve.

Why the reply is ready before it is spoken

Before jumping into the solution, we must know why the agent has more words than it spoke.

The reason is speed. The LLM writes text much faster than a human speaks. A full reply of 30 words can be written in a second or two. But speaking those 30 words out loud takes around 12 seconds.

The TTS is also fast. It often creates the audio faster than the audio plays. So, the audio is sent ahead and waits in a queue, which is called a buffer, on the user's device or in the phone connection. The speaker plays this buffer at normal human speed.

Let's see this as a diagram:

+-------------------------------+
|  LLM writes the full reply    |
|  in about 1 second            |
+-------------------------------+
                ↓
+-------------------------------+
|  TTS creates all the audio    |
|  in about 2 seconds           |
+-------------------------------+
                ↓
+-------------------------------+
|  Audio waits in the buffer    |
+-------------------------------+
                ↓
+-------------------------------+
|  Speaker plays the audio      |
|  in about 12 seconds          |
+-------------------------------+

Here, we can see that the text and the audio are ready long before the speaker finishes playing. In real systems, these steps overlap, because the text and the audio are streamed piece by piece. But creating is still much faster than playing. So, at the moment of barge-in, there is always a gap between what was generated and what was played.

Only the played part reached the user.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

What goes wrong if we do not truncate

If the agent keeps the full reply in its memory, the next reply goes wrong in many ways.

Problem 1: The agent assumes the user knows something. In our example, the agent thinks it has already offered 8 PM and 9 PM. Later, if the user changes the plan and asks, "Do you have anything later?", the agent replies, "As I said, we have 8 PM and 9 PM." But the user never heard that. The agent sounds confused.

Problem 2: The agent misreads the user's answer. Suppose the agent was saying, "Your total is 500 rupees. Shall I confirm the order?" The user interrupts after "Your total is 500 rupees" and says "Yes." The user only meant "Yes, I understand the total." But the agent's memory contains the question "Shall I confirm the order?" So, the agent treats "Yes" as a confirmation and places the order. This is a costly mistake.

Problem 3: The agent skips information. Suppose the agent planned to say, "Please bring an ID card with you." The user interrupted before that sentence. The agent thinks it already said it, so it does not repeat it. The user never gets that information at all.

So, the memory must hold only what the user actually heard. Here comes truncation into the picture.

How the agent knows what the user actually heard

Truncation means cutting the agent's last reply in the memory so that it ends exactly where the user stopped hearing it.

To do this, the agent must answer one question: how much of the audio was played before the barge-in?

The answer comes from the playback side. The part of the system that plays the audio keeps track of time. When the reply starts playing, it notes the start time. When the user barges in, the agent stops the audio and notes the stop time.

The difference between the two gives the played duration:

played duration = stop time - start time

For example, if the reply started playing 12.0 seconds into the call and stopped at 15.2 seconds, then the played duration is 3.2 seconds, which is 3200 milliseconds.

Note: The played duration must be measured at the point of playback, not at the moment the audio was sent. Audio that was sent but was still waiting in the buffer was never heard.

Now, we know how many milliseconds the user heard. But the memory is text, not milliseconds. So, the next question is: which words were spoken in those 3200 milliseconds?

Turning played time into words

Here comes the word timestamp into the picture.

Many TTS systems can return a timestamp for each word they speak. A timestamp tells us at what time a word starts in the audio. Some TTS systems give timestamps for each character instead, and we can build word timestamps from them.

For our reply, the TTS returns timestamps like below. Here, only the first 14 words are shown for the sake of understanding.

Now, the played duration is 3200 milliseconds. So, we keep every word that started before 3200 milliseconds. The words "We have three tables free tonight. One at 7 PM," started before 3200 milliseconds. The word "one" starts at 3400 milliseconds, which is after the stop. So, the user did not hear it.

This gives us the truncated reply:

"We have three tables free tonight. One at 7 PM,"

When a TTS system does not give timestamps, the agent can estimate. It assumes that the voice speaks at a steady speed. For example, if the full audio is 10 seconds long and 3 seconds were played, the agent keeps about the first 30 percent of the characters of the reply. This estimate is less accurate, but it is much better than keeping the full reply.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Truncating the memory step by step

Now that we have learned how to turn played time into words, let's put all the pieces together. The agent follows these steps when the user barges in.

Step 1: The agent stops the audio right away and clears the buffer, so that no more words are played. It also stops the LLM and the TTS from creating any more of the reply.

Step 2: The playback side reports the played duration, for example 3200 milliseconds.

Step 3: The agent looks at the word timestamps and keeps only the words that started before the played duration.

Step 4: The agent replaces its last reply in the memory with the truncated reply.

Step 5: The agent adds the user's new words to the memory, and the LLM writes the next reply from this corrected memory.

Let's see the code for the truncation as below:

def truncate_reply(words, played_ms):
    # words is a list of (word, start_ms) from the TTS
    heard = []
    for word, start_ms in words:
        if start_ms < played_ms:
            heard.append(word)
        else:
            break
    return " ".join(heard)


def on_barge_in(memory, words, played_ms):
    heard_text = truncate_reply(words, played_ms)
    # replace the last agent reply with only what was heard
    memory[-1] = {"role": "assistant", "content": heard_text}
    return memory

Here, the memory is a list of messages, where each message has a role and a content. And we have two functions:

• truncate_reply goes through the words in order. It keeps a word only if that word started before the played duration. As soon as it finds a word that started after the stop, it breaks out of the loop, because every later word also started after the stop.
• on_barge_in calls truncate_reply to get the heard text, and then it replaces the last agent reply in the memory with only that heard text.

If we call truncate_reply with the timestamps from our table and played_ms as 3200, it will return the following:

We have three tables free tonight. One at 7 PM,

This is how the agent makes its memory match what the user actually heard.

Some agents also add a marker at the end of the truncated reply, such as "[interrupted by the user]". This tells the LLM that its last reply did not finish. Then, the LLM knows that it must not assume the rest was said.

Some agents solve the same problem from the other side. They do not save the full reply first and cut it later. Instead, they add words to the memory only as those words are played. At barge-in, the memory already holds only the played words.

To master Memory in Agents, Context Engineering, and how to Design a Real-Time Voice AI Agent, check out our AI and Machine Learning Program at Outcome School.

A complete walkthrough

First, the user says, "Four people." The STT turns this into text, and the agent adds it to the memory.

Then, the LLM writes the full reply: "We have three tables free tonight. One at 7 PM, one at 8 PM, and one at 9 PM. Which one would you like?" The TTS turns it into audio along with word timestamps.

After that, the speaker starts playing the audio. At 3200 milliseconds, the user says, "7 PM is perfect."

The agent detects the barge-in. It stops the audio and clears the buffer. The playback side reports that 3200 milliseconds of audio were played.

The agent looks at the word timestamps and keeps only "We have three tables free tonight. One at 7 PM," in the memory.

Finally, the agent adds "7 PM is perfect." to the memory. The memory now looks like below:

User:  Four people.
Agent: We have three tables free tonight. One at 7 PM,
User:  7 PM is perfect.

The LLM reads this and replies, "Great. Let me book a table for four people at 7 PM tonight."

It works perfectly. The agent does not believe that it has offered 8 PM and 9 PM, so it can offer them later as new information.

How a speech-to-speech API does this

Now, it's time to learn about speech-to-speech models.

A speech-to-speech model takes the user's audio in and gives the agent's audio out directly, without a separate STT step and TTS step. The same problem exists here too. The model creates audio faster than it plays, so at barge-in, the model has generated more than the user heard.

For example, the OpenAI Realtime API provides the conversation.item.truncate event for this. The agent sends three things:

• item_id: which reply to cut.
• content_index: which audio part of that reply to cut.
• audio_end_ms: how many milliseconds of the audio were played.

The API then cuts the audio of that reply after audio_end_ms. As per the official documentation, it also deletes the server-side text transcript of that reply, so that there is no text in the context that the user has not heard. If audio_end_ms is longer than the actual audio, the API returns an error.

Here, we can notice that when our code receives the audio and plays it, the API does not know by itself how much audio was played. Only the playback side knows that. This is why the agent must measure the played duration and send it.

The idea is the same. The agent measures what was played, and then the memory is cut to match it, either by our code or by the API.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Edge cases

The agent must handle a few cases carefully.

A word was cut in the middle. Suppose the user barges in while the agent is saying "tonight". The user heard "toni". One common choice is to keep a word if it started before the stop, because the user heard at least a part of it. Some agents keep a word only if it finished before the stop. The first choice keeps more words. The second choice never saves a word that the user did not fully hear.

The interruption was not real. Sometimes the agent pauses because of a cough, a background noise, or a short "uh-huh". If the agent decides that it was not a real interruption and continues the same reply, it must not truncate the memory. It truncates only when the reply actually stopped for good.

The playback time is not exact. On a phone call, the audio passes through the network and the phone line before it reaches the user. So, the time when the agent thinks audio was played and the time when the user heard it can be a little different. The agent must measure as close to the user as possible.

The reply had a tool action. Sometimes the LLM recommends an action, such as booking the table, inside the same reply. If the user interrupts before hearing the confirmation, the agent must check whether that action already ran. The memory must hold the result of the action, even if the user did not hear the sentence about it. Then, in the next reply, the agent can tell the user about it.

Now we must have understood how a voice agent truncates its memory to only what the user actually heard, why it needs to do it, and how it turns played time into words.

Frequently Asked Questions

Can a voice agent truncate its memory without word timestamps?

Yes. When the TTS system does not give word timestamps, the agent can estimate. It assumes the voice speaks at a steady speed and keeps the same share of characters as the share of audio that was played. For example, if 3 of 10 seconds were played, it keeps about the first 30 percent of the reply. This is less accurate, but much better than keeping the full reply.

Should the agent keep a word that was cut in the middle?

It depends on the choice the agent makes. One common choice is to keep a word if it started before the stop, because the user heard at least a part of it. Some agents keep a word only if it finished before the stop. The first choice keeps more words, and the second choice never saves a word that the user did not fully hear.

Why should the played duration be measured at playback and not when the audio was sent?

Because audio that was sent but was still waiting in the buffer was never heard. The TTS creates audio faster than the speaker plays it, so a lot of audio can be sent ahead and wait in the buffer. Only the part that was actually played reached the user, so the agent must measure at the point of playback, as close to the user as possible.

Should the agent truncate its memory when the user only coughs or says "uh-huh"?

No. If the agent decides that a cough, a background noise, or a short "uh-huh" was not a real interruption and continues the same reply, it must not truncate the memory. It truncates only when the reply actually stopped for good.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.