Outcome School

How does speculative LLM generation start replying before the user finishes?

Authors
How does speculative LLM generation start replying before the user finishes?

Speculative LLM generation is a technique where a voice agent starts preparing its reply while the user is still finishing their sentence, keeps that reply hidden, and plays it only if the user's final words match what the reply was based on.

In this blog, we will learn about how speculative LLM generation starts replying before the user finishes. We will also see where the waiting time goes in a normal voice agent, how the agent makes an early guess from the words it has so far, what happens when the guess is right or wrong, how it can prepare the voice early too, what it costs, and when to use it.

We will cover the following:

  • The normal way a voice agent replies
  • Where the waiting time goes
  • What is speculative LLM generation?
  • How speculative LLM generation works step by step
  • What happens when the guess is right
  • What happens when the guess is wrong
  • How does the agent know that the words changed?
  • Preparing the voice early too
  • The cost of speculative LLM generation
  • When to use it and when not to

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

The normal way a voice agent replies

Before jumping into speculative LLM generation, we must know how a normal voice agent replies.

A voice agent is a system that we can talk to with our voice, and it talks back with a voice. Many voice agents work as a chain of three parts:

  • Speech-to-Text (STT): It listens to the user's voice and turns it into text.
  • LLM (Large Language Model): It reads that text and writes a reply.
  • Text-to-Speech (TTS): It turns the reply text back into a voice.

There is one more part that works next to the STT. It decides when the user has finished speaking. We call this end-of-turn detection. It looks at things like silence and the words spoken so far, and then it says, "Okay, the user is done. Now it is our turn."

So, the normal flow looks like below:

User speaks
     ↓
STT turns voice into text
     ↓
End-of-turn detection waits and says "user is done"
     ↓
LLM starts writing the reply
     ↓
TTS turns the reply into voice
     ↓
User hears the reply

Here, we can notice that the LLM does not start until the end-of-turn detection says "user is done". Everything waits in a line.

Where the waiting time goes

Let's say the user says, "What is the weather in Delhi today?"

Now, the agent cannot reply the very moment the user stops. It must first be sure that the user has really finished. It is possible that the user is only taking a breath and will continue with "...and tomorrow?" So, the end-of-turn detection waits for a short period of silence before it says "user is done".

After that, the LLM needs some time to produce the first few tokens of the reply. A token is a small piece of text, like a word or a part of a word. Then, the TTS needs some time to produce the first bit of audio.

So, the user waits for all of these, one after the other:

  • The silence wait of end-of-turn detection
  • The time the LLM takes to produce its first tokens
  • The time the TTS takes to produce its first audio

In a real conversation between two people, the gap between one person stopping and the other person starting is usually well under a second. When a voice agent adds up all these waits, the gap becomes long, and the conversation feels slow and robotic. We have a detailed blog on how to build a latency budget for a voice AI agent that explains how each of these waits adds up.

The silence wait and the LLM time happen one after the other. Now, the question is: can we make them happen at the same time?

The answer is yes. Here comes speculative LLM generation into the picture.

What is speculative LLM generation?

Speculative LLM generation means that the agent sends the user's words to the LLM before it is sure that the user has finished speaking, and keeps the reply ready but unplayed.

The word speculative means "based on a guess". The agent guesses that the words it has right now are the final words of the user. It starts the work based on that guess. Later, when the end-of-turn detection confirms the turn has ended, the agent checks whether its guess was correct.

  • If the guess was correct, the reply is already ready, and the agent plays it at once.
  • If the guess was wrong, the agent throws away the early reply and starts again with the correct words.

Let's say we are at a tea shop that we visit every day. As soon as we walk in and say, "One cup of...", the shop owner already starts boiling the tea, because that is what we order most of the time. If we finish with "...tea", the tea comes out very fast. If we finish with "...coffee", he puts that tea aside and starts the coffee. He wastes one cup of tea, but most days, we get our tea much faster.

This is exactly what speculative LLM generation does with the reply. Some of this work gets thrown away, but most of the time, the user gets the reply faster.

Note: The term "speculative decoding" is a different idea. Speculative decoding makes the LLM itself produce tokens faster by letting a smaller, faster model guess a few tokens ahead. Here, we are talking about starting the LLM early in a voice agent, before the user's turn is confirmed. Some voice agent frameworks call this idea preemptive generation.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

How speculative LLM generation works step by step

The best way to learn this is by taking an example.

Let's say the user says, "Book a table for two at 8 PM."

The STT does not wait for the whole sentence. It keeps sending text while the user is still speaking. We call these early pieces of text interim transcripts (or partial transcripts). When the STT is confident about a piece of speech, it sends a final transcript for it.

Step 1: The user is speaking. The STT sends interim text like "Book a", then "Book a table", then "Book a table for two".

Step 2: The user says "...at 8 PM" and stops. The STT sends the final transcript "Book a table for two at 8 PM."

Step 3: The agent does not wait for the end-of-turn detection. It sends "Book a table for two at 8 PM." to the LLM right away. This is the speculative request.

Step 4: The LLM starts writing the reply: "Sure, let me check if a table for two is free at 8 PM." The agent stores this reply in a holding area. We call this holding area a buffer. Nothing is played to the user yet.

Step 5: At the same time, the end-of-turn detection is still waiting through its silence period.

Step 6: The silence wait ends, and the end-of-turn detection says "user is done". By this time, the first part of the reply is already in the buffer.

Step 7: The agent compares the words it used for the speculative request with the final words of the turn. They are the same. So, the agent plays the reply from the buffer at once.

We can see the two timelines one after the other as below:

Normal agent:

User stops
     ↓
Silence wait
     ↓
End of turn confirmed
     ↓
LLM starts
     ↓
First LLM tokens ready
     ↓
TTS makes the first audio
     ↓
Reply starts playing


Speculative agent:

User stops
     ↓
LLM work and silence wait run at the same time
     ↓
End of turn confirmed, first LLM tokens already in the buffer
     ↓
TTS makes the first audio
     ↓
Reply starts playing

Here, we can see that in the speculative agent, the LLM work and the silence wait overlap. The time that the LLM needs gets hidden inside the time that the agent was going to wait anyway.

This is how speculative LLM generation starts replying before the user finishes.

What happens when the guess is right

When the guess is right, the final words match the words that were sent to the LLM, so the reply in the buffer is exactly the reply the agent would have produced anyway.

Means, the user gets the same answer, only faster.

The saving can be up to the time the LLM needs to produce its first tokens, because that work now runs inside the silence wait. In many voice agents, that is one of the biggest waits in the whole chain. So, hiding it makes the agent feel much more natural.

Let's take some numbers just for the sake of understanding. Assume that the silence wait is 500 milliseconds, and the LLM needs 400 milliseconds to produce its first tokens.

  • Normal agent: 500 + 400 = 900 milliseconds before the TTS can even start.
  • Speculative agent: The 400 milliseconds of LLM work run inside the 500 milliseconds of silence wait. So, the TTS can start after 500 milliseconds.

Here, we can see that the agent saves 400 milliseconds on every turn where the guess is right. If the LLM needs more time than the silence wait, the saving is equal to the silence wait, because only that part can overlap.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

What happens when the guess is wrong

Now, let's see the other case.

Let's say the user says, "Book a table for two..." and takes a pause to think. The STT sends a final transcript for these words, and the agent treats this pause as a possible end. It sends "Book a table for two" to the LLM. The LLM starts writing: "Sure, for what time?"

But the user continues: "...at 8 PM, near the window."

Now, the words have changed. The early reply "Sure, for what time?" is wrong for the new words. If the agent plays it, the user will be confused, because they have already told the time.

So, as soon as the new words arrive from the STT, the agent does the following:

  • It cancels the running LLM request, so it stops producing more tokens.
  • It throws away everything in the buffer.
  • When the STT sends the final transcript for the full words "Book a table for two at 8 PM, near the window.", it sends them to the LLM as a fresh speculative request.

The user never hears the wrong reply. The early reply was only in the buffer. It was never played.

In this case, the first request is wasted, but the fresh request still starts before the end of the turn is confirmed. So, the agent loses nothing in correctness. The only loss is the wasted LLM work of the first request.

How does the agent know that the words changed?

The agent keeps a copy of the exact words it sent with the speculative request. When the end-of-turn detection says "user is done", the agent compares that copy with the final words of the turn.

  • If both are the same, the agent uses the buffered reply.
  • If they are different, the agent throws the buffered reply away and makes a new request.

Some agents also look at the other things that were sent with the request, not only the words. For example, the conversation history and the list of tools that the LLM can recommend. A tool is a function that our agent code can run, like checking a booking system. If any of these changed in between, the early reply is no longer valid, and it must be thrown away too.

Note: The agent must never play a speculative reply until the end of the turn is confirmed. The whole idea is safe only because the early reply stays hidden until the check passes.

We have a complete program on Tool use in Agents, Agent Architecture, and how to Design a Real-Time Voice AI Agent - check out our AI and Machine Learning Program at Outcome School.

Preparing the voice early too

Now, it's time to learn about starting the voice early.

We can apply the same idea to the TTS. As soon as the LLM produces the first sentence of the speculative reply, the agent can send that sentence to the TTS and keep the produced audio in the buffer too.

Then, when the end of the turn is confirmed and the words match, the agent does not even need to wait for the TTS. The first audio is already ready, and it starts playing immediately.

If the words do not match, the agent throws away both the buffered text and the buffered audio.

Advantage: The reply can start playing almost the very moment the turn is confirmed.

Disadvantage: If the guess is wrong, the agent wastes both LLM work and TTS work.

This is why many setups run only the LLM early by default, and run the TTS early only when the extra speed is worth the extra waste.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

The cost of speculative LLM generation

Speculative LLM generation has a cost. Every wrong guess is a request that we pay for but never use.

Let's say the user pauses three times in the middle of a long sentence. The agent can end up sending four speculative requests for one single turn, one for each pause and one for the real end, and only the last one gets used. If our LLM provider charges per token, those thrown-away tokens still cost money. The same is true for TTS.

The other cost is load. More requests mean more work for the LLM servers. If many users talk at the same time, this extra load can slow down replies for everyone.

So, the agent must balance two things:

  • Starting early, so that the reply is fast.
  • Not starting too often, so that the waste stays low.

A common way to keep the waste low is to start the speculative request only on a final transcript from the STT, not on every interim transcript. Interim transcripts change very often while the user speaks, so starting on each of them would cause a lot of thrown-away work.

When to use it and when not to

Speculative LLM generation works well when:

  • The users speak in short, clear sentences, like booking, ordering, and simple questions.
  • Fast replies matter a lot, like in customer support calls.
  • The extra cost of some wasted requests is acceptable.

It works less well when:

  • The users speak in long sentences with many pauses, like telling a story or dictating a message, because the guess keeps going wrong and most of the work gets thrown away.
  • The LLM request is very expensive, and every wasted request hurts the budget.

One more thing to notice. A speculative request must only prepare a reply. It must never take an action in the real world, because the guess can still turn out to be wrong.

Let's say the speculative reply includes a tool call to place an order. The LLM only recommends the tool call, and our agent code runs it. So, the agent must hold that tool call and run it only after the turn is confirmed, never during the speculative step.

Now we must have understood how speculative LLM generation starts replying before the user finishes, how it hides the LLM time inside the silence wait, what happens when the guess is right or wrong, and when to use it.

Frequently Asked Questions

What is the difference between speculative LLM generation and speculative decoding?

They solve different problems. Speculative decoding makes the LLM itself produce tokens faster by letting a smaller, faster model guess a few tokens ahead. Speculative LLM generation starts the LLM early in a voice agent, before the end of the user's turn is confirmed, so that the LLM time overlaps with the silence wait.

How much time can speculative LLM generation save?

It can save up to the time the LLM needs to produce its first tokens, because that work runs inside the silence wait. If the LLM needs more time than the silence wait, the saving is equal to the silence wait, because only that part can overlap. For example, with a 500 millisecond silence wait and 400 milliseconds of LLM work, the agent saves 400 milliseconds.

Should the agent start a speculative request on every interim transcript?

No. A common way to keep the waste low is to start the speculative request only on a final transcript from the STT. Interim transcripts change very often while the user speaks, so starting on each of them would cause a lot of thrown-away work and extra cost.

Can a speculative request run a tool call?

No. A speculative request must only prepare a reply, because the guess can still turn out to be wrong. The LLM only recommends a tool call, like placing an order, and our agent code runs it. So, the agent holds that tool call and runs it only after the end of the turn is confirmed.

What is preemptive generation in a voice agent?

Preemptive generation is the name some voice agent frameworks use for the same idea as speculative LLM generation. The agent sends the user's words to the LLM before it is sure the user has finished, keeps the reply in a buffer, and plays it only if the final words match.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.