How does a TTS model start speaking before the full sentence arrives?

A TTS model starts speaking before the full sentence arrives by reading the text in small pieces as it comes in, staying only a few words behind the text, and turning each piece into sound while the rest of the sentence is still being written.

In this blog, we will learn about how a TTS model starts speaking before the full sentence arrives. We will also see why waiting for the full sentence makes a voice assistant feel slow, why a TTS model wants to see the future words, how speaking sentence by sentence and word by word compare, how a small lookahead of a few words solves the problem, how newer models read text and speak at the same time, and where it works well and where it fails.

We will cover the following:

• The problem: the text arrives slowly
• Why does a TTS model want the full sentence?
• Approach 1: Wait for the full reply
• Approach 2: Speak sentence by sentence
• Approach 3: Speak word by word
• Approach 4: Speak with a small lookahead
• Approach 5: Models that read and speak at the same time
• A walkthrough with numbers
• Where it works well and where it fails

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

The problem: the text arrives slowly

TTS (Text-to-Speech) is a model that takes text as input and gives us spoken audio as output.

Let's say we are talking to a voice assistant. We ask a question. The assistant works like below:

Our voice
    ↓
ASR (speech to text)
    ↓
LLM (writes the reply)
    ↓
TTS (text to speech)
    ↓
Speaker

Here, the LLM (Large Language Model) does not write the whole reply at once. It writes the reply one token at a time. A token is a small piece of text, often a word or a part of a word.

So, the text reaches the TTS model a little at a time, like below:

"Sure,"
"Sure, the"
"Sure, the weather"
"Sure, the weather in"
"Sure, the weather in Delhi"
"Sure, the weather in Delhi is"
...

Now, the question is: when should the TTS model start speaking?

If it waits for the full reply, we sit in silence. If it starts too early, the voice sounds wrong.

Before jumping into the approaches, we must know why a TTS model wants the full sentence in the first place.

Why does a TTS model want the full sentence?

A human reader looks ahead while reading aloud. Our eyes are always a few words ahead of our mouth. A TTS model also needs this future context, for three reasons.

Reason 1: Pronunciation depends on the next words.

Consider the word "read".

• "I read books every day." Here, "read" sounds like "reed".
• "I read that book last year." Here, "read" sounds like "red".

When the TTS model sees only "I read", it cannot know which sound is correct. The answer is in the words that come later.

Reason 2: The tune of the sentence depends on the end.

The rise and fall of our voice is called prosody. It includes pitch, speed, pauses, and stress on words.

Consider these two sentences:

• "You are coming tomorrow."
• "You are coming tomorrow?"

The words are the same. Only the last mark is different. But, the second one is a question, so our voice rises at the end. The TTS model learns this only when it sees the question mark at the very end.

Reason 3: Numbers and short forms depend on context.

Consider "Dr." and "St.". "Dr." can mean "Doctor" or "Drive". And, a number like "2025" can be read as "two thousand twenty-five" or "twenty twenty-five". The TTS model needs the nearby words to decide.

Most TTS models are trained on full sentences, so they always see the future words. This is the reason they sound natural, and this is also the reason they are slow to start.

Now, let's see the approaches one by one.

Approach 1: Wait for the full reply

This is the simplest approach. We wait until the LLM finishes the whole reply, and then give the full text to the TTS model.

LLM writes the full reply (3 seconds)
    ↓
TTS creates the full audio (1 second)
    ↓
Audio starts playing

Here, the user hears nothing for about 4 seconds. In a conversation, this feels very slow. A normal pause between two people talking is usually well under a second.

Advantage: The voice sounds the most natural, because the TTS model sees everything.

Disadvantage: The silence before the first sound grows with the length of the reply. A long reply means a long wait.

The time from the end of our question to the first sound we hear is called time to first audio. This is the number we want to make small. We have a detailed blog on how to build a latency budget for a voice AI agent that explains how each step adds to the wait.

The issue with this approach is that the user waits for the whole reply. Let's see how the next approach solve this issue.

Approach 2: Speak sentence by sentence

Here, we wait only for the first full sentence. We keep collecting the text from the LLM in a buffer until a sentence is complete. As soon as we see a sentence-ending mark like ".", "?" or "!", we send that sentence to the TTS model.

Let's see the idea in code as below:

buffer = ""

for token in llm_stream():
    buffer += token
    if buffer.strip().endswith((".", "?", "!")):
        audio = tts.synthesize(buffer)
        play(audio)
        buffer = ""

Here, we have:

• llm_stream() gives us one token at a time from the LLM.
• buffer collects the tokens.
• When the buffer ends with a sentence-ending mark, we send it to tts.synthesize().
• We play the audio and empty the buffer for the next sentence.

For the sake of understanding, the code is kept simple. In a real system, play() runs in the background, so the loop keeps reading new tokens while the audio plays.

So, while the first sentence is playing, the LLM keeps writing the second sentence, and the next audio is usually ready by the time the first one ends.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Advantage: Each sentence is complete, so the model gets all the hints it needs for the pronunciation and the tune.

Disadvantage: We still wait for the full first sentence. If the first sentence is long, the wait is long.

Note: Real systems also split at commas when a sentence grows too long, but the wait still depends on how long the first piece is. Also, a simple rule like ours splits wrongly at short forms like "Dr.", so real systems use better sentence splitters.

This approach is very common in voice assistants, because it is simple and it works with any TTS model. We have a detailed blog on how sentence chunking feeds LLM output into TTS faster that explains this step by step.

The issue with this approach is that the time to first audio still depends on the length of the first sentence. Let's see how the next approach solve this issue.

Approach 3: Speak word by word

Now, let's go to the other extreme. We send every word to the TTS model the moment it arrives.

Arrives: "Sure,"     Speaks: "Sure,"
Arrives: "the"       Speaks: "the"
Arrives: "weather"   Speaks: "weather"

Advantage: The time to first audio is very small. We start speaking right after the first word.

Disadvantage: The voice sounds broken and robotic.

Why? Because each word is spoken alone, without any idea of what comes next. All three reasons that we learned earlier come back:

• "I read" is said with a guess, and the guess is often wrong.
• The voice does not rise for a question, because the question mark has not arrived yet.
• "Dr." is read without knowing whether it means "Doctor" or "Drive".

On top of this, each word gets its own little start and end, so there is no smooth flow between words.

The issue with this approach is that the TTS model sees no future at all. Let's see how the next approach solve this issue.

Approach 4: Speak with a small lookahead

Here comes the lookahead into the picture.

Lookahead means that before the TTS model speaks a word, it waits until it has seen a few more words after it.

So, we do not wait for the full sentence, and we do not speak each word alone. We stay just a few words behind the text, like a human reader whose eyes are a little ahead of the mouth.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Researchers from Baidu described this idea in the 2020 paper "Incremental Text-to-Speech Synthesis with Prefix-to-Prefix Framework". Incremental TTS means creating speech piece by piece as the text arrives. A prefix is the beginning part of a sentence. Means, the model creates the beginning of the audio from only the beginning of the text, and it keeps going as more text arrives. In their tests on English and Chinese, a lookahead of just 1 or 2 words gave speech that sounded close to full-sentence TTS.

Let's say our lookahead is 2 words. Then, it works like below:

Text seen so far:  "I"
Spoken so far:     (nothing yet)

Text seen so far:  "I read"
Spoken so far:     (nothing yet)

Text seen so far:  "I read that"
Spoken so far:     "I"

Text seen so far:  "I read that book"
Spoken so far:     "I read"

Text seen so far:  "I read that book last"
Spoken so far:     "I read that"

Here, we can see that the word "read" is spoken only after the model has seen "that book". Now, the model has more context to pick the right sound. But, the clearest hint here is "last year", which is 4 words away. A lookahead of 2 words does not reach it, so the model must still guess. We will come back to this in the failure cases.

One more thing to notice: the model does not throw away the audio it has already made. It keeps the earlier audio and continues smoothly from where it left off. So, the joins between words sound natural.

The best part is that the wait no longer depends on the length of the sentence. It depends only on the lookahead. Whether the sentence has 5 words or 50 words, we start speaking after the first few words.

Let me tabulate the differences between the first four approaches for your better understanding.

The issue with this approach is that the TTS model was trained on full sentences. When we feed it half sentences, it is seeing something they did not see during training, and this can hurt the quality. Let's see how the next approach solve this issue.

Approach 5: Models that read and speak at the same time

The newer idea is to train the model itself to take in text and give out audio at the same time. Such models are often called dual-streaming or bistreaming TTS models. These models handle two streams at once: text coming in and audio going out.

Before jumping into how they work, we must know about speech tokens. Many modern TTS models do not create sound waves directly. First, they create speech tokens, which are small codes, and each code stands for a tiny slice of sound. Then, a second part turns these codes into the actual sound wave, chunk by chunk, so the sound can start playing early.

Now, let's see two ways to do this.

Way 1: Interleaving text tokens and speech tokens.

CosyVoice 2, a TTS model from Alibaba, does this. During training, it mixes text tokens and speech tokens in a fixed ratio. In the paper, the ratio is 5 text tokens followed by 15 speech tokens. CosyVoice 2 makes 25 speech tokens for every second of audio, so 15 speech tokens are about 0.6 seconds of speech.

5 text tokens
    ↓
15 speech tokens
    ↓
next 5 text tokens
    ↓
next 15 speech tokens
    ↓
...

Here, the model learns a simple habit: read a little text, speak a little audio, read a little more, speak a little more. So, at the time of use, it does not need the full sentence. It needs only the next 5 text tokens to produce the next 15 speech tokens. When the model is ready for more text, it outputs a special filling token, which tells the system to bring in the next 5 text tokens.

Way 2: A fixed delay between the text and the audio.

Delayed Streams Modeling from Kyutai, an AI research lab in Paris, does this. It puts text and audio on the same clock, and the audio stream is kept a fixed number of steps behind the text stream. So, when the model speaks a moment of audio, it has already read a little bit of the text that comes after it.

This is the same lookahead idea from Approach 4. The difference is that the model is trained with this delay from the very start, so the half sentence is normal for it, not a surprise.

Advantage: The voice sounds natural, and the first audio comes quickly.

Disadvantage: We need a model that is trained this way. We cannot take any TTS model and get this behavior.

To learn Multimodal Generative Models and how to Design a Real-Time Voice AI Agent end to end, check out our AI and Machine Learning Program at Outcome School.

Now, let's see a walkthrough with numbers.

A walkthrough with numbers

Let's say the LLM writes 10 tokens per second, and the reply is:

"Sure, the weather in Delhi is sunny today, with a high of thirty degrees."

For the sake of understanding, let's treat every word as one token. This reply has 14 tokens. Let's also assume that the TTS model takes 0.2 seconds to create the first piece of audio once it has the text it needs.

Approach 1: Wait for the full reply.

• The LLM takes 14 / 10 = 1.4 seconds to write all 14 tokens.
• Then, the TTS model takes 0.2 seconds to create the first piece of audio.
• The first sound comes at about 1.4 + 0.2 = 1.6 seconds.

Approach 2: Sentence by sentence.

• This reply is one sentence, so this is the same as Approach 1.
• The first sound comes at about 1.6 seconds.

Approach 4: Lookahead of 2 words.

• To speak the first word "Sure,", we need to see 2 more words, "the weather". That is 3 tokens.
• The LLM takes 3 / 10 = 0.3 seconds to write them.
• The first sound comes at about 0.3 + 0.2 = 0.5 seconds.

Approach 5: Interleaving 5 text tokens with 15 speech tokens.

• The model needs the first 5 text tokens before it speaks.
• The LLM takes 5 / 10 = 0.5 seconds to write them.
• The first sound comes at about 0.5 + 0.2 = 0.7 seconds.

Here, we can notice that reading ahead by only a few tokens brings the first sound from 1.6 seconds down to 0.5 or 0.7 seconds. And, if the reply had 3 sentences, Approach 1 would get much slower, but Approach 4 and Approach 5 would still start at the same time.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Where it works well and where it fails

It works well when:

• The reply comes from an LLM, token by token, in a live conversation.
• The hints that a word needs are close to it, within the next few words.
• The text is plain and simple, without many numbers or short forms.

It fails or struggles when:

• The meaning of an early word depends on a word far ahead. A lookahead of 2 or 3 words cannot see that far, so the model must guess.
• The question mark arrives at the very end. With a short lookahead, the model learns that it is a question only near the end, so the earlier part of the sentence is already spoken in the tune of a normal statement.
• In some languages, like Hindi and Japanese, the verb often comes at the end of the sentence. The model sees the main action only at the end, and the tune of the whole sentence depends on it.
• The text has things like "2025", "Dr." or "$5.50" that often need the words after them to be read correctly.

So, there is always a balance between speed and quality. A bigger lookahead gives better speech but a longer wait. A smaller lookahead gives a faster start but more mistakes. We must choose the lookahead based on our use case.

Now we must have understood how a TTS model starts speaking before the full sentence arrives, why it wants the future words, and how a small lookahead lets it speak early while still sounding natural.

Frequently Asked Questions

What is time to first audio in a voice assistant?

Time to first audio is the time from the end of our question to the first sound we hear. This is the number we want to make small. When we wait for the full reply, it grows with the length of the reply. With a small lookahead, it depends only on the lookahead, not on the length of the sentence.

What is prosody in TTS?

Prosody is the rise and fall of our voice. It includes pitch, speed, pauses, and stress on words. For example, in "You are coming tomorrow?" our voice rises at the end because it is a question. A TTS model learns this only when it sees the question mark at the very end of the sentence.

What are speech tokens in a TTS model?

Speech tokens are small codes, and each code stands for a tiny slice of sound. Many modern TTS models do not create sound waves directly. First, they create speech tokens. Then, a second part turns these codes into the actual sound wave, chunk by chunk, so the sound can start playing early.

How many words of lookahead does a TTS model need?

Only a few. In the Baidu paper on incremental TTS, a lookahead of just 1 or 2 words gave speech that sounded close to full-sentence TTS in their tests on English and Chinese. A bigger lookahead gives better speech but a longer wait, and a smaller lookahead gives a faster start but more mistakes.

What is a filling token in CosyVoice 2?

A filling token is a special token that CosyVoice 2 outputs when it is ready for more text. It tells the system to bring in the next 5 text tokens. This fits the habit the model learns during training: read a little text, speak a little audio, read a little more, and speak a little more.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.