Outcome School

How does sentence chunking feed LLM output into TTS faster?

Authors
How does sentence chunking feed LLM output into TTS faster?

Sentence chunking feeds LLM output into TTS faster by cutting the reply into sentences while the LLM is still writing it, and sending each finished sentence to the TTS right away, so the voice starts speaking after the first sentence instead of after the full reply.

In this blog, we will learn about how sentence chunking feeds LLM output into TTS faster. We will also see why waiting for the full reply makes a voice agent slow, how a buffer collects tokens until a sentence is complete, how the chunker finds where a sentence ends, how it handles tricky cases like "Dr." and "3.5", how the first chunk is made smaller to start speaking even sooner, and where it works well and where it fails.

We will cover the following:

  • The problem: the voice waits for the full reply
  • What is sentence chunking?
  • Why not send every token to the TTS?
  • How does the chunker work?
  • Finding the end of a sentence
  • The tricky cases
  • Making the first chunk smaller
  • A walkthrough with numbers
  • What happens when the user interrupts?
  • Where it works well and where it fails

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

The problem: the voice waits for the full reply

A voice agent is a system that listens to us, thinks, and talks back. Most voice agents work as a pipeline of three parts:

  • STT (Speech-to-Text): It turns our voice into text.
  • LLM (Large Language Model): It reads that text and writes a reply.
  • TTS (Text-to-Speech): It turns the reply text into spoken audio.

The flow looks like below:

Our voice
    ↓
   STT
    ↓
   LLM
    ↓
   TTS
    ↓
Audio we hear

The LLM does not write the reply all at once. It writes one token at a time. A token is a small piece of text, usually a word or a part of a word. Tokens come out one after another, which is called streaming.

Let's say the LLM starts its reply like below:

"Sure, I can help with that. Your order was shipped yesterday. It will reach you by Friday. ..."

Suppose the full reply is long, and it takes the LLM 3 seconds to write all of it. If we wait for the full reply and only then send it to the TTS, the user hears nothing for those 3 seconds. Then, the TTS also needs time to make the audio. In a conversation, this silence feels very long.

But, the first sentence "Sure, I can help with that." is ready after a fraction of a second. The TTS could already start speaking it while the LLM is still writing the rest.

Sentence chunking solves this problem.

What is sentence chunking?

Sentence chunking is the step that collects the tokens coming out of the LLM, cuts them into complete sentences as soon as each sentence ends, and sends each sentence to the TTS one by one.

The piece of code that does this is called the chunker (some frameworks call it an aggregator or a sentence tokenizer).

With sentence chunking, the flow becomes like below:

LLM writes tokens
        ↓
Chunker collects tokens
        ↓
Sentence 1 is complete
        ↓
TTS speaks sentence 1

And while the TTS is speaking sentence 1, the LLM keeps writing sentence 2, and the chunker keeps collecting it. Means, the three parts now work at the same time instead of waiting for each other.

The finished sentences wait in a queue, so the TTS speaks them in the same order in which the LLM wrote them. The user now waits only for the first sentence, not for the full reply.

Why not send every token to the TTS?

Now, the question is: if smaller is faster, why not send every single token to the TTS the moment it arrives?

The answer is that a TTS needs some context to sound natural.

Let's understand this with an example. Consider the word "read". In "I read books every day" it sounds like "reed". In "I read that book yesterday" it sounds like "red". Here, the word that decides the sound comes later in the sentence. If the TTS has to speak "read" before it sees the rest, it can easily pick the wrong one.

The same is true for the tone of the voice. A yes-or-no question often goes up at the end, like "Is it ready?". A plain statement usually goes down at the end. If the TTS gets the words one by one, it does not know how the sentence will end, so it cannot plan the rise and fall of the voice. This rise and fall is called prosody.

So, we have two extremes:

  • Send the full reply: sounds natural, but the user waits too long.
  • Send every token: starts fast, but sounds broken.

A sentence sits in the middle. It is short enough to be ready quickly, and it carries enough context for the TTS to say it naturally. This is why the sentence is a common unit for chunking.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

How does the chunker work?

The chunker keeps a buffer, which is a holding area for text.

It follows these steps:

Step 1: A new token arrives from the LLM, and the chunker adds it to the buffer.

Step 2: The chunker checks whether the buffer now contains a complete sentence.

Step 3: If yes, it cuts that sentence out, sends it to the TTS, and keeps the leftover text in the buffer.

Step 4: If no, it waits for the next token.

Step 5: When the LLM finishes, whatever is left in the buffer is sent to the TTS as the last chunk, even if it does not end with a full stop.

Let's see the code for a simple chunker as below:

def chunk_sentences(token_stream):
    buffer = ""
    for token in token_stream:
        buffer += token
        # look for the end of a sentence
        while True:
            end = find_sentence_end(buffer)
            if end == -1:
                break
            sentence = buffer[:end].strip()
            buffer = buffer[end:]
            yield sentence  # send to TTS right away
    # LLM is done, send whatever is left
    if buffer.strip():
        yield buffer.strip()

Here, we have a buffer that grows as tokens arrive. After every token, we call find_sentence_end to check if a sentence has ended. If it has, we cut the sentence out and yield it, which means we hand it to the TTS immediately without stopping the loop. The leftover text stays in the buffer for the next sentence. At the end, we send the remaining text so that the last words are not lost.

Now, let's see how find_sentence_end works.

Finding the end of a sentence

The simplest rule is: a sentence ends at a full stop, a question mark, or an exclamation mark.

But, we must be careful. The chunker sees the text token by token. When it sees a full stop, it does not yet know what comes next. For example, "3." can still become "3.5". So, a good chunker waits for one more thing after the punctuation: a space, a new line, or the end of the reply. Only then it decides that the sentence has really ended.

Let's take an example. The tokens arrive like below:

"Sure" "," " I" " can" " help" " with" " that" "." " Your" ...

Here, we can see that:

  • When "." arrives, the buffer is "Sure, I can help with that." The chunker does not cut yet.
  • When " Your" arrives, the chunker sees that a space came after the full stop. Now, it cuts "Sure, I can help with that." and sends it to the TTS.
  • The buffer now holds " Your", and the chunker starts collecting the second sentence.

We can write find_sentence_end as below:

def find_sentence_end(text):
    for i, ch in enumerate(text):
        if ch in ".?!" and i + 1 < len(text) and text[i + 1].isspace():
            return i + 1  # cut right after the punctuation
    return -1  # no complete sentence yet

Here, we are looking for a full stop, question mark, or exclamation mark that is followed by a space. If we find one, we return the position just after the punctuation. If not, we return -1, which tells the chunker to wait.

This works perfectly for simple text.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

The tricky cases

Not every full stop ends a sentence. If the chunker cuts at the wrong place, the TTS speaks half a sentence, pauses, and then speaks the other half, which sounds unnatural.

Let's see the common tricky cases:

  • Abbreviations: "Dr. Sharma will call you." If we cut after "Dr.", the TTS says "Dr." alone with a pause, and then "Sharma will call you." as a new sentence.
  • Decimal numbers: "The price is 3.5 dollars." Here, the full stop in "3.5" is not followed by a space, so our simple rule already handles it.
  • Initials and short forms: "U.S.", "a.m.", "p.m." in a sentence like "The meeting is at 10 a.m. tomorrow."
  • Web addresses and emails: "Visit example.com for details." There is no space after the dot, so our simple rule already handles it.
  • Ellipsis: "Well... let me check." Three dots in a row are a pause, not always the end of a sentence.

To handle these, the chunker uses some extra rules:

  • It keeps a list of common abbreviations like "Dr.", "Mr.", "Mrs.", and "etc." and does not cut after them.
  • It waits for the next word and checks that it starts with a capital letter before cutting, because a new sentence usually starts with a capital letter.
  • It treats a group of dots as one piece of punctuation.

Our updated code with the abbreviation rule:

ABBREVIATIONS = {"dr.", "mr.", "mrs.", "ms.", "etc."}

def find_sentence_end(text):
    for i, ch in enumerate(text):
        if ch in ".?!" and i + 1 < len(text) and text[i + 1].isspace():
            last_word = text[:i + 1].split()[-1].lower()
            if last_word in ABBREVIATIONS:
                continue  # "Dr." is not the end of a sentence
            return i + 1
    return -1

Here, before cutting, we take the last word before the punctuation. If it is in our ABBREVIATIONS list, we skip it and keep looking. Now, "Dr. Sharma will call you." stays as one sentence.

The second problem is the opposite one: some sentences take a very long time to end. An LLM can write a long sentence with many commas, or a list without any full stops. If the chunker waits only for a full stop, the TTS gets nothing for a long time, and the user hears silence again.

To solve this, the chunker also has a maximum length. If the buffer gets too long without a sentence end, the chunker cuts at the last comma, semicolon, or colon in the buffer. These are natural pause points, so the speech still sounds fine.

Similarly, the chunker can have a minimum length. In the middle of a reply, a short sentence like "Ok." can be joined with the next sentence, because sending many tiny pieces to the TTS makes the speech choppy.

Making the first chunk smaller

The user is waiting in silence only until the first audio starts. After that, the TTS is busy speaking, and the LLM has time to stay ahead. So, the first chunk matters the most for speed. The later chunks only need to be ready before the previous audio finishes playing.

Let's say the LLM's first sentence is long:

"Sure, I checked your account and I can see that your order was shipped yesterday from our warehouse."

If we wait for the full stop, the user waits for all of these tokens. Instead, the chunker can cut the first chunk at the first comma: "Sure," goes to the TTS immediately. The TTS starts speaking while the rest of the sentence is still being written.

So, a common approach is to use two rules:

  • For the first chunk: cut early, at a comma or after a small number of words, to start speaking as soon as possible.
  • For the later chunks: cut at full sentences, to get the most natural voice.

The cost is that the short first piece has less context, so its tone can be slightly less natural.

To learn how to Design a Real-Time Voice AI Agent and the LLM Inference Performance Metrics that decide how fast it starts speaking, check out our AI and Machine Learning Program at Outcome School.

A walkthrough with numbers

The best way to learn this is by taking an example. The numbers below are just for the sake of understanding, real numbers depend on the LLM, the TTS, and the hardware.

Let's assume:

  • The LLM writes about 10 tokens every 0.1 seconds after its first token.
  • The first token comes out 0.3 seconds after the LLM gets the question.
  • The TTS takes 0.2 seconds to start playing audio after it gets a chunk, no matter how long the chunk is.
  • The reply is three sentences of about 10 tokens each.

Without chunking:

  • The LLM finishes all 30 tokens at 0.3 + 0.3 = 0.6 seconds.
  • The TTS gets the full reply and starts playing at 0.6 + 0.2 = 0.8 seconds.
  • The user waits 0.8 seconds for the first sound.

With sentence chunking:

  • The first sentence is complete at about 0.3 + 0.1 = 0.4 seconds.
  • The TTS starts playing it at 0.4 + 0.2 = 0.6 seconds.
  • The user waits 0.6 seconds for the first sound.

Here, the gain looks small because our reply is short and the LLM is fast. Now, suppose the reply is ten sentences long. Without chunking, the LLM finishes 100 tokens at 0.3 + 1.0 = 1.3 seconds, and the user hears the first sound at 1.5 seconds. With chunking, the user still hears the first sound at 0.6 seconds. Means, the wait no longer grows with the length of the reply. It depends only on the first sentence.

For the ten-sentence reply, we can see the overlap in the diagram below:

Time (seconds)   LLM                         TTS and speaker
0.3 to 0.4       writes sentence 1           waiting
0.4 to 0.5       writes sentence 2           makes audio for sentence 1
0.5 to 0.6       writes sentence 3           makes audio for sentence 1
0.6 to 1.3       writes sentences 4 to 10    plays sentence 1, then 2, and so on

And while the first sentence is playing, which takes a couple of seconds of speech, the LLM is usually done writing the next sentences. So, as long as the LLM writes faster than the voice speaks, the TTS does not run out of text, and there are no gaps between the sentences.

We have a detailed blog on how to build a latency budget for a voice AI agent that explains how each step adds to the wait.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

What happens when the user interrupts?

In a real conversation, the user can start talking while the agent is still speaking. This is called barge-in.

When this happens, the agent must stop speaking immediately. With sentence chunking, there are several places where the old reply is still waiting:

  • Tokens inside the chunker buffer.
  • Sentences sent to the TTS but not yet turned into audio.
  • Audio made by the TTS but not yet played.

So, on interruption, the agent must clear all of them: empty the buffer, cancel the pending TTS requests, and stop the audio player. If we forget the buffer, the old reply leaks out after the user has started a new turn.

Chunking also helps here. Because the reply is already split into sentences, the agent can track which sentences were fully played before the interruption. It can save only those in the conversation history, so that the LLM knows what the user actually heard.

Where it works well and where it fails

Where it works well:

  • Long replies, because the wait depends only on the first sentence and not on the full reply.
  • TTS models that take a full sentence as input and do not accept a live stream of text.

Where it fails:

  • Replies with many abbreviations, numbers, web addresses, or code, where full stops do not mean the end of a sentence. The voice then pauses in odd spots.
  • Very long sentences without a full stop, where the chunker has to wait or fall back to cutting at commas.
  • Replies written in markdown, like bullet points and headings. Symbols like * and # must be cleaned before the TTS, or it reads them out loud.
  • Languages that do not use the same punctuation, such as Chinese or Japanese, which use their own full stop symbol 。 and usually no space after it. Our simple rule never finds a sentence end there, so the chunker needs rules for each language.
  • Each chunk is spoken on its own, so the TTS does not see the next sentence. The tone between two sentences can sometimes sound a little disconnected.

Some newer TTS models accept text as a live stream and keep the context across chunks by themselves. With them, the chunker can send smaller pieces without losing the natural flow. But even then, the idea stays the same: do not wait for the full reply, start speaking as soon as there is enough text to say it well.

Now we must have understood how sentence chunking feeds LLM output into TTS faster, how the chunker finds the end of a sentence, how it handles the tricky cases, and why the first chunk matters the most.

Frequently Asked Questions

Why does the chunker wait for a space after a full stop?

Because a full stop alone does not prove that the sentence has ended. The chunker sees the text token by token, so "3." can still become "3.5". A good chunker waits for one more thing after the punctuation, a space, a new line, or the end of the reply, and only then decides that the sentence has really ended.

What happens to the text left in the buffer when the LLM finishes?

It is sent to the TTS as the last chunk. When the LLM finishes, whatever is left in the buffer goes to the TTS even if it does not end with a full stop. This makes sure that the last words of the reply are not lost.

What is prosody and why does it need a full sentence?

Prosody is the rise and fall of the voice. A yes-or-no question often goes up at the end, and a plain statement usually goes down at the end. If the TTS gets the words one by one, it does not know how the sentence will end, so it cannot plan this rise and fall.

Do we still need sentence chunking with TTS models that accept streaming text?

The idea stays the same. Some newer TTS models accept text as a live stream and keep the context across chunks by themselves. With them, the chunker can send smaller pieces without losing the natural flow. But we still do not wait for the full reply, we start speaking as soon as there is enough text to say it well.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.