What are audio frames and chunk sizes in a real-time voice pipeline?

An audio frame is a small, fixed slice of sound, usually 10 or 20 milliseconds long, and the chunk size is how much of that sound we pass from one step of a voice pipeline to the next at one time.

In this blog, we will learn about audio frames and chunk sizes in a real-time voice pipeline. We will also see how sound becomes numbers, how those numbers are grouped into frames, how to calculate the size of a frame, how chunk size affects both delay and overhead, why every part of the pipeline wants a different size, and how to pick the right chunk size for our use case.

We will cover the following:

• How sound becomes numbers
• What is an audio frame?
• How to calculate the size of a frame
• What is a chunk size?
• How chunk size affects delay
• The cost of very small chunks
• Different parts want different sizes
• How to pick the right chunk size

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

How sound becomes numbers

Before jumping into audio frames, we must know how a computer stores sound.

Sound is a wave in the air. When we speak into a microphone, the microphone turns that wave into an electrical signal. But a computer cannot store a smooth wave. It can only store numbers.

So, the computer measures the height of the wave again and again, very quickly. Each measurement is called a sample. A sample is just one number that tells the height of the wave at that exact moment.

How many samples do we take every second? This is called the sample rate.

• 8 kHz means 8,000 samples every second. Regular phone calls use this.
• 16 kHz means 16,000 samples every second. Many speech recognition models use this.
• 24 kHz or 48 kHz means 24,000 or 48,000 samples every second. Many voice generation models and video calling apps use these.

Each sample also takes some space in memory. The most common format for raw audio is 16-bit PCM. PCM is the plain, uncompressed way of storing samples one after another, and 16-bit means each sample takes 2 bytes.

Let's say we record 1 second of speech at 16 kHz. We get 16,000 numbers. Each number takes 2 bytes. So, 1 second of audio takes 32,000 bytes.

What is an audio frame?

Now, the next big question is: how do we send this stream of numbers to the other parts of our system?

We cannot send one sample at a time. That would mean 16,000 tiny messages every second. And we cannot wait for the user to finish talking before sending anything, because then our voice agent, the AI that talks with the user, will feel slow.

So, here comes the audio frame to the rescue.

An audio frame is a small group of samples that covers a fixed, short amount of time.

Instead of sending sound one number at a time, we cut it into small equal pieces, and each piece is a frame.

The most common frame lengths in voice systems are 10 milliseconds and 20 milliseconds. One millisecond is one thousandth of a second. So, a 20 millisecond frame is one fiftieth of a second.

Think of it like slicing a long loaf of bread. The loaf is the full sound. Each slice is a frame. Every slice has the same thickness, so we always know how much bread is in one slice.

Full speech: "Hello, how are you?" (1 second)

| 20 ms   | 20 ms   | 20 ms   | 20 ms   | ... | 20 ms    |
| frame 1 | frame 2 | frame 3 | frame 4 | ... | frame 50 |

Here, we can see that 1 second of speech becomes 50 frames, because 1000 milliseconds divided by 20 milliseconds is 50.

How to calculate the size of a frame

The best way to learn this is by taking an example.

Let's say our audio is 16 kHz, 16-bit PCM, mono (one channel), and our frame length is 20 milliseconds.

Step 1: We find the number of samples in one frame.

Samples per frame = sample rate * frame length in seconds

So, 16,000 * 0.020 = 320 samples.

Step 2: We find the number of bytes in one frame.

Bytes per frame = samples per frame * bytes per sample * channels

So, 320 * 2 * 1 = 640 bytes.

We can write the code as below:

def frame_size(sample_rate, frame_ms, bytes_per_sample=2, channels=1):
    samples = int(sample_rate * frame_ms / 1000)
    size_in_bytes = samples * bytes_per_sample * channels
    return samples, size_in_bytes

print(frame_size(16000, 20))
print(frame_size(8000, 20))
print(frame_size(24000, 20))
print(frame_size(16000, 10))

Here, we have a small function that takes the sample rate and the frame length, and returns the number of samples and the number of bytes in one frame.

It will print the following:

(320, 640)
(160, 320)
(480, 960)
(160, 320)

Let me put these numbers in a table for your better understanding.

Here, we can notice one important thing. Look at the 20 ms rows. The frame length in time stays the same, but the number of samples changes with the sample rate. A frame is defined by time, not by the number of samples.

Many bugs in voice pipelines happen when one part thinks in milliseconds and another part thinks in samples or bytes. For example, if one part sends 320 bytes of 8 kHz audio, which is 20 ms of sound, and the next part reads it as 16 kHz audio, it treats those bytes as only 10 ms of sound. The voice then plays twice as fast and sounds high-pitched.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

What is a chunk size?

Now that we have learned about frames, it's time to learn about chunk size.

Chunk size is the amount of audio that one part of the pipeline hands over to the next part in a single step.

A chunk can be exactly one frame. Or it can be many frames joined together. So, a frame is the small unit that the audio is cut into, and a chunk is how much of that audio we move at one time.

To understand why this matters, let's first see what a real-time voice pipeline looks like.

+------------------+
|    Microphone    |
+------------------+
         ↓
+------------------+
|  Speech-to-Text  |
+------------------+
         ↓
+------------------+
|       LLM        |
+------------------+
         ↓
+------------------+
|  Text-to-Speech  |
+------------------+
         ↓
+------------------+
|     Speaker      |
+------------------+

Here, the speech-to-text part turns the user's voice into text, the LLM (Large Language Model) writes a reply, and the text-to-speech part turns that reply back into voice.

At every arrow, audio or text moves from one part to the next. The chunk size decides how big each piece is when it moves.

Let's say the microphone gives us 20 millisecond frames. We can:

• Send every 20 millisecond frame to the speech-to-text part as soon as it is ready. Here, the chunk size is 20 milliseconds.
• Collect 5 frames and send them together. Here, the chunk size is 100 milliseconds.
• Collect 50 frames and send them together. Here, the chunk size is 1 second.

All three choices send the same sound, but they feel very different to the user.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

How chunk size affects delay

When we use a chunk size of 1 second, the first word the user says has to wait until a full second of audio is collected. Only then is it sent. So, the very first sound is already 1 second old before the next part even sees it.

When we use a chunk size of 20 milliseconds, the first sound waits at most 20 milliseconds before it moves forward.

The bigger the chunk, the longer the first sample in it has to wait.

This waiting time is called buffering delay. And it happens at every step where we collect audio into chunks.

Let's see this with numbers. Suppose our pipeline collects audio into chunks at three places: when the microphone captures the voice, when we send it to speech-to-text, and when we play the reply from text-to-speech.

Here, we can see that the delay adds up very quickly. In a normal conversation, people usually reply to each other within a few hundred milliseconds. So, spending a few hundred milliseconds only on waiting for chunks to fill is a big loss, and that too before the agent has done any real work.

We have a detailed blog on how to build a latency budget for a voice AI agent that explains where every millisecond of a reply goes.

This is why real-time voice pipelines use small chunks.

The cost of very small chunks

Now, the question is: if small chunks are good, why not make them as small as possible, like 1 millisecond?

The answer is that every chunk has a fixed cost.

When we send audio over the internet, we send it in small pieces called packets. Every packet carries a header. A header is extra information that tells the network where the packet is going and in which order to put the packets back together. For voice calls, the standard headers (called IP, UDP, and RTP headers) together take about 40 bytes for every packet.

Let's say we use compressed voice at about 24,000 bits per second.

• With 20 millisecond packets, we send 50 packets every second. The voice data in each packet is about 60 bytes, and the header is 40 bytes.
• With 10 millisecond packets, we send 100 packets every second. The voice data in each packet is about 30 bytes, but the header is still 40 bytes.

Here, we can see that with 10 millisecond packets, the header becomes bigger than the actual voice inside the packet.

The same idea applies inside our own code. Every chunk means one more function call, one more message, and one more round of work for the model. So, very small chunks mean much more overhead for the same sound.

Also, many models need some amount of sound to make a good decision. A model cannot tell if a person is speaking or just breathing by looking at 1 millisecond of audio. It needs a bit more context.

So, we have a trade-off:

• Small chunks mean less delay, but more overhead.
• Big chunks mean less overhead, but more delay.

Many real-time voice systems use 10 to 20 milliseconds for audio frames on the network, because this range keeps the delay low and the overhead reasonable.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Different parts want different sizes

Now, here comes another challenge. Every part of the voice pipeline has its own preferred size.

Let's see a few real examples:

• A codec is the tool that compresses audio before sending it and decompresses it after receiving it. Opus, the codec used by WebRTC (the technology browsers use for live voice and video calls) and many voice apps, supports frame lengths from 2.5 milliseconds up to 60 milliseconds, and newer versions of its library go up to 120 milliseconds. 20 milliseconds is the common default for voice.
• A voice activity detector, or VAD, is a small model that tells whether a person is speaking or not. WebRTC VAD, a popular one, accepts only 10, 20, or 30 millisecond frames.
• Silero VAD, another popular voice activity detector, expects chunks of 512 samples at 16 kHz or 256 samples at 8 kHz. Both are 32 milliseconds.
• Phone networks commonly carry audio as 20 millisecond frames at 8 kHz, which is 160 samples per frame.

Here, we can notice the problem. The microphone gives us 20 millisecond frames. But Silero VAD wants 32 millisecond chunks. These numbers do not line up.

So, how can we connect two parts that want different sizes? The answer is a buffer.

A buffer is a waiting area where we collect incoming audio and take out pieces of exactly the size the next part wants. Taking audio in one size and giving it out in another size is called re-chunking.

We can write the code as below:

class Rechunker:
    def __init__(self, chunk_bytes):
        self.chunk_bytes = chunk_bytes
        self.buffer = b""

    def add(self, frame):
        self.buffer += frame
        chunks = []
        while len(self.buffer) >= self.chunk_bytes:
            chunks.append(self.buffer[:self.chunk_bytes])
            self.buffer = self.buffer[self.chunk_bytes:]
        return chunks

Here, we have a simple re-chunker:

• chunk_bytes is the size that the next part wants. For Silero VAD at 16 kHz, it is 512 samples * 2 bytes = 1024 bytes.
• add puts the new frame into the buffer and returns the list of chunks that are ready.
• The while loop takes out full chunks of the right size as long as there is enough audio.
• Any leftover audio stays in the buffer and waits for the next frame.

Now, let's walk through it step by step. Each incoming frame is 640 bytes (20 milliseconds at 16 kHz), and each outgoing chunk is 1024 bytes.

Step 1: Frame 1 arrives. The buffer has 640 bytes. Not enough, so nothing goes out.

Step 2: Frame 2 arrives. The buffer has 1280 bytes. One chunk of 1024 bytes goes out. 256 bytes stay.

Step 3: Frame 3 arrives. The buffer has 896 bytes. Not enough, so nothing goes out.

Step 4: Frame 4 arrives. The buffer has 1536 bytes. One chunk goes out. 512 bytes stay.

Here, we can see that no sound is lost. Every byte goes out, just in different-sized pieces. It works perfectly.

Note: Every buffer adds a little delay, because audio sits there until a full chunk is ready. So, we must keep the number of re-chunking steps in our pipeline as small as possible.

To master how to Design a Real-Time Voice AI Agent and the Numbers Every AI Engineer Should Know, check out our AI and Machine Learning Program at Outcome School.

How to pick the right chunk size

Now that we have learned about the trade-off between delay and overhead, it's time to see how to pick the right chunk size.

First, we must follow what each part accepts. If a codec or a model only works with certain sizes, we must give it exactly that. Otherwise, it will either reject the audio or give wrong results.

Then, we must keep the chunks small on the way in. The way in is the path from the user to the agent. The audio from the user must reach speech-to-text as fast as possible, because the agent cannot reply until it knows what the user said. 20 milliseconds is a safe starting point.

After that, we must think about the way out. The way out is the path from the agent back to the user. Text-to-speech also produces audio in chunks. If we wait for the full reply to be generated before playing it, the user hears silence for a long time. So, we must start playing the first small chunk of audio as soon as it is ready.

Finally, we must think about interruptions. If we send big chunks of the agent's voice to the user's device, and the user interrupts in the middle, the audio that is already waiting on the device keeps playing until we clear it. Smaller chunks on the way out let the agent stop talking faster when the user interrupts.

So, for a real-time voice agent, we must choose small chunks, and accept a bit of extra overhead in return for a conversation that feels natural.

Now we must have understood what audio frames and chunk sizes are, how to calculate them, and how to choose the right size in a real-time voice pipeline.

Frequently Asked Questions

How many bytes does one second of 16 kHz audio take?

One second of 16 kHz, 16-bit PCM, mono audio takes 32,000 bytes. At 16 kHz, we take 16,000 samples every second, and each 16-bit sample takes 2 bytes. So, a 20 millisecond frame of this audio has 320 samples and takes 640 bytes.

What happens if two parts of a voice pipeline use different sample rates?

The audio plays at the wrong speed. For example, if one part sends 320 bytes of 8 kHz audio, which is 20 milliseconds of sound, and the next part reads it as 16 kHz audio, it treats those bytes as only 10 milliseconds of sound. The voice then plays twice as fast and sounds high-pitched.

What chunk size does Silero VAD expect?

Silero VAD expects chunks of 512 samples at 16 kHz or 256 samples at 8 kHz. Both are 32 milliseconds of audio. Since a microphone often gives 20 millisecond frames, we use a buffer to re-chunk the audio into the size that Silero VAD wants.

Does re-chunking audio lose any sound?

No. Re-chunking does not lose any sound. Every byte goes out, just in different-sized pieces, and any leftover audio stays in the buffer until the next frame arrives. But every buffer adds a little delay, so we must keep the number of re-chunking steps in the pipeline as small as possible.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.