How does a jitter buffer keep voice audio smooth over a bad network?

A jitter buffer is a small waiting area on the receiving side that holds incoming voice packets for a short time, puts them back in the right order, and plays them out at a steady pace. This keeps the audio smooth even when the network delivers the packets unevenly.

In this blog, we will learn about how a jitter buffer keeps voice audio smooth over a bad network. We will also see how voice travels over the internet in small packets, what jitter is and why it breaks the audio, how the jitter buffer fixes it step by step, how a fixed jitter buffer differs from an adaptive one, what happens when a packet arrives too late or never arrives, and where the jitter buffer sits in a voice AI agent.

We will cover the following:

• How voice travels over the internet
• What is jitter?
• Why uneven arrival breaks the audio
• What is a jitter buffer?
• How a jitter buffer works step by step
• The trade-off: smoothness vs delay
• Fixed jitter buffer vs adaptive jitter buffer
• What happens when a packet is late or lost
• Where the jitter buffer sits in a voice AI agent

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

How voice travels over the internet

Before jumping into the jitter buffer, we must know how our voice moves from one device to another over the internet.

When we speak into a phone or a laptop, the microphone captures our voice as a continuous sound. The device does not send this sound as one long piece. Instead, it cuts the sound into small pieces. A common size is 20 milliseconds of audio per piece.

Each piece is compressed and put inside a small envelope called a packet. Each packet also carries two important labels:

• A sequence number, which tells the order of the packet: 1, 2, 3, 4, and so on.
• A timestamp, which tells which moment of the speech this piece of audio belongs to.

So, one second of speech becomes 50 packets, because 1000 milliseconds divided by 20 milliseconds is 50.

The sender sends these packets one after another, every 20 milliseconds, at a steady pace. The packets then travel across the internet through many routers and networks until they reach the other side.

On the receiving side, the device opens each packet, decompresses the audio, and plays it through the speaker.

If every packet took exactly the same time to travel, the receiver could simply play each packet the moment it arrives. But, the internet does not work like that.

What is jitter?

Jitter is the variation in the time that packets take to travel from the sender to the receiver.

In simple words, packets that leave at a steady pace do not arrive at a steady pace.

Let's say the sender sends one packet every 20 milliseconds. The travel time of each packet is different, because each packet shares the network with traffic from many other people. Sometimes a router is busy and the packet waits in a queue. Sometimes the packet even takes a different path.

Assume that the travel times look like below:

Here, we can notice a few things:

• Packet 1 and packet 2 arrive 20 milliseconds apart, as expected.
• Packet 3 got stuck somewhere and took 90 milliseconds instead of 50.
• Packet 4 arrives before packet 3, which means the packets arrive out of order.
• Packet 3 and packet 5 arrive almost together.

The average travel time can be fine, but the travel time keeps changing from packet to packet. This change is jitter. A bad network for voice calls is usually one with a lot of jitter, a lot of packet loss, or both.

Why uneven arrival breaks the audio

Now, let's see what happens if the receiver plays every packet the moment it arrives.

The speaker needs a new 20 millisecond piece of audio exactly every 20 milliseconds. If it pauses to wait, we hear the pause.

Using the table above:

• At 50 ms, packet 1 arrives and plays from 50 ms to 70 ms.
• At 70 ms, packet 2 arrives and plays from 70 ms to 90 ms.
• At 90 ms, the speaker needs packet 3, but packet 3 is not here. The speaker has nothing to play. We hear a gap or a click.
• At 100 ms, packet 4 arrives, but it belongs after packet 3. If we play it now, the words come out in the wrong order.
• At 130 ms and 135 ms, packet 3 and packet 5 arrive almost together, so we have more audio than we can play at that moment.

The result is choppy, robotic audio with small gaps, skipped sounds, and sounds in the wrong order.

The problem is not that the packets are slow. The problem is that they are uneven. So, how can we turn uneven arrival back into a steady stream? The answer is the jitter buffer.

What is a jitter buffer?

A jitter buffer is a small waiting area on the receiving side where incoming voice packets are held for a short time, put back in the correct order, and then released to the speaker at a steady pace.

The word buffer here means a temporary holding space. The jitter buffer is a buffer that exists to absorb jitter.

Let's say we are waiting at the gate of a movie hall for our friends. They come at random times. Some come early, some come late, and some come in a group. Instead of entering the hall one by one the moment each friend arrives, we decide to wait for 10 minutes at the gate, collect everyone, and then enter in a neat line. We spend 10 extra minutes waiting, but the entry becomes smooth.

The jitter buffer does exactly this with packets. It waits a little on purpose, so that late packets get time to catch up, and then it plays everything in a neat, steady line.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

How a jitter buffer works step by step

The best way to learn this is by taking an example. Let's use the same five packets and a jitter buffer that holds audio for 60 milliseconds before playing.

Step 1: Packet 1 arrives at 50 ms. The jitter buffer does not play it immediately. It stores it and schedules it to play at 50 + 60 = 110 ms.

Step 2: From now on, the playback runs on a steady clock. The jitter buffer uses the timestamp of each packet to give it a playback slot. Packet 1 plays at 110 ms, packet 2 at 130 ms, packet 3 at 150 ms, packet 4 at 170 ms, and packet 5 at 190 ms.

Step 3: Packet 2 arrives at 70 ms and goes into the buffer. Its slot is at 130 ms, so it is early. It waits.

Step 4: Packet 4 arrives at 100 ms, before packet 3. The jitter buffer reads the sequence number, sees that it is packet 4, and places it in the fourth position. This is how the wrong order gets corrected.

Step 5: Packet 3 arrives at 130 ms. Its slot is at 150 ms, so it is still in time. The jitter buffer places it in the third position, between packet 2 and packet 4.

Step 6: Packet 5 arrives at 135 ms. Its slot is at 190 ms, so it is in time too.

Now, the speaker receives packet 1, 2, 3, 4, and 5 in the correct order, every 20 milliseconds, with no gaps.

We can see the full flow as below:

Sender: packets leave every 20 ms
  ↓
Internet: packets get delayed by different amounts
  ↓
Jitter buffer: holds packets, reorders them by sequence number
  ↓
Speaker: plays one packet every 20 ms, in order

Here, the jitter buffer takes uneven input and gives steady output. The only cost is that every packet now plays 60 milliseconds later than it could have.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

The trade-off: smoothness vs delay

Now, the question is: how long must the jitter buffer wait?

The waiting time is called the buffer delay or the buffer depth. Choosing it is a balance between smoothness and delay, and these two pull in opposite directions.

In our example, packet 3 took 40 milliseconds longer than packet 1, so a buffer of 60 milliseconds was enough. If the buffer was only 20 milliseconds, packet 1 would play at 70 ms, the slot of packet 3 would be at 110 ms, and packet 3 would arrive at 130 ms, which is too late.

If the buffer is too small:

• Packets that are delayed by more than the buffer can absorb arrive after their slot.
• The jitter buffer has to treat them as missing.
• We hear more gaps and glitches.

If the buffer is too large:

• Almost every packet arrives in time, so the audio is very smooth.
• But every word reaches the listener later.
• In a conversation, this extra delay often makes people talk over each other, because each person hears the other one late.

Let me tabulate the differences between a small buffer and a large buffer for your better understanding.

The ideal buffer is just large enough to absorb the jitter of the current network, and not larger. The problem is that the jitter of a network keeps changing. A Wi-Fi connection can be calm one minute and very jittery the next minute. This brings us to the two types of jitter buffers.

Fixed jitter buffer vs adaptive jitter buffer

Fixed jitter buffer: It uses one constant waiting time for the whole call, for example 60 milliseconds.

• Advantage: It is simple to build and its behavior is predictable.
• Disadvantage: If the network becomes worse, 60 milliseconds is not enough and we hear glitches. If the network becomes better, 60 milliseconds is more than needed and we add delay for no reason.

Adaptive jitter buffer: It keeps measuring how much the arrival times vary and changes its waiting time during the call.

• When packets start arriving more unevenly, it grows the buffer to avoid gaps.
• When the network becomes calm, it shrinks the buffer to reduce delay.

Now, the next big question is: how does an adaptive jitter buffer grow or shrink without causing a glitch itself?

If it simply paused the audio to grow the buffer, we would hear a silence. If it simply dropped audio to shrink the buffer, we would hear a jump. So, adaptive jitter buffers do this gently in two common ways:

• Adjust during silence: People pause between sentences. The jitter buffer can make a silent pause slightly longer to grow, or slightly shorter to shrink. A pause that is a few milliseconds longer is hard to notice.
• Time stretching: The jitter buffer plays the speech slightly slower, so that packets arrive a little faster than they are played and the stored audio grows, or slightly faster to reduce the stored audio, without changing the pitch of the voice. A good example is NetEq, the jitter buffer inside WebRTC, which compares the amount of stored audio with a target delay and speeds up or slows down the audio in small amounts to reach that target.

Let me tabulate the differences between the fixed jitter buffer and the adaptive jitter buffer for your better understanding.

This is why real-time voice systems like WebRTC use an adaptive jitter buffer.

If we want to go deep into AI and ML System Design and how to Design a Real-Time Voice AI Agent, our AI and Machine Learning Program at Outcome School covers them from the ground up.

What happens when a packet is late or lost

No matter how good the jitter buffer is, some packets do not make it in time. There are two cases.

Case 1: The packet arrives too late. Its playback slot has already passed. The speaker has already moved past that moment. So, the jitter buffer throws the late packet away. For the listener, a late packet is the same as a lost packet.

Case 2: The packet never arrives. Some packets get dropped by the network. This is called packet loss.

In both cases, the speaker still needs 20 milliseconds of audio for that slot. If we play silence, we hear a gap. So, here comes Packet Loss Concealment to the rescue.

Packet Loss Concealment means filling the missing slot with a guess of what the audio was likely to be, so that the listener does not hear a hard gap. Our voice does not change much within a few tens of milliseconds, so the receiver can create a short piece of sound that continues the previous sound.

• For one missing packet, the guess is often good enough that the listener does not notice.
• For many missing packets in a row, the guessed sound slowly fades out, because a long guess starts to sound unnatural.

The audio codec can also help here. A codec is the method that compresses the audio before sending and decompresses it after receiving. Opus is a popular codec for voice calls. When Opus expects packet loss, it can carry a low-quality copy of a piece of audio inside the packet that comes after it. Opus calls this in-band Forward Error Correction. If a packet is lost but the next one arrives, the receiver can rebuild a rough version of the lost audio from it. This works because the jitter buffer is often already holding the next packet before the slot of the lost packet comes up for playback.

So, the jitter buffer corrects the timing and the order, and packet loss concealment fills the holes that are left.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Where the jitter buffer sits in a voice AI agent

Now, let's take a real use case: a voice AI agent. We talk to it, and it talks back.

When the audio travels over a real-time connection like WebRTC, it flows in two directions, and there is a jitter buffer on each receiving side:

• User to agent: Our voice goes from our device to the agent's server. The server receives packets, puts them through a jitter buffer, and then sends the steady audio to the speech recognition model.
• Agent to user: The agent's voice goes from the server to our device. Our device receives packets, puts them through a jitter buffer, and then plays the audio through the speaker.

We can see it as below:

User speaks
  ↓
Network (jitter)
  ↓
Jitter buffer on server
  ↓
Speech recognition
  ↓
LLM
  ↓
Text to speech
  ↓
Network (jitter)
  ↓
Jitter buffer on user's device
  ↓
User hears the reply

The jitter buffer appears twice in one turn of the conversation. Every millisecond the jitter buffer waits is added to the total time between the user finishing a sentence and hearing the agent's reply.

This is why the jitter buffer is an important item in the latency budget of a voice agent, which is the total time we allow for each step of a reply. A large buffer gives clean audio but makes the agent feel slow. A small buffer makes the agent feel fast, but on a bad network the speech recognition hears broken audio and the user hears a choppy voice. An adaptive jitter buffer helps the voice agent stay smooth on a bad network and fast on a good network.

We have a detailed blog on why TTS audio buffering makes interruptions slow that explains how the audio waiting on the user's device affects interruptions.

Now we must have understood how a jitter buffer keeps voice audio smooth over a bad network, how it reorders packets and plays them at a steady pace, how a fixed jitter buffer differs from an adaptive one, how late and lost packets are handled, and where it sits in a voice AI agent.

Frequently Asked Questions

Does a jitter buffer add delay to a voice call?

Yes. A jitter buffer adds delay on purpose. It holds every packet for a short time before playing it, so that late packets get time to catch up. In a voice AI agent, every millisecond the jitter buffer waits is added to the time between the user finishing a sentence and hearing the reply.

What is a good jitter buffer size?

The ideal jitter buffer is just large enough to absorb the jitter of the current network, and not larger. A small buffer, like 20 to 40 milliseconds, suits a stable network. A large buffer, like 150 to 200 milliseconds, gives very smooth audio on a very jittery network, but every word reaches the listener later.

Can a jitter buffer recover a lost packet?

No. A jitter buffer corrects the timing and the order of packets, but it cannot bring back a packet that never arrives. Packet loss concealment fills the missing slot with a guess of the audio. A codec like Opus can also rebuild a rough version of the lost audio from the next packet using in-band Forward Error Correction.

What is NetEq in WebRTC?

NetEq is the adaptive jitter buffer inside WebRTC. It compares the amount of stored audio with a target delay. Then it speeds up or slows down the audio in small amounts to reach that target, without changing the pitch of the voice.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.