How does the Opus codec compress voice for real-time calls?

The Opus codec compresses voice by cutting the sound into tiny slices, usually 20 milliseconds long, keeping only the parts of each slice that our ears can actually hear, and describing those parts with a small set of numbers instead of the raw sound wave. This makes the voice around 30 times smaller, so it can travel over the internet fast enough for a live call.

In this blog, we will learn about how the Opus codec compresses voice for real-time calls. We will also see why raw voice is too big to send, what a codec does, how Opus cuts the voice into small frames, how its two engines SILK and CELT shrink the sound, how it handles silence and lost packets, and why it is used in WebRTC and voice AI agents.

We will cover the following:

• Why raw voice is too big for a call
• What is a codec?
• What is the Opus codec?
• Step 1: Cutting the voice into small frames
• Step 2: Keeping only what our ears can hear
• Step 3: SILK, the speech engine
• Step 4: CELT, the music engine
• Step 5: Hybrid mode, using both engines together
• How Opus saves data during silence
• How Opus handles lost packets
• Why Opus is used for real-time calls and voice agents

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

Why raw voice is too big for a call

When we speak into a microphone, our voice is a continuous wave of air pressure. A computer cannot store a continuous wave. So, it measures the height of the wave many times every second. Each measurement is called a sample.

For high-quality audio, the computer takes 48,000 samples every second, and it stores each sample as a 16-bit number.

Let's do the math for one second of voice:

48,000 samples * 16 bits = 768,000 bits per second = 768 kbps

Means, a single person talking needs around 768 kilobits every second (kbps). That is just for one direction of the call.

Now, the question is: what is the problem with this?

The problem is that many people are on slow mobile networks. On top of that, the network is shared with videos, downloads, and other apps. Sending 768 kbps for every speaker, every second, is too heavy. The voice will arrive late, break up, or not arrive at all.

So, we need a way to make the voice much smaller before sending it, and that too without making it sound bad. Here comes the codec into the picture.

What is a codec?

A codec is a tool that compresses audio on the sender's side and decompresses it back on the receiver's side.

The word itself tells us what it does:

Codec = Coder + Decoder

• The coder (also called the encoder) takes the raw voice and turns it into a small packet of data.
• The decoder takes that small packet and rebuilds the voice so that we can hear it.

The flow looks like below:

+---------------------+
|  Speaker's voice    |
|  (768 kbps raw)     |
+---------------------+
           ↓
+---------------------+
|  Encoder            |
+---------------------+
           ↓
+---------------------+
|  Small packets      |
|  (around 24 kbps)   |
+---------------------+
           ↓
       Internet
           ↓
+---------------------+
|  Decoder            |
+---------------------+
           ↓
+---------------------+
|  Listener hears     |
|  the voice          |
+---------------------+

Here, we can see that the raw voice goes in at 768 kbps, and with a voice codec like Opus, only around 24 kbps travels over the internet. That is around 32 times smaller.

Note: Most voice codecs, including Opus, are lossy. Means, the rebuilt voice is not exactly the same as the original. The codec throws away the details that our ears will not notice. This is how it gets so small.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

What is the Opus codec?

Opus is an open, royalty-free audio codec designed for real-time communication over the internet. It was standardized in 2012 by the IETF (Internet Engineering Task Force), the group that creates internet standards, as RFC 6716.

Opus was built by combining two older codecs:

• SILK, which came from Skype and is very good at human speech.
• CELT, which came from the Xiph.Org Foundation and is very good at music and high-quality sound with very low delay.

The bitrate is the number of bits a codec sends every second. Opus can work at any bitrate from 6 kbps up to 510 kbps. At the low end, it can carry a clear phone-quality voice. At the high end, it can carry studio-quality music.

It also supports different audio bandwidths, which means how much of the sound range it keeps:

Here, a higher range sounds more natural, but it needs more data. An old phone call is close to narrowband. A good Opus call is usually wideband or fullband. For speech, RFC 7587, the standard for sending Opus in live calls, lists around 16 to 20 kbps as the best range for wideband voice, and around 28 to 40 kbps for fullband voice.

Now, let's understand how Opus actually compresses the voice, step by step.

Step 1: Cutting the voice into small frames

Opus does not wait for the full sentence. It cuts the voice into small slices of time called frames.

A frame can be 2.5, 5, 10, 20, 40, or 60 milliseconds long. For calls, the most common choice is 20 milliseconds.

Let's say a person says "Hello, how are you?" and it takes 1 second. Opus cuts this into 50 frames of 20 ms each, and it compresses and sends each frame as soon as it is ready.

Let's do the math for one 20 ms frame at 48 kHz:

• Samples in one frame: 48,000 * 0.020 = 960 samples
• Raw size: 960 * 16 bits = 15,360 bits
• Opus size at 24 kbps: 24,000 * 0.020 = 480 bits = 60 bytes

Here, we can notice that 15,360 bits become just 480 bits. Each 20 ms frame becomes one small packet of 60 bytes.

Why small frames? Because the encoder cannot compress a frame until the full frame is recorded. So, the frame size sets the minimum delay. The total delay added by Opus with 20 ms frames is around 26.5 ms, which is the 20 ms frame plus a 6.5 ms look-ahead. This is small enough for a natural conversation.

The trade-off: Smaller frames mean lower delay, but every packet also carries around 40 bytes of network headers, so very small frames waste more data on headers than on voice. Larger frames save data, but add delay. 20 ms is the balance point for most calls.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Step 2: Keeping only what our ears can hear

Opus uses the fact that our ears are not perfect.

Our ears have a few limits:

• We hear some frequencies much better than others. We are most sensitive to the middle range, where most of human speech is.
• A loud sound hides a quieter sound that is close to it in pitch. This is called masking.
• Very fine details in very high frequencies are hard for us to notice.

Opus uses these limits to decide where to spend its bits. It spends more bits on the parts we clearly hear, and fewer bits, or zero bits, on the parts that are hidden or that we barely notice.

Let's say there is a loud vowel sound and a very soft hiss at a nearby pitch. Our ears will not hear the hiss anyway. So, Opus does not waste bits on it.

This idea of compressing based on how humans hear is called psychoacoustics. Both engines inside Opus use it to decide where the small errors of compression are least noticed.

Now, let's see the two engines.

Step 3: SILK, the speech engine

SILK is the engine that Opus uses for speech at low bitrates. It works on the idea that human speech is very predictable.

Let's understand how our voice is produced. Air comes from our lungs. Our vocal cords vibrate to create a buzz. Then, our throat, mouth, tongue, and lips shape that buzz into different sounds like "aa", "ee", or "ss".

SILK copies this idea. Instead of sending the sound wave itself, it sends a description of how the voice was made. This method is called Linear Prediction.

It works in three parts:

• Predicting the next sample: Speech changes smoothly. So, each sample can be guessed well from the few samples just before it. SILK finds a small set of numbers, called filter coefficients, that describe the shape of the mouth and throat for that frame. With these numbers, the decoder can guess most of the wave by itself.
• Pitch prediction: When we speak a vowel, the buzz from our vocal cords repeats again and again at a steady rate. That rate is the pitch. SILK sends the pitch, so the decoder can repeat the pattern instead of receiving it again and again.
• The leftover error: The guess is never perfect. The small difference between the guess and the real wave is called the residual. Along with the filter numbers and the pitch, SILK sends this residual, and that too with very few bits.

Let's take a simple example for the sake of understanding. Suppose the samples are 10, 12, 14, 16. If the rule is "add 2 to the last sample", then the next guess is 18. If the real sample is 19, SILK needs to send the rule once and the small error of 1, not the full value 19. The decoder knows the same rule, so it makes the same guess of 18 and adds the error of 1 to get back 19. Small numbers need far fewer bits than big numbers.

This is how SILK can carry clear speech at low bitrates like 8 to 16 kbps.

SILK works only up to wideband, which means sounds up to 8 kHz. The issue with this engine is that it is not good for music or full-range audio. Let's see how the next engine solve this issue.

Step 4: CELT, the music engine

CELT is the engine that Opus uses for music, full-range sound, and very low delay. It works in a different way from SILK.

Instead of describing how the sound was made, CELT looks at which frequencies are present in each frame. It uses a math tool called the MDCT (Modified Discrete Cosine Transform) to convert the frame from a wave over time into a list of frequencies and their strengths.

Then, it groups these frequencies into bands. The bands are narrow at low frequencies and wide at high frequencies, because our ears separate low pitches finely and high pitches roughly.

For each band, CELT sends two things, which means it splits every band into a loudness part and a detail part:

• The energy: how loud that band is. This is the most important part, so CELT always keeps it accurate.
• The shape: the fine detail inside the band. CELT spends more bits on the shape of important bands and fewer bits on bands our ears care less about.

Since CELT always keeps the energy of every band correct, the overall sound stays balanced even when the fine detail is rough. This is why CELT sounds natural even at lower bitrates. In fact, CELT stands for Constrained Energy Lapped Transform, and the "constrained energy" part refers to this idea of keeping the energy of every band fixed.

CELT also works with very short frames, down to 2.5 ms, which makes it well suited for very low delay.

Now that we have learned about SILK and CELT, it's time to learn about how Opus uses both of them together.

Step 5: Hybrid mode, using both engines together

Now, the next big question is: which engine does Opus pick?

The answer is that Opus can use SILK, CELT, or both at the same time. It has three modes:

In the hybrid mode:

• SILK handles the low frequencies, up to 8 kHz, where most of the meaning of speech is.
• CELT handles the high frequencies, above 8 kHz, which make the voice sound clear and natural.

Then, the decoder adds both parts together to rebuild the full voice. Hybrid mode works with 10 ms and 20 ms frames.

Let's see the flow of hybrid mode as below:

+---------------------------+
|  One 20 ms voice frame    |
+---------------------------+
              ↓
     Split by frequency
              ↓
+-------------+-------------+
|  0 to 8 kHz | 8 to 20 kHz |
|  SILK       | CELT        |
+-------------+-------------+
              ↓
+---------------------------+
|  One Opus packet          |
+---------------------------+

The encoder switches between these modes automatically, frame by frame, based on the bitrate and the type of sound. If the network gets slow, it can move to a lower bitrate and a narrower bandwidth in the middle of the call, without dropping the call.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Now, let's see how Opus saves data when nobody is speaking.

How Opus saves data during silence

In a normal conversation, each person is silent for a large part of the time, because they are listening to the other person.

Sending full packets for silence is a waste. So, Opus has a feature called DTX (Discontinuous Transmission).

When DTX is on and the speaker is silent, Opus stops sending normal packets. It only sends one tiny packet around every 400 ms. The receiver fills the gap with soft background noise, called comfort noise, so that the listener does not think the call has dropped.

This saves a lot of data in a two-way call.

Now, let's move to the next problem, which is lost packets.

How Opus handles lost packets

On the internet, some packets get lost or arrive too late. For a live call, we cannot wait for a lost packet to be sent again, because that will add delay. So, Opus has its own ways to deal with loss.

• Packet Loss Concealment (PLC): When a packet is missing, the decoder makes a guess of the missing 20 ms from the frames that came just before it. For short losses, the listener usually does not notice.
• Forward Error Correction (FEC): In SILK and hybrid modes, Opus can put a lower-quality copy of the previous frame inside the current packet. If the previous packet is lost, the decoder uses this copy instead.
• Deep learning tools: Newer versions of Opus, starting with Opus 1.5 in 2024, added small neural networks to improve the guessing during packet loss, and a feature called DRED (Deep Redundancy) that can recover up to one second of lost voice.

If we want to go deep into AI and ML System Design and how to Design a Real-Time Voice AI Agent, our AI and Machine Learning Program at Outcome School covers them in depth.

Why Opus is used for real-time calls and voice agents

Opus is one of the mandatory audio codecs in WebRTC, the technology that browsers use for live audio and video calls. So, most browser-based calls use Opus.

For voice AI agents, Opus matters a lot:

• Low delay: A voice agent must reply fast, so every extra delay hurts. Opus adds only around 26.5 ms with 20 ms frames, and even less with smaller frames.
• Small size: The user's voice reaches the server quickly, even on a mobile network.
• Good quality: Clear audio helps the speech-to-text model understand the words correctly.
• Handles loss: PLC and FEC keep the audio usable on weak networks.

Note: Speech-to-text models usually need raw audio samples, often at 16 kHz. So, the server decodes the Opus packets back into raw audio before passing them to the model. In the same way, the voice made by the text-to-speech model is encoded into Opus before it is sent to the user.

Now we must have understood how the Opus codec compresses voice, how SILK and CELT work together, and why it is used for real-time calls and voice agents.

Frequently Asked Questions

Is Opus a lossy codec?

Yes. Opus is a lossy codec, which means the rebuilt voice is not exactly the same as the original. It throws away the details that our ears will not notice, such as a quiet sound hidden by a louder one at a nearby pitch. This is how it makes the voice small enough for a live call.

How much delay does Opus add to a call?

With the common 20 ms frames, Opus adds around 26.5 ms of delay. This is the 20 ms frame plus a 6.5 ms look-ahead, because the encoder cannot compress a frame until the full frame is recorded. Smaller frames, down to 2.5 ms, add even less delay, but they waste more data on network headers.

What bitrate should we use for Opus voice?

For speech, around 16 to 20 kbps is the best range for wideband voice, and around 28 to 40 kbps for fullband voice, as listed in RFC 7587. Opus can work at any bitrate from 6 kbps up to 510 kbps. At the low end it carries phone-quality voice, and at the high end studio-quality music.

Is Opus free to use?

Yes. Opus is an open, royalty-free audio codec. It was standardized in 2012 by the IETF (Internet Engineering Task Force) as RFC 6716. It is also one of the mandatory audio codecs in WebRTC, so most browser-based calls already use it.

Do speech-to-text models take Opus audio directly?

No. Speech-to-text models usually need raw audio samples, often at 16 kHz. So, the server decodes the Opus packets back into raw audio before passing them to the model. In the same way, the voice made by the text-to-speech model is encoded into Opus before it is sent to the user.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.