Why do sample rates like 8 kHz, 16 kHz, and 24 kHz matter for voice agents?

Sample rates like 8 kHz, 16 kHz, and 24 kHz matter for voice agents because the sample rate decides how much detail of a voice is kept, and every part of a voice agent expects audio at a specific sample rate. If we pick the wrong one, or mix them up, the agent hears worse, sounds worse, or breaks completely.

In this blog, we will learn about why sample rates like 8 kHz, 16 kHz, and 24 kHz matter for voice agents. We will also see how a computer turns sound into numbers, what a sample rate really is, how much of our voice each sample rate keeps, where 8 kHz, 16 kHz, and 24 kHz are used inside a voice agent, what goes wrong when the sample rates do not match, and how to pick the right one for each part of the agent.

We will cover the following:

• How Does a Computer Store Sound?
• What is a Sample Rate?
• How Much of Our Voice Does a Sample Rate Keep?
• 8 kHz: The Phone Call Sample Rate
• 16 kHz: The Speech Recognition Sample Rate
• 24 kHz: The Voice Output Sample Rate
• How Sample Rates Flow Through a Voice Agent
• What Happens When Sample Rates Do Not Match?
• Sample Rate and the Size of the Audio
• How to Pick the Right Sample Rate

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

How Does a Computer Store Sound?

Before jumping into sample rates, we must know how a computer stores sound.

When we speak, our voice pushes the air back and forth very quickly. These pushes travel as waves and reach the microphone. The microphone turns them into a smooth electrical signal that goes up and down, just like the wave.

But a computer cannot store a smooth wave. A computer can only store numbers.

So, the computer looks at the wave again and again, at fixed moments, and writes down how high or low the wave is at each moment. Each of these written-down numbers is called a sample.

Let's say the computer takes samples of a wave like below:

   wave height
       ↑
  high |      *  *
       |    *      *
   mid |  *          *          *
       |               *      *
   low |                 *  *
       +--------------------------→ time

Here, each * is one sample. The computer keeps only these points, not the smooth wave. Later, when we play the audio, the speaker connects these points back into a wave.

This is how sound becomes a list of numbers inside a computer.

What is a Sample Rate?

The sample rate is the number of samples the computer takes every second.

It is measured in Hertz (Hz), which means "times per second". And kHz means "thousand times per second".

So:

• 8 kHz means 8,000 samples every second.
• 16 kHz means 16,000 samples every second.
• 24 kHz means 24,000 samples every second.

The best way to learn this is by taking an example.

Suppose we want to capture a ball flying through the air by taking photos. If we take only 2 photos every second, we see the ball here and then suddenly there, and we miss everything in between. If we take 60 photos every second, we see the full smooth path of the ball.

Sound works in the same way. More samples per second means the computer catches more of the fast movements of the wave. Fewer samples per second means the fast movements are lost.

Now, the question is: which parts of our voice are the "fast movements", and why do they matter?

How Much of Our Voice Does a Sample Rate Keep?

Our voice is not one single wave. It is a mix of many waves at the same time. Some waves move slowly, and some move very fast.

How fast a wave moves up and down is called its frequency, and it is also measured in Hz. For example, a wave at 1,000 Hz goes up and down 1,000 times every second.

• Low frequencies are the slow waves. They carry the deep, main body of our voice, and most of the energy of vowel sounds like "a", "o", and "u".
• High frequencies are the fast waves. They carry the sharp, hissing sounds like "s", "f", "sh", and "th". They also carry the crispness that makes a voice sound clear and natural.

Now, let's learn the rule that decides how much of our voice is kept.

A sample rate can only capture frequencies up to half of its own value.

This rule comes from the Nyquist-Shannon sampling theorem, and the half value is called the Nyquist frequency. To capture one full up-and-down movement of a wave, the computer needs at least two samples: one for the "up" and one for the "down". So, if we take 8,000 samples per second, we can capture at most 4,000 up-and-down movements per second.

Let's apply this rule to our three sample rates:

Anything in our voice above that limit is not stored, and it can never be brought back later.

For comparison, the human ear can hear up to around 20 kHz. That is why music on a CD uses 44.1 kHz.

Now, let's see what each of these three sample rates means for a voice agent.

8 kHz: The Phone Call Sample Rate

8 kHz is the sample rate of the traditional phone network.

When the phone network was designed, the goal was to make speech understandable while using as little capacity as possible. So, phone audio keeps roughly the range from 300 Hz to 3,400 Hz. An 8 kHz sample rate is enough to hold this range, because it can keep frequencies up to 4 kHz.

This is why a voice on a normal phone call sounds "thin". The deep part and the sharp part of the voice are cut off.

This matters a lot for voice agents, because many voice agents answer real phone calls. When a call comes from the phone network, the audio usually arrives at 8 kHz. For example, Twilio Media Streams send phone audio as 8 kHz audio in a format called mu-law, which is a way of squeezing each sample into 8 bits.

Remember that a large part of the "s" and "f" sounds lives in the high frequencies, above 4 kHz. At 8 kHz, that part is cut off. So, on a phone line:

• "sixty" and "fifty" are easy to confuse.
• "S" and "F" are hard to tell apart when the caller spells a name or an email.
• Numbers, names, and addresses get misheard more often.

That is why, on a phone call, people often say "S as in Sam" or "F as in Frank". The same problem affects a voice agent. Means, the speech recognition model receives less information, so it does more mistakes.

Note: We cannot fix this on our side. If the audio arrives at 8 kHz, the missing high frequencies are already lost before the audio reaches us.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

16 kHz: The Speech Recognition Sample Rate

16 kHz is the most common input sample rate for speech recognition, which is the part of the voice agent that turns speech into text. This part is also called STT (Speech-to-Text) or ASR (Automatic Speech Recognition).

At 16 kHz, the audio keeps frequencies up to 8 kHz. This range holds almost everything that matters for understanding speech. The "s", "f", and "sh" sounds now come through clearly. 16 kHz audio is often called wideband audio, while 8 kHz phone audio is called narrowband audio.

So, why not go even higher for speech recognition?

Because, going above 16 kHz adds very less useful information for understanding words. The extra high frequencies help a voice sound rich, but they do not help much in figuring out which word was spoken. At the same time, higher sample rates mean more data to send and more numbers for the model to process.

This is why many speech recognition models are trained on 16 kHz audio. For example, OpenAI's Whisper model expects its input audio at 16 kHz. Audio at any other sample rate must first be converted to 16 kHz.

One more thing to notice: a model trained mostly on clear 16 kHz audio can struggle with 8 kHz phone audio, because phone audio does not look like what the model learned from. That is why some speech recognition services offer special models tuned for phone calls.

24 kHz: The Voice Output Sample Rate

Now, it's time to learn about the sample rate used for speaking.

When the voice agent speaks back, it uses TTS (Text-to-Speech), which turns the reply text into audio. And here, the goal is different.

For speech recognition, the goal is to understand the words. For text-to-speech, the goal is to sound like a real, pleasant human. And a natural-sounding voice needs more of the high frequencies, the breathiness, the crisp "s", and the small details that make a voice sound human.

24 kHz keeps frequencies up to 12 kHz. This is enough for a voice to sound clear and natural, while keeping the audio size reasonable. That is why 24 kHz is a very common output sample rate for modern text-to-speech models. For example, the raw audio format of the OpenAI Realtime API is 16-bit audio at 24 kHz.

Some text-to-speech models go even higher, to 44.1 kHz or 48 kHz. But for a voice agent, 24 kHz is a common balance between quality and speed.

Note: If the voice agent is talking on a phone call, the 24 kHz voice is converted down to 8 kHz before it goes to the phone network. The caller hears phone-quality audio, no matter how good the original voice was.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

How Sample Rates Flow Through a Voice Agent

Now that we have learned about each sample rate, let's put them together and see how audio moves through a voice agent that answers phone calls.

+----------------------------+
| Caller speaks on phone |
+----------------------------+
              ↓
     8 kHz mu-law audio
              ↓
+----------------------------+
| Convert 8 kHz to 16 kHz |
+----------------------------+
              ↓
+----------------------------+
| Speech-to-Text (16 kHz) |
+----------------------------+
              ↓
            text
              ↓
+----------------------------+
| LLM writes the reply |
+----------------------------+
              ↓
            text
              ↓
+----------------------------+
| Text-to-Speech (24 kHz) |
+----------------------------+
              ↓
+----------------------------+
| Convert 24 kHz to 8 kHz |
+----------------------------+
              ↓
     8 kHz mu-law audio
              ↓
+----------------------------+
| Caller hears the reply |
+----------------------------+

Here, we can see that the same conversation passes through three different sample rates:

• The audio enters at 8 kHz from the phone network.
• It is converted to 16 kHz because the speech recognition model expects 16 kHz.
• The reply is generated at 24 kHz by the text-to-speech model.
• It is converted back to 8 kHz because the phone connection carries only 8 kHz.

This conversion from one sample rate to another is called resampling. Almost every voice agent does it somewhere.

Very important: Converting 8 kHz audio to 16 kHz does not bring back the lost high frequencies. It only adds more samples by filling in the gaps between the existing ones. The audio now has the right format for the model, but it still has the quality of a phone call.

If the user talks from a web browser or a mobile app instead of a phone line, the microphone usually records at 48 kHz or 44.1 kHz. This audio is converted down to 16 kHz for speech recognition, and it keeps much more detail than phone audio.

If we want to go deep into Agent Architecture and how to Design a Real-Time Voice AI Agent, our AI and Machine Learning Program at Outcome School covers them from the ground up.

What Happens When Sample Rates Do Not Match?

Most of the time, we do mistakes with sample rates without noticing them, because raw audio is just a list of numbers. The list itself does not say how many of those numbers belong to one second.

Let's say the text-to-speech model produces 24,000 samples for every second of speech. But our code tells the speaker that this audio is 16 kHz. The speaker plays only 16,000 samples per second. So, one second of speech now takes one and a half seconds to play. The voice becomes slow and deep.

Now, let's flip it. Suppose we have 16 kHz audio, and we tell the speaker that it is 24 kHz. The speaker plays 24,000 samples per second. The speech plays one and a half times faster, and the voice becomes high-pitched and squeaky.

The same thing happens on the input side. If we send 8 kHz phone audio to a speech recognition model and tell it the audio is 16 kHz, the model hears the caller speaking at double speed with a squeaky voice. The text that comes out is wrong or completely empty.

Also, when the conversion itself is done badly, the audio sounds crackly or robotic.

This is why, at every step, we must make sure that the sample rate written in our code matches the sample rate the audio really has.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Sample Rate and the Size of the Audio

The sample rate also decides how much data we send every second.

Each sample is stored as a number. In the most common raw format, called 16-bit PCM, each sample takes 2 bytes. Voice agents usually use single channel (mono) audio. So, we can calculate the data size like below:

data per second = sample rate * bytes per sample

Here, we can see that 24 kHz audio is three times the size of 8 kHz audio. Phone networks squeeze 8 kHz audio even more with mu-law (or a similar format called A-law), which uses 1 byte per sample, so it becomes 8,000 bytes per second.

For a single call, these numbers look small. But a voice agent streams audio in both directions for the whole call, often for many calls at the same time. More data means more network usage and more samples for the models to process, and in a voice agent every millisecond of delay matters.

This is the trade-off:

• A higher sample rate keeps more detail of the voice.
• A lower sample rate means less data and less processing.

We have a detailed blog on audio frames and chunk sizes in a real-time voice pipeline that explains how this audio is cut into small pieces and passed from one step to the next.

How to Pick the Right Sample Rate

Now, the next big question is: which sample rate do we choose?

The answer is: we do not choose one sample rate for the whole agent. We choose the right one for each part.

Let me tabulate the differences between 8 kHz, 16 kHz, and 24 kHz for your better understanding.

A few rules follow from this:

• Use the sample rate that each model expects. If the speech recognition model expects 16 kHz, send it 16 kHz. If the text-to-speech model gives 24 kHz, treat it as 24 kHz.
• Do not convert audio down and then back up without a reason. Every time we go down, detail is lost permanently.
• For phone calls, accept that the audio is 8 kHz. Use a speech recognition model that handles phone audio well, and make the agent confirm important details like numbers, names, and email addresses.
• For web and mobile apps, capture at a higher sample rate and convert it to 16 kHz for speech recognition.
• Always pass the correct sample rate along with the audio, so that no part of the system has to guess.

Now we must have understood why sample rates like 8 kHz, 16 kHz, and 24 kHz matter for voice agents, how each one keeps a different amount of our voice, where each one is used, and what goes wrong when they do not match.

Frequently Asked Questions

What is the Nyquist frequency?

The Nyquist frequency is half of the sample rate, and it is the highest frequency that a sample rate can capture. It comes from the Nyquist-Shannon sampling theorem. To capture one full up-and-down movement of a wave, we need at least two samples. So, 8 kHz audio keeps up to 4 kHz, and 16 kHz audio keeps up to 8 kHz.

What is the difference between narrowband and wideband audio?

Narrowband audio is 8 kHz phone audio, and wideband audio is 16 kHz audio. Narrowband audio keeps frequencies only up to 4 kHz, so the voice sounds thin. Wideband audio keeps frequencies up to 8 kHz, which holds almost everything that matters for understanding speech, so sounds like "s", "f", and "sh" come through clearly.

Why are "s" and "f" hard to tell apart on a phone call?

Because a large part of the "s" and "f" sounds lives in the high frequencies, above 4 kHz. Phone audio uses an 8 kHz sample rate, which keeps frequencies only up to 4 kHz, so that part is cut off. This is why "sixty" and "fifty" are easy to confuse, and why numbers, names, and addresses get misheard more often on a phone line.

Does converting 8 kHz audio to 16 kHz improve its quality?

No. Converting 8 kHz audio to 16 kHz does not bring back the lost high frequencies. It only adds more samples by filling in the gaps between the existing ones. The audio now has the right format for a speech recognition model that expects 16 kHz, but it still has the quality of a phone call.

What sample rate does Whisper expect?

OpenAI's Whisper model expects its input audio at 16 kHz. Audio at any other sample rate must first be converted to 16 kHz before it goes to the model. Many speech recognition models are trained on 16 kHz audio, because going above 16 kHz adds very little useful information for understanding words, but it means more data to send and process.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.