Why does phone audio make speech recognition worse?

Authors
Why does phone audio make speech recognition worse?

Phone audio makes speech recognition worse because a phone call throws away a big part of our voice before it reaches the model. The call cuts off the high sounds, compresses the audio heavily, loses small pieces on a bad network, and picks up background noise, so the model gets a thinner, damaged version of what we said.

In this blog, we will learn about why phone audio makes speech recognition worse. We will also see how our voice travels over a phone call, why the phone cuts off the high sounds of our voice, how compression and lost packets damage the audio further, how noise and echo add to the problem, why a model trained on clean audio struggles with phone audio, and what we can do to make speech recognition work better on phone calls.

We will cover the following:

  • What is Speech Recognition?
  • How Does Our Voice Travel Over a Phone Call?
  • Reason 1: The Phone Cuts Off the High Sounds
  • Reason 2: Compression Throws Away More Detail
  • Reason 3: Lost Packets Leave Holes in the Audio
  • Reason 4: Noise, Echo, and Small Microphones
  • Reason 5: The Model Learned From Different Audio
  • How All the Problems Stack Up
  • How to Make Speech Recognition Work Better on Phone Audio

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is Speech Recognition?

Speech Recognition is the task of turning spoken audio into written text. It is also called ASR, which stands for Automatic Speech Recognition.

When we talk to a voice agent, the first thing the agent does is run speech recognition on our voice. It converts "I want to book a table for two" from sound into text. Then, the rest of the agent reads that text and decides what to do.

So, if speech recognition gets a word wrong, the rest of the agent works with the wrong text. If the model hears "sixty" when we said "fifty", the agent acts on the wrong number.

The model only sees the numbers that represent the sound wave. If those numbers are missing some information, the model has less to work with. Now, the question is: what happens to those numbers on a phone call?

How Does Our Voice Travel Over a Phone Call?

Let's say we call a voice agent from our mobile phone. Our voice goes through many steps before the model sees it.

The journey of our voice looks like below:

+-----------------------------+
| We speak into the phone     |
+-----------------------------+
              ↓
+-----------------------------+
| Small microphone picks it   |
| up, with background noise   |
+-----------------------------+
              ↓
+-----------------------------+
| Phone keeps only the        |
| 300 Hz to 3400 Hz sounds    |
+-----------------------------+
              ↓
+-----------------------------+
| Codec squeezes the audio    |
| into very few bits          |
+-----------------------------+
              ↓
+-----------------------------+
| Network carries it, and     |
| some pieces get lost        |
+-----------------------------+
              ↓
+-----------------------------+
| Speech recognition model    |
| receives the damaged audio  |
+-----------------------------+

Here, we can see that the audio is changed at almost every step. Let's understand each of these steps one by one.

Reason 1: The Phone Cuts Off the High Sounds

Before jumping into this, we must know what a frequency is.

Our voice is made of many vibrations happening together. Some vibrations are slow, and we hear them as low sounds, like a deep hum. Some vibrations are fast, and we hear them as high sounds, like a hiss. The speed of a vibration is called its frequency, and it is measured in Hz (Hertz), which means vibrations per second. 1 kHz is 1000 Hz.

Human speech has sounds spread from below 100 Hz all the way up to around 8000 Hz and beyond.

Now, a traditional phone call keeps only the sounds between about 300 Hz and 3400 Hz. Everything below 300 Hz and everything above 3400 Hz is cut off. This is called narrowband audio.

The phone network was designed decades ago, when sending sound was expensive. People can still understand each other with only this middle band, so the network kept only this band and saved a lot of space.

This band is also tied to the sample rate. The sample rate is how many times per second the sound wave is measured and stored as a number. A phone call uses a sample rate of 8 kHz, which means 8000 measurements per second. A sample rate can only hold sounds up to half of itself. So, 8 kHz can hold sounds only up to 4 kHz, and the phone band stops a little before that, at 3400 Hz.

Speech recognition models usually expect 16 kHz audio, which holds sounds up to 8 kHz. So, a phone call gives the model less than half of the range of sounds it expects.

Let me tabulate the difference between phone audio and the audio a model usually expects for your better understanding.

PropertyPhone audio (narrowband)Typical model input (wideband)
Sample rate8 kHz16 kHz
Highest sound keptabout 3.4 kHzabout 8 kHz
Lowest sound keptabout 300 Hzclose to 0 Hz
Sounds like "s", "f"mostly lostkept

Now, the next big question is: which parts of our speech live above 3400 Hz?

The answer is: many of the consonants. Sounds like "s", "f", "th", "sh", and "z" are hissing sounds. A large part of their energy sits above 4000 Hz. The "s" sound especially has most of its energy in the high range.

When the phone cuts off the high range, these sounds become weak and sound similar to each other. Let's say we say "fat" and "sat" on the phone. The "f" and the "s" both lose their high part, and what remains in both is a faint noise. The two words become hard to tell apart.

This is why on a phone call, we often say "S as in Sugar" or "F as in Frank". People struggle with the same problem. The model struggles too, but it cannot ask "Did you say F or S?" unless we build that into the agent.

Some common confusions on phone audio:

  • "fifty" and "sixty"
  • "f" and "s" when spelling a name or an email
  • "fin" and "thin"
  • "mass" and "math"

These are exactly the words that matter most for a voice agent: numbers, names, email addresses, and account IDs. The model can use context to guess a word, but for a random name or a number, there is no context to help.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Reason 2: Compression Throws Away More Detail

Cutting the band is only the first step. After that, the audio is compressed by a codec.

A codec is a piece of software that shrinks audio so that it takes less space on the network, and then expands it back on the other side. Codec = Coder + Decoder.

Let's see how much space the audio takes. One second of 16 kHz audio, stored with 16 bits per sample, takes 256,000 bits. The classic phone codec, called G.711, uses 8 kHz and 8 bits per sample, which comes to 64,000 bits per second. Mobile calls squeeze it even more. The AMR codec used on many mobile calls runs at only about 4,750 to 12,200 bits per second.

Audio typeBits per second
16 kHz, 16-bit (model input)256,000
G.711 phone call64,000
AMR mobile callabout 4,750 to 12,200

Here, we can see that a mobile call carries about 20 to 50 times less data than the audio the model is used to.

To shrink the audio this much, the codec has to throw away details. Mobile codecs like AMR are designed so that a person can still understand the speech. They keep what the human ear cares about and drop the rest. But, small details that help the model tell two sounds apart can disappear.

The codec also adds its own small distortions, called artifacts. The voice can sound a little robotic, buzzy, or muffled. A person easily ignores this. A model that never heard this kind of sound during training gets confused by it.

Now, there is one more problem. A call often crosses more than one network. For example, our mobile call can go from a mobile network to a landline network and then to the voice agent's server. At each boundary, the audio can be decoded and encoded again with a different codec. This is called transcoding. Each round of transcoding adds a little more damage, just like making a photocopy of a photocopy.

Reason 3: Lost Packets Leave Holes in the Audio

Now, let's understand what happens on the network.

On a modern mobile or internet call, the audio is cut into small pieces, usually 20 milliseconds each, and each piece is sent as a packet. A packet is a small bundle of data that travels across the network by itself. We have a detailed blog on how SIP and RTP carry a phone call to a voice agent that explains how these packets travel step by step.

On a weak mobile signal, some packets arrive late, and some never arrive at all. A packet that arrives too late is treated as lost, because the call cannot wait for it.

When a packet is lost, there is a hole in the audio. The phone system does not leave silence there, because sudden silence sounds like the call has dropped. Instead, it uses packet loss concealment, which means it makes up a guess of what the missing sound was, usually by stretching or repeating the sound just before the hole.

Let's say we say "My account number is 4 7 2 9", and two packets in the middle of the "7" are lost. The system fills those 40 milliseconds with a stretched copy of the sound just before the gap. A person hears a tiny glitch. The model hears a broken "7", and it can write "4 1 2 9" or skip the digit completely.

So, the model is now not only getting less information. It is also getting made-up sound that was added to fill the gaps.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Reason 4: Noise, Echo, and Small Microphones

Phone calls happen everywhere: on the street, in a car, in a busy shop. A phone microphone is small and it picks up everything around us.

So, along with our voice, the model receives:

  • Background noise: traffic, fans, other people talking, music.
  • Echo: on a speakerphone, the agent's voice comes out of the speaker and goes back into the microphone, mixed with ours.
  • Distance: when the phone is far from the mouth, our voice becomes quieter and the room's sound becomes louder.

Phones also clean up the audio before sending it with noise suppression, echo cancellation, and automatic volume control. These help the person on the other side, but they also change the voice. Strong noise suppression can cut off the start or end of a soft word, or remove a quiet "s" because it looks like noise.

The phone band has already removed the high part of "s" and "f", so what is left is weak and noise-like. The noise suppression then removes it too.

Reason 5: The Model Learned From Different Audio

Now, let's understand the last reason. A speech recognition model learns by listening to a huge amount of audio along with the correct text. It becomes good at the kind of audio it hears during training.

Many large speech recognition models are trained mostly on wideband audio: podcasts, videos, audiobooks, and recordings made with good microphones. This audio is usually clean and full, with the high sounds present.

When such a model gets phone audio, the audio looks unfamiliar to it. The difference between the audio a model was trained on and the audio it gets in real use is called a domain mismatch.

Suppose a person learned English only by watching movies with clear sound. Now, we put the same person on a crackly phone line with traffic in the background. The person knows English very well, but the sound is so different from what they practiced on that they miss many words. The model has the same problem.

Here, we often do a mistake. Many models expect 16 kHz input, so we upsample the 8 kHz phone audio to 16 kHz before giving it to the model, and we expect it to become as good as real 16 kHz audio. Upsampling means adding extra in-between samples so that the audio has 16,000 samples per second.

But, upsampling does not bring back the lost sounds. The audio now has the right sample rate, but everything above 4 kHz is still empty, because that information was thrown away at the start of the call. The model gets 16 kHz audio with the top half silent, which is not what most of its training audio looks like.

No processing step can recover the sounds that the phone call has already thrown away.

To learn AI and ML System Design and how to Design a Real-Time Voice AI Agent, from the audio that reaches the model to the reply it speaks back, check out our AI and Machine Learning Program at Outcome School.

How All the Problems Stack Up

On a real call, all of these reasons happen together, one after another.

Let's follow the word "fifty" through a phone call as below:

+------------------------------------------+
| We say "fifty" clearly                   |
+------------------------------------------+
                     ↓
+------------------------------------------+
| Phone band cuts the high part of "f"     |
| The "f" becomes faint                    |
+------------------------------------------+
                     ↓
+------------------------------------------+
| Noise suppression treats the faint "f"   |
| like noise and weakens it further        |
+------------------------------------------+
                     ↓
+------------------------------------------+
| Codec drops more detail                  |
| The weak "f" gets fuzzier                |
+------------------------------------------+
                     ↓
+------------------------------------------+
| A packet is lost in the middle           |
| A guessed sound fills the gap            |
+------------------------------------------+
                     ↓
+------------------------------------------+
| Model trained on clean audio hears       |
| something between "fifty" and "sixty"    |
+------------------------------------------+

Here, we can notice that no single step destroyed the word. Each step made it a little worse, and together they made it hard to recognize. This is why the same model can work very well on a clean laptop recording and much worse on a phone call, even for the same person saying the same sentence.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

How to Make Speech Recognition Work Better on Phone Audio

Now, let's see what we can do about it. We cannot bring back the lost sounds. But, we can make the model better at handling phone audio, and we can design the agent around the weak spots.

Use a model that is trained on phone audio. Many speech recognition providers offer models tuned for phone calls. If we train our own model, we can take clean training audio and make it sound like phone audio: cut it to the 300 Hz to 3400 Hz band, pass it through a phone codec, and add noise. This is called data augmentation, and it teaches the model what phone audio sounds like.

Send the audio in the right format. If the model supports 8 kHz input directly, we must send 8 kHz and tell the model the correct sample rate. If the model needs 16 kHz, we must upsample properly with a good resampler. Telling the model the wrong sample rate makes the voice sound too fast or too slow, and recognition fails badly.

Use wideband audio when we can. Many mobile networks now support HD Voice, which uses a wideband codec called AMR-WB at 16 kHz and keeps sounds up to about 7000 Hz. But, HD Voice works only when both phones and every network on the way support it. Otherwise, the call falls back to narrowband. If the user talks to our agent through an app or a web page instead of a normal phone number, we can use a codec like Opus, which can keep even more of the voice. More of the voice reaches the model, and recognition improves.

Give the model hints. Many speech recognition systems let us pass a list of words we expect, such as product names, city names, or menu items. The model then prefers those words when the audio is unclear. This is often called keyword boosting.

Design the conversation for the weak spots. Numbers, names, and email addresses suffer the most on phone audio. So, we can do the following:

  • Read important numbers back to the user and ask them to confirm.
  • Ask the user to say digits one by one instead of as a big number.
  • Ask for spelling with a word for each letter, like "S as in Sugar".
  • Use the keypad for things like account numbers and PINs when possible.

Be careful with extra cleanup. Strong noise suppression before the model can remove parts of the speech, as we saw in Reason 4. We must test the model with and without it on real phone calls and keep whichever gives fewer mistakes.

Test on real phone audio. We must measure accuracy on real phone recordings, from real networks, with real background noise, not only on a clean test set.

Now we must have understood why phone audio makes speech recognition worse, how each step of a phone call damages our voice, and what we can do to make speech recognition work better on phone calls.

Frequently Asked Questions

What sample rate does a phone call use?

A traditional phone call uses a sample rate of 8 kHz, which means 8000 measurements of the sound wave per second. An 8 kHz sample rate can hold sounds only up to 4 kHz, and the phone band keeps only the sounds between about 300 Hz and 3400 Hz. This is called narrowband audio.

Does upsampling phone audio to 16 kHz improve speech recognition?

No. Upsampling 8 kHz phone audio to 16 kHz does not bring back the lost sounds. It only adds extra in-between samples, so the audio has the right sample rate, but everything above 4 kHz is still empty. The model gets 16 kHz audio with the top half silent, which is not what most of its training audio looks like.

Why do voice agents confuse "fifty" and "sixty" on phone calls?

Voice agents confuse "fifty" and "sixty" because the "f" and "s" sounds keep a large part of their energy in the high range that the phone cuts off. What remains of both is a faint noise, so the two sounds become hard to tell apart. Noise suppression, the codec, and lost packets can weaken them further.

Does HD Voice fix speech recognition on phone calls?

HD Voice helps, but only when both phones and every network on the way support it. It uses a wideband codec called AMR-WB at 16 kHz and keeps sounds up to about 7000 Hz, so more of the voice reaches the model. If any part of the path does not support it, the call falls back to narrowband.

Should we use noise suppression before speech recognition on phone audio?

Not always. Strong noise suppression can cut off the start or end of a soft word, or remove a quiet "s" because it looks like noise. We must test the model with and without it on real phone calls and keep whichever gives fewer mistakes.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.