How does voicemail and answering machine detection work?

Authors
How does voicemail and answering machine detection work?

Voicemail and answering machine detection is a check that listens to the first few seconds of an outgoing phone call and decides whether a real person or a recorded greeting has answered. It looks at the pattern of speech and silence, listens for the beep, and sometimes reads the words, so that the caller knows whether to start talking or to leave a message.

In this blog, we will learn about how voicemail and answering machine detection works. We will also see why a calling system needs it, how a person and a machine sound different in the first few seconds, how simple timing rules catch many machines, how the beep is found, how AI models and transcripts make the decision more accurate, and where the detection works well and where it fails.

We will cover the following:

  • What is Answering Machine Detection?
  • Why do we need Answering Machine Detection?
  • How a person and a machine sound different
  • Approach 1: Timing rules on speech and silence
  • Finding the beep
  • Approach 2: An AI model that listens to the audio
  • Approach 3: Reading the words
  • Speed vs accuracy
  • Where it fails
  • How a voice agent uses it end to end

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is Answering Machine Detection?

Answering Machine Detection, also called AMD, is the process of finding out, within the first few seconds of a call, whether the call was picked up by a live person or by a voicemail system.

A voicemail system answers the call when a person does not pick up, plays a recorded greeting, and then records a message. We all have heard such a greeting: "Hi, you have reached Priya. I cannot take your call right now. Please leave a message after the beep."

From the sound alone, the detector gives one of a few answers:

  • Human: a live person picked up.
  • Machine: a voicemail greeting is playing.
  • Machine, greeting ended: the greeting has finished, usually with a beep, and now the voicemail is recording.
  • Fax: a fax machine answered.
  • Unknown: the detector could not decide in time.

Why do we need Answering Machine Detection?

Let's say a clinic uses a voice agent to call 1,000 patients and remind them about their appointments tomorrow.

When a patient picks up, the voice agent must talk to them and answer their questions.

When a voicemail picks up, the voice agent must do something else. It must either wait for the beep and leave a short message, or hang up and try again later.

Without detection, the system must guess, and both guesses fail:

  • If it always starts talking, the voice agent talks while the greeting is still playing. The voicemail records only after the beep, so the patient later hears only the second half of the message.
  • If it always waits for a beep, a live person says "Hello?" and hears silence because the system is waiting for a beep that will never come. The person hangs up.

So, the system needs to know who answered, and it needs to know quickly. Here comes the Answering Machine Detection into the picture.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

How a person and a machine sound different

Before jumping into the approaches, we must understand the one difference that every approach depends on.

When a person answers a phone, they usually say something short and then stop to wait for us, like below:

"Hello?"  ->  silence (waiting for the caller)

When a voicemail answers, it plays a long greeting without stopping to wait for anyone, like below:

"Hi, you have reached Priya. I cannot take your call right now. Please leave a message after the beep."  ->  beep

Means, a person speaks briefly and then goes quiet. A machine speaks for a long time and then beeps.

The detector turns this difference into a decision. Let's see how.

Approach 1: Timing rules on speech and silence

This approach is fast and cheap, and it is still widely used.

First, the detector cuts the incoming audio into tiny pieces called frames, often 10 to 20 milliseconds each. For every frame, it measures how loud the sound is. This loudness is called energy.

  • If the energy is above a level, the frame is marked as speech.
  • If the energy is below that level, the frame is marked as silence.

Now, the detector does not see audio anymore. It sees a timeline of speech and silence, like below:

silence | speech | silence | speech | speech | silence | ...

Then, it applies a set of simple timing rules on this timeline.

For the sake of example, let's look at the rules in the AMD module of Asterisk, a popular open-source phone system. Its default settings are as below:

SettingDefault valueWhat it means
Initial silence2500 msLongest silence allowed before anyone speaks
Greeting1500 msLongest a greeting can last before it is a machine
After greeting silence800 msSilence after a greeting that means a person waits
Total analysis time5000 msMaximum time allowed to decide
Minimum word length100 msShortest burst of speech counted as a word
Between words silence50 msShortest gap that splits two words

And the rules work like below:

  • If nobody speaks for more than 2.5 seconds, it decides Machine, because a person who picks up a phone usually speaks quickly.
  • If the first greeting keeps going for more than 1.5 seconds, it decides Machine, because "Hello?" is short and a recorded greeting is long.
  • If the greeting has too many words, it decides Machine. A word here is any burst of speech of at least 100 ms, and two words are split by a gap of at least 50 ms.
  • If a short greeting is followed by 0.8 seconds of silence, it decides Human, because the person has stopped to wait for us.
  • If 5 seconds pass without any of the above, it gives up and returns Unknown.

The best way to learn this is by taking an example.

Call 1: A person picks up.

  • 0.0 s to 0.4 s: silence. This is under the 2.5-second limit.
  • 0.4 s to 0.9 s: speech, "Hello?". This is one short word, 0.5 seconds long.
  • 0.9 s to 1.7 s: silence for 0.8 seconds.

A short greeting followed by 0.8 seconds of silence. The decision is Human at 1.7 seconds.

Call 2: A voicemail picks up.

  • 0.0 s to 0.6 s: silence. This is also under the limit.
  • 0.6 s onwards: speech, "Hi, you have reached Priya...". The speech keeps going.
  • At 2.1 s, the greeting has already lasted 1.5 seconds.

The greeting is too long. The decision is Machine at 2.1 seconds.

This is how simple timing rules catch many voicemails within two or three seconds, without reading any words.

The issue with this approach is that it only looks at loudness and time. A person who answers with a long sentence looks like a machine, and a voicemail with a very short greeting looks like a person. Let's first see how the beep is found, and then we will see how the next approaches solve this issue.

Finding the beep

Knowing that a machine answered is only half of the job. To leave a message, we must start speaking after the beep, or the first part of our message is lost.

So, how can the detector find the beep? The answer is that a beep is a simple sound.

Human speech is a mix of many frequencies that change every moment. Frequency means how fast the sound wave goes up and down, and it is measured in Hertz (Hz). A beep is different. It is one steady frequency, held for a fraction of a second or more.

On the incoming audio, the detector does this:

  • It splits each frame of audio into its frequencies, to see how much of each frequency is present.
  • It checks whether almost all the energy sits at one single frequency.
  • It checks whether that same frequency stays steady across many frames in a row.

If both checks pass, it is a beep. We have a detailed blog on how DTMF keypad input works in a voice agent that explains how the same kind of pure-tone check finds key presses on a call.

The frequency of the beep is not fixed. Different voicemail systems use different tones, and some greetings end with no beep at all. So, a good beep detector does not look for one exact frequency. It looks for any pure, steady tone. If no beep comes, it waits for a long silence after the greeting and treats that silence as the end of the greeting.

This is why many detectors report two different machine results: one when they first know it is a machine, and one when the greeting has ended and it is the right time to start speaking.

The same idea works for fax machines. A fax machine that answers a call sends a long, steady, high tone at 2100 Hz. The detector looks for that pure tone and returns Fax.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Approach 2: An AI model that listens to the audio

Now, let's see how the next approach solves the issue of timing rules. Timing rules use only loudness and time, but the sound carries much more information than that.

A recorded greeting often sounds cleaner and more even than a live voice, with a steady rhythm, because it was usually recorded in a quiet room. A live person on a phone has background noise, breathing, and an uneven, natural rhythm.

So, instead of writing rules by hand, we can train a model to learn these differences from examples.

First, we collect thousands of recorded call openings, each labeled as human or machine.

Then, we turn the first few seconds of each call into numbers that describe the sound. A common choice is a spectrogram, which is like a picture of the audio that shows which frequencies are present at every moment.

After that, we train a classifier on these examples. A classifier is a model that puts an input into one of a few groups. Here, the groups are human and machine.

Finally, on a live call, the model looks at the incoming audio every fraction of a second and gives a score, for example "92% machine". When the score crosses a set level, the detector decides.

The model learns patterns that are hard to write as rules, such as the clean quality of a recording or the way a person's voice rises in "Hello?". Hence, it is usually more accurate than plain timing rules on the tricky cases. We have a detailed blog on how silence-based and model-based turn detection differ that covers the same trade-off for deciding when a user has finished speaking.

But, a person who says "Hello, this is Rahul speaking, how can I help you?" gives a long and clear greeting, and this can still confuse the model.

Approach 3: Reading the words

The issue with the audio model is that it does not understand the words. The next approach solves it by using what modern voice agents already have: a speech-to-text model.

A speech-to-text model converts spoken audio into text as the audio comes in. So, while the greeting is playing, the voice agent already has its text.

Now, the detector can read the words. Voicemail greetings use common phrases:

  • "You have reached..."
  • "I cannot take your call right now."
  • "Please leave a message after the tone."
  • "The person you are calling is not available."

A live person usually says "Hello?", "Yes?", "Who is this?", or their own name.

The text can be checked in two ways:

  • Keyword matching: look for known voicemail phrases. It is fast, but it misses greetings that use other words.
  • An LLM classifier: send the transcript to an LLM (Large Language Model) with a short instruction, like below:
Here is what was said when a phone call was answered:
"Hi, you have reached Priya. I cannot take your call right now."
Was this answered by a live person or a voicemail greeting?
Answer with one word: human or voicemail.

Here, the LLM reads the meaning of the words, not just exact phrases. It can tell that "Sorry I missed you, drop me a message" is also a voicemail, even though it does not contain the words "leave a message".

The issue with this approach is time. Speech-to-text needs a moment to produce the words, and the LLM needs a moment to answer. So, in practice, many systems combine the approaches: timing rules and the beep detector give a fast first answer, and the transcript confirms it or corrects it.

To learn Prompt Engineering, LLM Fundamentals, and how to Design a Real-Time Voice AI Agent that combines speech-to-text with an LLM, check out our AI and Machine Learning Program at Outcome School.

Speed vs accuracy

Every detector faces a trade-off. If it decides very fast, it has heard very little, so it makes more mistakes. If it waits longer, it is more accurate, but the person on the other side waits in silence. With the default settings we saw earlier, a timing-rules detector can take up to 5 seconds to decide.

And waiting in silence is costly. When a person says "Hello?" and hears nothing for two or three seconds, some of them think it is a spam call and hang up.

That is why systems offer two ways to run detection:

  • Blocking: the call waits until the detector decides, and only then does the voice agent start speaking. It is simple, but the person hears silence during detection.
  • Background: the voice agent starts the conversation right away, and the detector keeps listening in the background. When it decides "Machine", the system stops the conversation and switches to the voicemail flow.

The background way gives a better experience to real people, because they hear a reply quickly. The cost is that, on a voicemail, the voice agent sometimes speaks a few words over the greeting before the detector decides.

Where it fails

No detector is perfect, because people and greetings do not always follow the usual pattern.

A person sounds like a machine:

  • A person answers with a long sentence: "Good morning, Sharma Electronics, Rahul speaking, how may I help you?"
  • A person answers while talking to someone else in the room.

A machine sounds like a person:

  • A very short greeting: "Priya. Leave a message." This is as short as a person saying hello.
  • A greeting that starts with "Hello?" and then a pause before the rest of the message.
  • A phone carrier's message, such as "The number you have dialed is not reachable."

The beep is hard to find:

  • The greeting ends with no beep.
  • Background music in the greeting has steady tones that look like a beep.

Here, we can notice that each approach fixes some of these cases. Timing rules fail on long human greetings. The audio model handles some of them. The transcript catches many of the rest, because "Rahul speaking, how may I help you" does not mean "leave a message".

When the detector still cannot decide, it returns Unknown, and the system must pick a safe default. A common choice is to treat Unknown as a human, because hanging up on a real person is worse than speaking to a recording for a few seconds.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

How a voice agent uses it end to end

Now, let's see the full flow for one outgoing call, like below:

        The voice agent dials the number
                      ↓
           The other side picks up
                      ↓
   The detector listens to the first few seconds
     (timing rules + audio model + transcript)
                      ↓
       +--------------+--------------+
       ↓              ↓              ↓
     Human         Machine          Fax
       ↓              ↓              ↓
  Start the      Wait for the     Hang up
  conversation   beep or silence
                      ↓
                Leave a short
                message, or hang
                up and retry later

Here, the detector combines the three approaches. Timing rules give a quick first guess, the audio model adds a score, and the transcript confirms it. The result decides the path of the call.

This way, the clinic from our example can call 1,000 patients, talk to everyone who picks up, and leave a complete reminder message for everyone who does not.

Now we must have understood what voicemail and answering machine detection is, how it works, and where it is used.

Frequently Asked Questions

Does answering machine detection need speech-to-text?

No. Simple timing rules on speech and silence, together with a beep detector, look only at loudness, time, and frequency, and they catch many voicemails within two or three seconds without reading any words. Speech-to-text is added when we want the transcript to confirm or correct that fast first answer.

How long does answering machine detection take to decide?

It usually takes a few seconds. Timing rules catch many voicemails within two or three seconds. With the default settings of the AMD module in Asterisk, the detector can take up to 5 seconds, and if it still has no answer by then, it returns Unknown.

What should a voice agent do when the detection result is Unknown?

It should pick a safe default. A common choice is to treat Unknown as a human and start the conversation, because hanging up on a real person is worse than speaking to a recording for a few seconds.

What happens if a voicemail greeting has no beep?

The detector waits for a long silence after the greeting instead. Different voicemail systems use different tones, and some greetings end with no beep at all, so a good detector treats a long silence after the greeting as the end of the greeting and the right time to start speaking.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.