How does a voice AI agent answer a phone call?
- Authors
- Name
- Amit Shekhar
- Published on
A voice AI agent answers a phone call by receiving the call from the phone network through a telephony provider and getting the caller's voice as a live stream of small audio chunks. It turns that voice into text, decides on a reply with an LLM, turns the reply back into voice, and sends it back into the same call.
In this blog, we will learn about how a voice AI agent answers a phone call. We will also see how a call travels from the caller's phone to our server, what the phone number and the telephony provider do, how the caller's voice reaches our agent as a stream of audio chunks, how the agent listens, thinks, and speaks back into the call, how it handles interruptions, and how the call ends or gets transferred to a human.
We will cover the following:
- What is a voice AI agent on a phone call?
- The journey of a phone call
- Step 1: The caller dials a phone number
- Step 2: The telephony provider receives the call
- Step 3: Our server decides what to do with the call
- Step 4: The caller's voice reaches our agent
- Step 5: The agent listens, thinks, and speaks
- Step 6: The agent's voice goes back into the call
- Handling interruptions on a phone call
- How the call ends
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning at Outcome School.
Let's get started.
What is a voice AI agent on a phone call?
A voice AI agent on a phone call is a system that picks up a real phone call, listens to the caller, understands what they want, and talks back in a natural voice, without any human on our side of the call.
Let's say we call a clinic to book an appointment. A voice says, "Hello, thank you for calling. How can I help you?" We say, "I want to book an appointment for tomorrow." The voice replies, "Sure, I have a slot at 11 AM. Does that work?"
There is no receptionist here. The one talking to us is a voice AI agent.
From the caller's side, it feels like a normal phone call. But, behind that call, the caller's voice leaves their phone, travels through the phone network, reaches our server, gets understood by AI models, and a reply comes back, often in around a second. We have a detailed blog on how to design a real-time voice AI agent that covers the full system end to end.
So, the question is: how does a normal phone call, which was built for humans talking to humans, reach an AI agent running on our server?
The best way to learn this is by following one phone call from start to end.
The journey of a phone call
Let's see the full journey in one diagram.
+----------------------------+
| Caller's phone |
+----------------------------+
↓
+----------------------------+
| Phone network |
+----------------------------+
↓
+----------------------------+
| Telephony provider |
+----------------------------+
↓
+----------------------------+
| Our server (the agent) |
| Listen -> Think -> Speak |
+----------------------------+
↓
+----------------------------+
| Telephony provider |
+----------------------------+
↓
+----------------------------+
| Phone network |
+----------------------------+
↓
+----------------------------+
| Caller's phone |
+----------------------------+
Here, we can see that the caller's voice travels through the phone network and the telephony provider to our server. Then, the agent's voice travels back through the same provider and the same phone network to the caller. Both directions stay open at the same time for the whole call.
Now, let's understand each step one by one.
Step 1: The caller dials a phone number
When we build a voice AI agent for phone calls, we first buy a phone number from a telephony provider.
A telephony provider is a company that connects the normal phone network to the internet, so that our software can make and receive real phone calls.
The normal phone network is the traditional network that connects landline and mobile phones around the world. It is called the PSTN, which stands for Public Switched Telephone Network. Our server, on the other hand, lives on the internet. These two worlds do not talk to each other directly.
Here comes the telephony provider into the picture. It sits between the phone network and the internet, and it translates between them.
Now, a caller dials that number from their mobile phone. The phone network looks at the number and finds out that this number belongs to our telephony provider. So, the phone network sends the call to the telephony provider.
Step 2: The telephony provider receives the call
Now, the call has reached the telephony provider. The provider does not know what to do with this call yet. It can play a message, forward the call, or connect it to an AI agent. So, the provider asks our server.
When we set up the phone number, we give the provider a URL of our server. This URL is called a webhook.
A webhook is a URL on our server to which the provider sends a request automatically whenever something happens, like a new incoming call.
So, when the call arrives, the provider sends a web request to our webhook. This request carries details about the call, like the number of the caller, the number that was dialed, and a unique ID for this call.
Let's say the request looks like below:
{ "call_id": "CA123", "from": "+15551234567", "to": "+15559876543" }
Here, we can see that the provider tells us who is calling, which of our numbers they dialed, and the ID it will use for this call from now on.
There is a second common way to connect a call to our system. Instead of a webhook, the call can be sent over SIP, which stands for Session Initiation Protocol. SIP is the standard way to start, manage, and end voice calls over the internet. In this setup, the provider forwards the call to a SIP server on our side, which turns the call audio into a stream for our agent. The rest of the idea stays the same. For the sake of understanding, we will follow the webhook path in this blog.
Planning your move into AI?
No matter which tech domain you work in today, with your effort, you can transition to these roles:
- AI Engineer
- Forward Deployed Engineer
- AI Agent Engineer
- Harness Engineer
- Voice AI Engineer
- AI Architect
- LLM Engineer
- LLM Inference Engineer
- Gen AI Engineer
- MLOps Engineer
- ML Engineer
- Deep Learning Engineer
Our AI and Machine Learning Program at Outcome School takes you there.
Step 3: Our server decides what to do with the call
Now, our server has received the webhook request. It must reply with instructions for the provider. For a voice AI agent, our server replies with two instructions:
- Answer the call.
- Open a live audio connection to an address on our server, like
wss://example.com/call-audio.
This live audio connection is usually a WebSocket.
A WebSocket is a connection between two computers that stays open, so that both sides can keep sending data to each other at any time.
A normal web request is like sending a letter and waiting for a reply. A WebSocket is like an open phone line between two computers. This is exactly what we need for a live call, because the caller keeps talking, and the agent keeps replying.
So, the provider answers the call, and opens a WebSocket connection to our server. Now, the caller is connected to our agent.
Step 4: The caller's voice reaches our agent
Now, the caller starts speaking. Let's see how their voice reaches our agent.
The provider does not wait for the caller to finish a sentence. It sends the voice to us as it happens, in small pieces. Each piece is called an audio chunk or an audio frame.
On a phone call, each chunk usually carries 20 milliseconds of sound. Means, every second of the call arrives as 50 small chunks, one after another, even when the caller is silent.
Let's say the caller says, "I want to book an appointment." This takes around 2 seconds to say. So, our server receives around 100 small chunks, one every 20 milliseconds, while the caller is still talking.
Now, the next question is: what does each chunk look like?
The audio that reaches our server is usually in an old and simple phone format. The sound is measured 8,000 times per second, which is called a sample rate of 8 kHz. Each measured value is squeezed into 1 byte using a method called mu-law (written as μ-law). This format is part of a standard called G.711, and it has been used in telephone networks for decades. Some networks use a very similar method called A-law, but the idea is the same.
So, the math for one chunk is as below:
8,000 samples per second * 0.02 seconds = 160 samples = 160 bytes
Means, each 20-millisecond chunk is just 160 bytes of audio.
The provider wraps each chunk inside a small message and sends it over the WebSocket. Before that, it converts the audio bytes into text using base64 encoding, so that they can travel inside a text message.
A media message looks like below:
{ "event": "media", "call_id": "CA123", "payload": "fn5+fn5+..." }
Here, we can see that:
eventtells us that this message carries audio.call_idtells us which call this audio belongs to.payloadis the 20 milliseconds of caller audio, encoded as base64 text.
Apart from the audio messages, the provider also sends a message when the stream starts and when it stops.
Now, our server has a steady stream of the caller's voice.
A quick note for you
No matter which tech domain you work in, get familiar with these topics:
- LLM
- RAG
- MCP
- Agent
- Fine-tuning
- Quantization
We put it all together in one video:
AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization
No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.
Now, let's get back to the topic.
Step 5: The agent listens, thinks, and speaks
Before the AI models can use the audio, our server prepares it. First, it decodes the base64 text back into bytes. Then, it converts the mu-law bytes back into normal audio values. After that, it often converts the 8 kHz audio to 16 kHz, because many speech-to-text models expect 16 kHz audio. This conversion is called resampling.
Note: Resampling from 8 kHz to 16 kHz does not bring back the sound details that were lost in the 8 kHz phone audio. It only puts the audio in the shape that the model expects.
Now, the prepared audio goes through a pipeline with a few parts. A common pipeline looks like below:
+---------------------------------+
| Caller audio (16 kHz) |
+---------------------------------+
↓
+---------------------------------+
| VAD: Is someone speaking? |
+---------------------------------+
↓
+---------------------------------+
| STT: Speech -> Text |
+---------------------------------+
↓
+---------------------------------+
| LLM: Text -> Reply text |
+---------------------------------+
↓
+---------------------------------+
| TTS: Reply text -> Speech |
+---------------------------------+
↓
+---------------------------------+
| Agent audio |
+---------------------------------+
Let's understand each part.
VAD (Voice Activity Detection): It looks at each small chunk and decides whether it contains human speech or just silence and noise. It helps the agent find out when the caller starts speaking and when the caller stops.
STT (Speech-to-Text): It converts the caller's speech into text. For phone calls, STT usually works in streaming mode. It gives partial text while the caller is still talking, and final text once the caller stops.
LLM (Large Language Model): It reads the text, along with the conversation so far and the instructions we gave it, like "You are a receptionist for a clinic." Then, it writes the reply text. If the reply needs real data, like free appointment slots, the LLM recommends a tool and its input, and our code runs that tool and gives the result back to the LLM.
TTS (Text-to-Speech): It converts the reply text into a natural voice.
We have a detailed blog on cascaded vs speech-to-speech voice agents that explains how this pipeline compares with a single speech-to-speech model.
Let's see this with our clinic example:
- The caller says, "I want to book an appointment for tomorrow."
- VAD notices that the caller started speaking, and later notices around half a second of silence. So, the agent decides that the caller has finished the turn.
- STT gives the text: "I want to book an appointment for tomorrow."
- The LLM recommends a tool to check free slots for tomorrow. Our code runs it, finds a slot at 11 AM, and gives this result back to the LLM. Then, the LLM writes: "Sure, I have a slot at 11 AM. Does that work?"
- TTS turns that sentence into voice.
To keep the delay low, these parts do not wait for each other to fully finish. The LLM sends out the reply in small pieces while it is still writing it, and as soon as the first sentence is ready, TTS starts turning it into voice. So, the caller starts hearing the reply while the rest of the reply is still being written.
One more thing to notice: in many agents, the agent speaks first. As soon as the call connects, it plays a greeting like "Hello, thank you for calling. How can I help you?" This greeting is TTS audio sent right at the start of the call.
To learn Tool use in Agents, Agent Architecture, and how to Design a Real-Time Voice AI Agent, and to build an AI Coding Agent from scratch, check out our AI and Machine Learning Program at Outcome School.
Step 6: The agent's voice goes back into the call
Now, the agent has a reply in voice. But this voice is in the format of the TTS model, which is often 16 kHz, 24 kHz, or higher. It is not in the 8 kHz mu-law format of the phone call.
So, our server does the same preparation in the reverse direction as below:
- First, it resamples the TTS audio down to 8 kHz.
- Then, it encodes the audio into mu-law, so that each sample becomes 1 byte.
- After that, it cuts the audio into small chunks and encodes each chunk as base64.
- Finally, it sends each chunk to the provider over the same WebSocket.
The message going back looks like below:
{ "event": "media", "call_id": "CA123", "payload": "//7+/v7+..." }
Here, we can see that it has the same shape as the message we received. Only the direction is different.
The provider takes these chunks, in order, and plays them into the phone call. The phone network carries the sound to the caller's phone, and the caller hears, "Sure, I have a slot at 11 AM. Does that work?"
Then, the caller replies, and the whole loop runs again for every turn until the call ends.
Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.
Handling interruptions on a phone call
Callers do not always wait for the agent to finish. Many times, they start speaking while the agent is still talking. This is called barge-in.
Let's say the agent is saying, "Sure, I have a slot at 11 AM, and I also have a slot at..." and the caller says, "11 AM is fine."
The agent must stop talking right away. But, our server has already sent many audio chunks to the provider. The provider keeps these chunks in a queue, called a buffer, and plays them in the order they arrived. Even if our server stops sending new chunks, the caller keeps hearing the old ones.
So, how can we stop the agent instantly? The answer is a clear message. Many telephony providers support a message like this, and the exact name can differ from provider to provider.
Our server keeps listening to the caller even while the agent is speaking. When VAD finds that the caller has really started speaking, and it is not just a short noise, our server does three things:
- It stops the LLM and TTS from producing more of the reply.
- It sends a clear message to the provider, like
{ "event": "clear", "call_id": "CA123" }, which tells the provider to throw away all the agent audio that is waiting to be played. - It updates the conversation history, so that the LLM knows that the caller heard only part of the reply.
Now, the agent goes quiet and listens to what the caller is saying.
How the call ends
There are three common ways in which a call ends.
The caller hangs up: The provider notices that the call has ended. It sends a stop event to our server and closes the WebSocket. Our server stops the pipeline and saves the details of the call, like the transcript and the duration.
The agent hangs up: When the task is done, the agent says something like, "Your appointment is booked. Have a nice day." Then, our server tells the provider to end the call. In many agents, the LLM recommends an "end call" tool, and our code runs it after the goodbye has finished playing.
The agent transfers the call to a human: Sometimes, the caller asks for a real person, or the request is too complex for the agent. In this case, the LLM recommends a "transfer call" tool, and our code tells the provider to connect the caller to another phone number, like the clinic's front desk. The provider moves the call to that number, and our agent leaves the call.
Now we must have understood how a voice AI agent answers a phone call, how the call travels from the phone network to our server, how the caller's voice arrives as small audio chunks, how the agent listens, thinks, and speaks back into the call, how it handles interruptions, and how the call ends.
Frequently Asked Questions
What does a telephony provider do for a voice AI agent?
A telephony provider connects the normal phone network to the internet, so that our software can make and receive real phone calls. It gives us a phone number, receives the call from the phone network, asks our server what to do through a webhook, and then streams the caller's voice to our agent and plays the agent's voice back into the call.
How big is one audio chunk on a phone call?
One audio chunk on a phone call usually carries 20 milliseconds of sound. Phone audio is measured 8,000 times per second and each value is squeezed into 1 byte using mu-law, so one chunk is 160 samples, which is 160 bytes. The provider encodes these bytes as base64 text before sending them over the WebSocket.
Why does a voice AI agent convert phone audio to 16 kHz?
A voice AI agent converts the 8 kHz phone audio to 16 kHz because many speech-to-text models expect 16 kHz audio. This conversion is called resampling. It does not bring back the sound details that were lost in the 8 kHz phone audio. It only puts the audio in the shape that the model expects.
Can a phone call reach a voice AI agent without a webhook?
Yes. Instead of a webhook, the call can be sent over SIP, which stands for Session Initiation Protocol. SIP is the standard way to start, manage, and end voice calls over the internet. In this setup, the provider forwards the call to a SIP server on our side, which turns the call audio into a stream for our agent.
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
You can connect with me on:
Follow Outcome School on:
