How do SIP and RTP carry a phone call to a voice agent?
- Authors
- Name
- Amit Shekhar
- Published on
SIP and RTP are the two protocols that connect a phone call to a voice agent. SIP sets up the call and ends it, and RTP carries the actual voice, in small packets, in both directions while the call is going on.
In this blog, we will learn about how SIP and RTP carry a phone call to a voice agent. We will also see how a normal phone call reaches the internet, how SIP rings the voice agent and agrees on the audio format, how RTP packs the voice into small packets, what happens inside the voice agent when those packets arrive, how the agent speaks back, and how the call finally ends.
We will cover the following:
- The journey of a phone call to a voice agent
- What is SIP?
- What does a SIP call setup look like?
- What is SDP?
- What is RTP?
- What is inside an RTP packet?
- What the voice agent does with the RTP packets
- How the voice agent speaks back
- How the call ends
- SIP vs RTP
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning at Outcome School.
Let's get started.
The journey of a phone call to a voice agent
Let's say a customer picks up a normal mobile phone and dials the support number of a company. On the other side, there is no human. There is a voice agent, which is a system that listens to the caller, understands what the caller says, and replies with a voice.
The voice agent runs on a server on the internet. But, the customer is on the normal phone network, the one that has been carrying phone calls for decades. This network is called the PSTN (Public Switched Telephone Network).
So, the first question is: how does a call on the phone network reach a server on the internet?
The answer is a SIP trunk provider. This is a company that provides phone numbers and connects the phone network to the internet. When a call comes to one of its numbers, it converts the call into internet traffic and forwards it to our server.
We have a detailed blog on how a voice AI agent answers a phone call that covers the full call flow end to end.
From this point onwards, two protocols do the main work:
- SIP sets up the call, and later ends it.
- RTP carries the voice while the call is going on.
In many setups, the voice agent does not handle SIP and RTP directly. A media server sits in front of it, handles both of them, and passes the plain audio to the voice agent. For the sake of understanding, we will treat them as one voice agent server in this blog.
The full journey looks like below:
+----------------------+
| Caller's phone |
+----------------------+
↓
+----------------------+
| Phone network (PSTN) |
+----------------------+
↓
+----------------------+
| SIP trunk provider |
+----------------------+
↓
SIP: set up the call
RTP: carry the voice
↓
+----------------------+
| Voice agent server |
+----------------------+
Now, let's understand each of these two protocols one by one.
What is SIP?
SIP (Session Initiation Protocol) is a protocol used to start, manage, and end a call between two parties over the internet.
Let's say we call a friend. First, the phone of our friend rings. Then, our friend picks up. Then, we talk. Finally, one of us hangs up.
SIP handles every one of these steps except the talking. It rings the other side, it learns that the other side picked up, and it handles the hang up. The talking itself is the job of RTP.
SIP does not carry any voice. It only carries messages about the call.
SIP messages are plain text, very similar to the messages a web browser sends to a website. A SIP message has a first line that says what it wants, then a few header lines, and sometimes a body.
The main SIP messages that we must know are:
INVITE: "I want to start a call with you."100 Trying: "I got your request, I am working on it."180 Ringing: "The phone is ringing."200 OK: "Yes, the call is accepted."ACK: "I got your OK, the call is now set up."BYE: "I want to end the call."
The first few lines of a simplified INVITE message look like below:
INVITE sip:+15550100@agent.example.com SIP/2.0
From: <sip:+15550199@provider.example.com>
To: <sip:+15550100@agent.example.com>
Call-ID: a84b4c76e66710
Here, the first line says "start a call with this number". The From line is the caller, the To line is the number that was dialed, and the Call-ID is a unique id that ties all the messages of this one call together.
Now, let's see how these messages work together to set up a call.
What does a SIP call setup look like?
Let's say the SIP trunk provider receives the customer's call and wants to hand it over to our voice agent server.
Step 1: The SIP trunk provider sends an INVITE to the voice agent server. This message says who is calling, which number was dialed, and which audio formats the provider can use.
Step 2: The voice agent server replies with 100 Trying, so that the provider knows the message has reached and does not resend it.
Step 3: The voice agent server replies with 180 Ringing. The caller hears the ringing tone.
Step 4: The voice agent server replies with 200 OK. This means the call is answered. This message also says which audio format the server picked and where it wants to receive the audio.
Step 5: The SIP trunk provider sends an ACK. Now, the call is fully set up.
The flow looks like below:
SIP trunk provider Voice agent server
| |
|-------- INVITE ----------->|
|<------- 100 Trying --------|
|<------- 180 Ringing -------|
|<------- 200 OK ------------|
|-------- ACK -------------->|
| |
|<====== RTP (voice) =======>|
| |
Here, we can notice that once the ACK is sent, SIP has no more work to do for now, and the voice starts flowing as RTP packets.
Planning your move into AI?
No matter which tech domain you work in today, with your effort, you can transition to these roles:
- AI Engineer
- Forward Deployed Engineer
- AI Agent Engineer
- Harness Engineer
- Voice AI Engineer
- AI Architect
- LLM Engineer
- LLM Inference Engineer
- Gen AI Engineer
- MLOps Engineer
- ML Engineer
- Deep Learning Engineer
Our AI and Machine Learning Program at Outcome School takes you there.
But, one question is still pending. When the provider sends the INVITE, how does it tell the server which audio formats it can use and where to send the voice? The answer is SDP.
What is SDP?
SDP (Session Description Protocol) is a short text description of a call that tells the other side where to send the audio and in which format.
SDP travels inside the body of the SIP messages. Both the INVITE and the 200 OK carry one.
A simplified SDP inside the INVITE looks like below:
c=IN IP4 203.0.113.10
m=audio 40000 RTP/AVP 0 8
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
Before reading it, we must know one word. A codec is the method used to turn sound into bytes and bytes back into sound.
Here, we have the following:
c=IN IP4 203.0.113.10is the internet address where the provider wants to receive audio.m=audio 40000 RTP/AVP 0 8says that this is an audio stream over RTP, received on port40000, and the provider supports formats0and8.a=rtpmap:0 PCMU/8000says that format0is PCMU, which is the G.711 μ-law codec, at 8000 samples per second.a=rtpmap:8 PCMA/8000says that format8is PCMA, which is the G.711 A-law codec, also at 8000 samples per second.
G.711 is the classic codec of the phone network. It is very simple and it is supported almost everywhere.
The voice agent server reads this list and picks one format. Then, it puts its own SDP inside the 200 OK, with its own address, its own port, and the format it picked.
This is called the offer/answer model. The INVITE makes an offer, and the 200 OK gives the answer.
Now, both sides know where to send the audio and in which format to send it.
Now, it's time to learn about the part that carries the actual voice.
What is RTP?
RTP (Real-time Transport Protocol) is a protocol that carries audio and video over the internet in small packets, with the information needed to play them in the right order and at the right time.
The voice is cut into small pieces, and each piece goes in its own packet. For phone calls, a common size is 20 milliseconds of audio per packet.
So, one second of speech becomes 50 packets, because 1000 milliseconds divided by 20 milliseconds is 50.
RTP packets are usually sent over UDP, which is a fast way of sending data on the internet that does not wait for lost data to be resent.
Now, the question is: why not use a method that resends lost data?
The answer is time. In a live conversation, a piece of voice that arrives late is useless. If a packet is lost, waiting to get it again would freeze the call. It is better to skip a 20 millisecond gap and keep the conversation moving.
We have a detailed blog on how WebRTC and WebSocket differ for voice AI agents that explains how UDP and TCP behave for live voice in depth.
What is inside an RTP packet?
Every RTP packet has a header, usually 12 bytes, followed by the audio itself.
The important fields of the header are:
- Payload type: the number of the audio format, the same number that was agreed in SDP. For example,
0means PCMU. - Sequence number: starts from a random number and goes up by 1 for every packet. The receiver uses it to put packets back in order and to find missing packets.
- Timestamp: tells which moment of the audio this packet belongs to. The receiver uses it to play the audio at the right time.
- SSRC: a random number that identifies the source of the audio, so that the receiver can tell one audio stream apart from another.
Let's do the math for a G.711 call with 20 millisecond packets.
G.711 uses 8000 samples per second, and each sample takes 1 byte.
So, 20 milliseconds of audio is:
8000 samples per second * 0.020 seconds = 160 samples = 160 bytes
The timestamp goes up by 160 for every packet, because each packet carries 160 samples. In a real call, it starts from a random number, but just for the sake of understanding, we start it at 0. The packets look like below:
| Packet | Sequence number | Timestamp | Audio bytes |
|---|---|---|---|
| 1 | 1000 | 0 | 160 |
| 2 | 1001 | 160 | 160 |
| 3 | 1002 | 320 | 160 |
| 4 | 1003 | 480 | 160 |
Let's say packet 3 is lost on the way. The receiver sees 1000, 1001, and then 1003. It immediately knows that 1002 is missing, and it can fill that 20 millisecond gap with a smooth guess of the missing sound instead of a click. This is called packet loss concealment.
Similarly, if packet 4 arrives before packet 3, the receiver uses the sequence numbers to put them back in the right order. This waiting and reordering is done by a jitter buffer on the receiving side.
A quick note for you
No matter which tech domain you work in, get familiar with these topics:
- LLM
- RAG
- MCP
- Agent
- Fine-tuning
- Quantization
We put it all together in one video:
AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization
No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.
Now, let's get back to the topic.
What the voice agent does with the RTP packets
Now, the voice arrives at the voice agent server as a flow of RTP packets. Let's see what happens next.
Step 1: The server reads each RTP packet, checks the sequence number, and puts the packets in order with a jitter buffer.
Step 2: The server removes the RTP header and takes out the 160 bytes of G.711 audio.
Step 3: The server decodes the G.711 bytes into plain audio samples.
Step 4: The server converts the audio from 8000 samples per second to the rate that the speech-to-text model expects, which is often 16000 samples per second.
Step 5: The server sends the audio to the speech-to-text model, which turns the voice into text, piece by piece, while the caller is still speaking.
Step 6: The text goes to the LLM (Large Language Model), which decides what to reply.
The flow looks like below:
+----------------------------+
| RTP packets (G.711, 8 kHz) |
+----------------------------+
↓
+----------------------------+
| Jitter buffer (reorder) |
+----------------------------+
↓
+----------------------------+
| Decode G.711 to samples |
+----------------------------+
↓
+----------------------------+
| Resample 8 kHz to 16 kHz |
+----------------------------+
↓
+----------------------------+
| Speech-to-text |
+----------------------------+
↓
+----------------------------+
| LLM |
+----------------------------+
Note: Resampling from 8 kHz to 16 kHz does not bring back the sound details that the phone network already removed. It only changes the format, so that the speech-to-text model can accept the audio.
To master AI and ML System Design and learn how to Design a Real-Time Voice AI Agent, from the audio pipeline to the LLM, check out our AI and Machine Learning Program at Outcome School.
How the voice agent speaks back
Now, the voice agent must send its reply back to the caller as voice.
RTP works in both directions at the same time. The provider sends the caller's voice to the server, and the server sends the agent's voice to the provider. Each side sends to the address and port that the other side gave in its SDP.
Step 1: The LLM writes the reply text.
Step 2: A text-to-speech model turns the text into audio. Often, it produces audio at a higher rate like 24000 samples per second.
Step 3: The server converts the audio down to 8000 samples per second.
Step 4: The server encodes the audio into G.711, the format agreed in SDP.
Step 5: The server cuts the audio into 160 byte pieces, adds an RTP header to each piece with the next sequence number and a timestamp that goes up by 160, and sends one packet every 20 milliseconds.
If the server sends all the packets at once, the receiving side gets a big burst of audio that its jitter buffer cannot hold, and parts of the reply can get dropped. So, the server sends them at the same pace at which the audio is played.
Now, let's say the caller interrupts the agent in the middle of a reply. The voice agent detects the caller's voice in the incoming RTP packets, and it immediately stops sending its own outgoing RTP packets and throws away the rest of the reply audio. The caller hears the agent stop.
Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.
How the call ends
Finally, the call must end. This is the job of SIP again.
Let's say the caller hangs up. The SIP trunk provider sends a BYE to the voice agent server. The server replies with 200 OK. Both sides stop sending RTP packets, and the call is over.
Similarly, when the voice agent has finished helping the caller, it can end the call itself by sending the BYE, and the provider replies with 200 OK.
The complete life of the call looks like below:
SIP trunk provider Voice agent server
| |
|-------- INVITE ----------->|
|<------- 200 OK ------------|
|-------- ACK -------------->|
| |
|<====== RTP (voice) =======>|
| |
|-------- BYE -------------->|
|<------- 200 OK ------------|
| |
Here, we can see that in a simple call, SIP appears only at the start and at the end, and RTP fills the whole middle part of the call.
SIP can also come back in the middle of a call. For example, when the voice agent must transfer the caller to a human, it uses SIP messages to move the call to the human's phone.
SIP vs RTP
Let me tabulate the differences between SIP and RTP for your better understanding.
| Point | SIP | RTP |
|---|---|---|
| Job | Sets up, manages, and ends the call | Carries the voice during the call |
| What it carries | Text messages about the call | Small packets of audio |
| When it is used | Mostly at the start and at the end | During the whole conversation |
| Example | INVITE, 200 OK, ACK, BYE | Packets with sequence number and audio |
| Usually runs over | UDP, TCP, or TLS | UDP |
Now we must have understood how SIP and RTP carry a phone call to a voice agent, from the moment the phone rings to the moment the call ends.
Frequently Asked Questions
Why is RTP sent over UDP?
RTP is usually sent over UDP because UDP is fast and does not wait for lost data to be resent. In a live conversation, a piece of voice that arrives late is useless, and waiting for a lost packet would freeze the call. It is better to skip a 20 millisecond gap and keep the conversation moving.
How many RTP packets does a phone call send every second?
A phone call usually sends 50 RTP packets every second in each direction. Each packet carries 20 milliseconds of audio, and 1000 milliseconds divided by 20 milliseconds is 50. With G.711 at 8000 samples per second and 1 byte per sample, each packet carries 160 bytes of audio behind a header of usually 12 bytes.
What happens when an RTP packet is lost?
The receiver notices the gap from the sequence numbers. For example, if it sees 1000, 1001, and then 1003, it knows that 1002 is missing. It does not wait for the packet again. Instead, it fills that 20 millisecond gap with a smooth guess of the missing sound instead of a click, which is called packet loss concealment.
What is a SIP trunk provider?
A SIP trunk provider is a company that provides phone numbers and connects the normal phone network, the PSTN, to the internet. When a call comes to one of its numbers, it converts the call into internet traffic and forwards it to our voice agent server, using SIP to set up the call and RTP to carry the voice.
Does resampling phone audio to 16 kHz improve its quality?
No. Resampling phone audio from 8 kHz to 16 kHz does not bring back the sound details that the phone network already removed. It only changes the format, so that a speech-to-text model that expects 16000 samples per second can accept the audio.
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
You can connect with me on:
Follow Outcome School on:
