How do WebRTC and WebSocket differ for voice AI agents?

WebRTC and WebSocket are two ways to carry live audio between a user and a voice AI agent. WebSocket sends every piece of audio over one reliable connection and waits for anything that is lost, while WebRTC is built for live voice, so it skips lost or late audio and keeps the conversation moving.

In this blog, we will learn about how WebRTC and WebSocket differ for voice AI agents. We will also see what a voice AI agent needs from the network, how WebSocket carries audio, how WebRTC carries audio, why a lost packet hurts one of them more than the other, what extra work WebRTC does for voice, and when to use which one.

We will cover the following:

• What a voice AI agent needs from the network
• How audio travels over the internet
• What is WebSocket?
• How a voice agent uses WebSocket
• What is WebRTC?
• How a voice agent uses WebRTC
• What happens when a packet is lost
• The extra work WebRTC does for voice
• WebRTC vs WebSocket
• When to use which one

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What a voice AI agent needs from the network

A voice AI agent is a system that listens to a person, understands what they said, and speaks back in a natural voice. We talk to it the same way we talk to a person on a phone call.

Let's say we open an app and say, "What is the weather in Delhi today?" The following things happen:

• Our microphone captures our voice.
• Our voice travels over the internet to the agent's server.
• The agent understands the question and prepares a reply.
• The reply audio travels back over the internet to our device.
• Our speaker plays the reply.

The agent can be very smart, but if the audio is slow to travel in either direction, the conversation feels broken. A gap of even one or two seconds feels awkward in a live conversation. We have a detailed blog on how to build a latency budget for a voice AI agent that explains where every millisecond of this delay goes.

So, the network must give the voice agent three things:

• Low delay: audio must reach the other side as fast as possible.
• Steady flow: audio must keep flowing without long freezes.
• Both directions at once: the user can talk while the agent is talking, for example to interrupt it.

Both WebRTC and WebSocket can carry audio in both directions at the same time. The difference is in how they handle delay and problems on the network. To understand that, we must first see how audio travels over the internet.

How audio travels over the internet

Our voice is a continuous sound. The device records it and cuts it into small pieces, for example 20 milliseconds of sound in each piece. Each piece is put inside a packet and sent over the internet.

So, one second of speech becomes around 50 packets, one after another.

These packets share the internet with everyone else. On the way, the following problems happen:

• Some packets arrive late.
• Some packets arrive in the wrong order.
• Some packets get lost and never arrive.

There are two main rules, called protocols, for sending packets, and each one handles these problems differently.

TCP is the careful rule. It makes sure every packet arrives, and in the correct order. If a packet is lost, TCP asks for it again and holds back all the packets behind it until the lost one arrives.

UDP is the fast rule. It sends packets and does not wait. If a packet is lost, UDP does not resend it. The receiver gets whatever arrives.

Let's say someone is reading a long message to us over a loudspeaker. With TCP, if we miss one word, everything stops until that word is repeated, and only then the reading continues. With UDP, if we miss one word, the reader does not stop. They keep going, and we understand the meaning from the words around it.

For a file download, TCP is perfect, because we need every byte. For a live voice, it is different. A piece of audio that arrives one second late is useless, because the conversation has already moved on.

Now, we know the two rules. WebSocket is built on TCP, and WebRTC sends its audio over UDP. Let's understand each of them.

What is WebSocket?

WebSocket is a way to keep one connection open between a client, such as the user's app, and a server, so that both sides can send messages to each other at any time.

Normally, the web works in a request-and-response style. The client asks, the server answers, and that exchange ends.

WebSocket changes this. The client opens one connection, and it stays open. The connection starts as a normal web request, and then both sides agree to switch it to a WebSocket. Now, the server can push a message to the client whenever it wants, and the client can push a message to the server whenever it wants.

WebSocket runs on top of TCP. Means, every message we send over a WebSocket arrives completely, and in the same order in which we sent it.

WebSocket is simple to use. Almost every programming language and every browser supports it, and it talks directly to our server.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

How a voice agent uses WebSocket

The best way to learn this is by taking an example.

Consider there is a user with a browser and a voice agent on a server. The flow is like below:

+--------------------------+
| User's microphone        |
+--------------------------+
             ↓
+--------------------------+
| Cut audio into chunks    |
+--------------------------+
             ↓
+--------------------------+
| Send chunks on WebSocket |
+--------------------------+
             ↓
+--------------------------+
| Voice agent on server    |
+--------------------------+

The reply comes back over the same WebSocket in the opposite direction.

Here, our code does all the work by itself:

• Our code reads raw audio from the microphone.
• Our code cuts it into chunks, for example 20 to 100 milliseconds each.
• Our code sends each chunk as a WebSocket message, often with a small label that says "this is audio".
• The server sends back reply audio as messages, and our code plays them in order.

Let's see a simple sketch of the client side in JavaScript as below:

const socket = new WebSocket('wss://our-agent.example.com/voice')

// send every small audio chunk to the agent
function onAudioChunk(chunk) {
  socket.send(chunk)
}

// play every audio chunk that comes back
socket.onmessage = (event) => {
  playAudio(event.data)
}

Here, we have opened one WebSocket connection to our agent. The wss:// at the start means a secure WebSocket connection. Every time the microphone gives us a small chunk, we send it. Every time the agent sends a chunk back, we play it. The playAudio function is our own code, and it must handle the timing and smooth playback by itself.

This works well on a good network. The problem starts when the network is not good.

Now that we have learned about WebSocket, it's time to learn about WebRTC.

What is WebRTC?

WebRTC (Web Real-Time Communication) is an open standard for sending live audio, video, and data between devices with very low delay.

It was designed for audio and video calls in the browser. It is built into all major browsers, and there are libraries for mobile apps and servers.

WebRTC is not one single thing. It is a complete package for live media. It includes:

• A way to send audio and video over UDP, so that late packets do not block new ones.
• Encryption, so that the audio is always protected while it travels.
• A way to find a path between two devices, even when they sit behind home routers and office firewalls.
• Tools that measure the network and adjust the audio quality when the network gets worse.
• Audio cleanup in the browser, such as echo cancellation and noise suppression.

WebRTC also has a data channel, which can send text and other data next to the audio. A voice agent can use it to send events like "the user started speaking" or the text of the reply.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

How a voice agent uses WebRTC

Again consider there is a user with a browser and a voice agent on a server.

With WebRTC, the two sides first need to agree on how to talk. They exchange a short description of the call, such as which audio format to use and which network addresses to try. This step is called signaling. WebRTC does not define how signaling happens, so it is usually done with a normal HTTP request or a WebSocket.

After signaling, the audio flows as a live media stream, like below:

+------------------------------+
| User's microphone            |
+------------------------------+
               ↓
+------------------------------+
| Echo cancel + noise suppress |
+------------------------------+
               ↓
+------------------------------+
| Encode with Opus codec       |
+------------------------------+
               ↓
+------------------------------+
| Send over UDP, encrypted     |
+------------------------------+
               ↓
+------------------------------+
| Voice agent on server        |
+------------------------------+

Here, the browser does most of the work for us. We hand the microphone to WebRTC, and it cleans the audio, compresses it with the Opus codec, and sends it. A codec is a tool that shrinks audio so that it needs fewer bytes, and Opus works very well for speech with low delay.

On the receiving side, WebRTC uses a jitter buffer. Jitter means the uneven changes in how long each packet takes to arrive. The jitter buffer holds incoming audio for a very short time, so that packets that arrive a little late or out of order can be put back in the right place before playing. The size of this buffer grows and shrinks with the network.

So, with WebRTC, we do not write the timing and playback logic by ourselves. It comes built in.

Now, let's see the most important difference between the two.

What happens when a packet is lost

Let's say the user is speaking, and the device sends five audio packets: 1, 2, 3, 4, and 5. On the way, packet 2 gets lost.

With WebSocket over TCP:

• Packet 1 arrives and is handed to the agent.
• Packet 2 is lost. TCP notices and asks for it again.
• Packets 3, 4, and 5 arrive, but TCP holds them back, because they must be delivered in order.
• The resent packet 2 arrives later, after at least one more trip across the network.
• Only now, packets 2, 3, 4, and 5 are handed over together.

This waiting is called head-of-line blocking. One lost packet blocks every packet behind it. The audio freezes for a moment, and then a burst of old audio arrives at once. The voice agent hears the user late, and the user hears the agent's reply late.

Packet loss is rare on a stable connection. But on a weak mobile network, with many lost packets, these small freezes add up, and the audio can fall further and further behind.

With WebRTC over UDP:

• Packet 1 arrives and is handed to the agent.
• Packet 2 is lost. Nobody waits for it.
• Packets 3, 4, and 5 arrive and are handed over on time.
• To cover the gap of packet 2, WebRTC fills in a short guessed sound based on the audio just before it. This is called packet loss concealment.

A missing 20 milliseconds is so short that the listener usually does not notice it. The conversation stays on time.

This is how WebRTC keeps the delay low on a bad network. It accepts a tiny loss of quality in return for a steady, live flow of audio.

If we want to go deep into AI and ML System Design and how to Design a Real-Time Voice AI Agent, our AI and Machine Learning Program at Outcome School covers them from the ground up.

The extra work WebRTC does for voice

Apart from UDP, WebRTC gives a voice agent many features that we would have to build by ourselves with WebSocket.

Echo cancellation: When the agent speaks through the user's speaker, the microphone picks up that sound again. Without echo cancellation, the agent hears its own voice and thinks the user is interrupting. WebRTC in the browser removes this echo.

Noise suppression and gain control: WebRTC lowers background noise and keeps the voice at a steady loudness. This helps the speech recognition part of the agent.

Adapting to the network: WebRTC watches how many packets are lost and how long they take. When the network gets worse, it lowers the audio quality to keep the audio flowing, and raises it again when the network improves. This is called congestion control.

Recovering lost audio: Opus can add a little extra information to each packet about the previous one. If one packet is lost, the receiver can rebuild a rough version of it from the next packet. This is called forward error correction.

Finding a path: Most devices sit behind a home router or a firewall. WebRTC uses helper servers called STUN and TURN to find a working path. STUN helps a device learn its public address, which is the address that the rest of the internet sees, and TURN relays the audio when a direct path is not possible.

With WebSocket, most of these are not available. We can ask the browser's microphone for echo cancellation and noise suppression, but the timing, the buffering, the recovery from loss, and the adjustment to the network are all our own job.

But WebRTC has a cost too. It has many moving parts: signaling, STUN and TURN servers, encryption keys, and media servers. It is harder to set up and harder to debug than a WebSocket. And on the server side, we need a WebRTC library or a media server that understands all of this.

WebRTC vs WebSocket

Let me tabulate the differences between WebRTC and WebSocket for your better understanding so that you can decide which one to use based on your use case.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

When to use which one

Now, the question is: which one must we pick for our voice AI agent?

Use WebRTC when the audio comes from the user's device. A user on a phone or a laptop is on a network that we do not control. It can be weak Wi-Fi, a moving train, or a crowded mobile network. Here, WebRTC keeps the delay low, removes the echo, and handles packet loss. This is why realtime voice APIs commonly recommend WebRTC for browser and mobile clients.

Use WebSocket when the audio moves between servers. Let's say our backend receives the user's audio and passes it to a speech model in the cloud. Both servers sit in data centers with fast and stable networks. Packet loss is very rare, so head-of-line blocking rarely hurts, and the simplicity of WebSocket matters more. This is why realtime voice APIs commonly offer a WebSocket connection for server-to-server use.

Use WebSocket for phone calls coming through a telephony provider. When a user calls a phone number, the phone network carries the call to the telephony provider, which is a company that connects phone calls to the internet. The provider then often streams that call audio to our server over a WebSocket. Here, the phone network already carries the call from the user, and the last step is between two servers.

Use WebSocket for a quick prototype. When we are building and testing an idea, WebSocket lets us get a working voice agent quickly. Later, when real users start using it on real networks, we can move the user-facing part to WebRTC.

A common setup in production uses both: WebRTC between the user and our media server, and WebSocket between our server and the AI models behind it.

Now we must have understood how WebRTC and WebSocket differ for voice AI agents, why a lost packet affects them differently, and when to use which one.

Frequently Asked Questions

What is head-of-line blocking?

Head-of-line blocking happens when one lost packet blocks every packet behind it. WebSocket runs on TCP, and TCP delivers packets in order, so it holds back the later packets until the lost one is resent and arrives. For a voice agent, this means the audio freezes for a moment, and then a burst of old audio arrives at once.

How does WebRTC signaling work?

Signaling is the first step, where the two sides exchange a short description of the call, such as which audio format to use and which network addresses to try. WebRTC does not define how signaling happens. So, it is usually done with a normal HTTP request or a WebSocket, and the audio starts flowing after it.

Can a voice AI agent use both WebRTC and WebSocket together?

Yes. A common setup in production uses both. WebRTC carries the audio between the user and our media server, because the user's network is not under our control. WebSocket carries the audio between our server and the AI models behind it, because servers in data centers have fast and stable networks.

Why does a voice AI agent need echo cancellation?

A voice AI agent needs echo cancellation because the microphone picks up the agent's own voice from the user's speaker. Without it, the agent hears itself and thinks the user is interrupting. WebRTC in the browser removes this echo, while with WebSocket we can only ask the browser's microphone for echo cancellation.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.