How does function calling work in a voice AI agent?
- Authors
- Name
- Amit Shekhar
- Published on
Function calling in a voice AI agent is the way the agent takes a real action, like checking an order or booking a table, while we are talking to it. The LLM reads what we said and recommends the right function and its inputs, our code runs that function, and the agent speaks the result back to us.
In this blog, we will learn about how function calling works in a voice AI agent. We will also see how we describe our functions to the LLM, how the LLM picks a function from what the user said, how our code runs it and sends the result back, how the agent turns that result into a spoken reply, what changes when the input is a voice instead of typed text, and how it works with a speech-to-speech model.
We will cover the following:
- What is function calling?
- Where function calling fits in a voice AI agent
- Step 1: Describe the functions to the LLM
- Step 2: The user speaks
- Step 3: The LLM picks a function and its inputs
- Step 4: Our code runs the function
- Step 5: The result goes back to the LLM
- Step 6: The agent speaks the answer
- The complete flow in one diagram
- What changes when the input is a voice
- Function calling with a speech-to-speech model
- Putting it all together in code
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning at Outcome School.
Let's get started.
What is function calling?
An LLM (Large Language Model) produces text. On its own, it cannot look at our database, it cannot check the weather right now, and it cannot book anything. It only knows what it learned during training and what we put in front of it.
But, we want our agent to do real work. Function calling solves this problem.
Function calling is a feature where we tell the LLM about the functions that our code has, and the LLM replies with the name of the function to run and the inputs to pass to it, instead of a normal text reply.
The LLM never runs the function itself. It only recommends which function to run and with what inputs. Our code, which we call the agent, reads that recommendation, runs the real function, and sends the result back to the LLM.
Let's say we have a function get_order_status(order_id) in our code. The user asks, "Where is my order 4521?" The LLM does not answer from memory. It replies with something like below:
{ "name": "get_order_status", "arguments": { "order_id": "4521" } }
Our code sees this, calls get_order_status("4521"), gets the answer from the database, and gives it back to the LLM. Then, the LLM writes the final reply for the user.
Function calling is also called tool calling, and the functions are often called tools. Both names mean the same thing in this blog. We have a detailed blog on how function calling works in LLMs that explains this in depth.
Where function calling fits in a voice AI agent
A voice AI agent is an AI agent that we talk to with our voice, and it talks back with a voice. Many voice agents work as a pipeline of three steps:
- Speech-to-Text (STT): It turns what the user said into text.
- LLM: It reads that text and decides the reply.
- Text-to-Speech (TTS): It turns the reply text into a voice.
Function calling happens around the middle step. The LLM decides which function to call, and our code runs it. The STT and the TTS do not know anything about functions. They only move words in and out.
So, the LLM in a voice agent has two choices every time it replies:
- Reply with text, which goes straight to the TTS and is spoken.
- Reply with a function call, which goes to our code and is not spoken.
The best way to learn this is by taking an example. Let's say we are building a voice agent for a restaurant. It has two functions:
check_table(date, time, guests): checks if a table is free.book_table(date, time, guests, name): books the table.
Now, let's follow one function call from start to end.
Step 1: Describe the functions to the LLM
The LLM cannot use a function that it does not know about. So, before the phone call even starts, we write a description of every function.
Each description has three parts:
- Name: what the function is called.
- Description: in plain words, what it does and when to use it.
- Parameters: the inputs it needs, with their types, written in a format called JSON Schema.
We can write the description of check_table as below:
{
"type": "function",
"name": "check_table",
"description": "Check if a table is free at the restaurant for a given date, time, and number of guests. Use this before booking.",
"parameters": {
"type": "object",
"properties": {
"date": { "type": "string", "description": "Date in YYYY-MM-DD format" },
"time": { "type": "string", "description": "Time in 24-hour HH:MM format" },
"guests": { "type": "integer", "description": "Number of people" }
},
"required": ["date", "time", "guests"]
}
}
Here, we can see that the description tells the LLM when to use the function, and the parameters tell it exactly what to fill in. This text is all that the LLM knows about the function. So, a clear description leads to correct calls.
Similarly, we write a description for book_table. We send both descriptions to the LLM along with the system prompt, so that it is available on every turn of the conversation.
Step 2: The user speaks
Now, the user calls the restaurant and says:
"Do you have a table for four this Saturday at 8 in the evening?"
The voice agent listens and waits until the user has finished speaking. Then, the STT turns the audio into the same words as text.
This text is added to the conversation as the user's message, and the whole conversation, along with the function descriptions, goes to the LLM.
Step 3: The LLM picks a function and its inputs
The LLM reads the user's words and the function descriptions. The user is asking about a free table, and check_table is the function for that. So, instead of writing a spoken reply, the LLM returns a function call.
It also has to fill the inputs. The user said "this Saturday at 8 in the evening", but the function wants a date in YYYY-MM-DD format and a time in 24-hour format. Let's say today is Sunday, 2026-10-18, and we put this date in the system prompt. Then, the LLM can work out that this Saturday is 2026-10-24. And "8 in the evening" becomes 20:00.
The LLM replies with something like below:
{
"type": "function_call",
"call_id": "call_01",
"name": "check_table",
"arguments": "{ \"date\": \"2026-10-24\", \"time\": \"20:00\", \"guests\": 4 }"
}
Here, we have:
name: the function the LLM recommends.arguments: the inputs, written as a JSON string.call_id: a unique id for this call, so that we can match the result to it later.
This reply is not sent to the TTS. Nothing is spoken yet.
Planning your move into AI?
No matter which tech domain you work in today, with your effort, you can transition to these roles:
- AI Engineer
- Forward Deployed Engineer
- AI Agent Engineer
- Harness Engineer
- Voice AI Engineer
- AI Architect
- LLM Engineer
- LLM Inference Engineer
- Gen AI Engineer
- MLOps Engineer
- ML Engineer
- Deep Learning Engineer
Our AI and Machine Learning Program at Outcome School takes you there.
Step 4: Our code runs the function
Now, our code takes over. The real action happens in our code, not inside the LLM.
Our code does the following:
- Reads the function name,
check_table. - Parses the arguments from the JSON string.
- Checks that the arguments are valid, for example, that
guestsis a number. - Calls the real
check_tablefunction, which asks the restaurant's booking system.
Let's say the booking system replies that a table is free at 8 PM. Our function returns a result like below:
{ "available": true, "time": "20:00", "table": "T12" }
Here, available tells us that the table is free, and table is the id of that table.
Step 5: The result goes back to the LLM
The LLM still does not know the answer. So, our code adds the result to the conversation as a function result message, with the same call_id, as below:
{
"type": "function_call_output",
"call_id": "call_01",
"output": "{ \"available\": true, \"time\": \"20:00\", \"table\": \"T12\" }"
}
Here, the call_id tells the LLM that this result belongs to the check_table call it made in Step 3.
Then, our code asks the LLM to continue. The LLM now sees everything: the user's question, its own function call, and the result.
At this point, the LLM can make one more function call if it needs one. Or, it can write a normal text reply. This loop of call, run, and return can repeat as many times as the task needs.
Step 6: The agent speaks the answer
This time, the LLM writes a text reply like below:
Yes, we have a table for four this Saturday at 8 PM. Shall I book it for you?
This text goes to the TTS, and the user hears it. Many voice agents stream this, which means the TTS starts speaking the first few words while the LLM is still writing the rest.
The user says, "Yes, please, the name is Rahul." Steps 2 to 6 run again, this time with book_table. Our code books the table, the result goes back to the LLM, and the agent says, "Your table is booked, Rahul. See you on Saturday."
A quick note for you
No matter which tech domain you work in, get familiar with these topics:
- LLM
- RAG
- MCP
- Agent
- Fine-tuning
- Quantization
We put it all together in one video:
AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization
No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.
Now, let's get back to the topic.
The complete flow in one diagram
The whole journey of one function call is like below:
User speaks
"A table for four this Saturday at 8?"
↓
STT turns the audio into text
↓
LLM reads the text and the function descriptions
↓
LLM returns a function call
check_table(2026-10-24, 20:00, 4)
↓
Our code runs check_table
↓
Booking system returns { available: true }
↓
Our code sends the result back to the LLM
↓
LLM writes the reply text
↓
TTS turns the text into a voice
↓
User hears "Yes, we have a table..."
Here, we can see that the LLM is called twice for one question. The first time, it decides which function to call. The second time, it turns the result into a reply.
What changes when the input is a voice
Function calling in a voice agent uses the same idea as in a chat app. But a voice call adds a few problems that we must handle.
Problem 1: The STT can mishear the inputs.
In a chat, the user types "order 4521" and we get exactly that. On a call, the STT can write "for five two one" instead of "4521", or it can mishear one digit and write "4512". If the LLM passes the wrong value into the function, the function looks up the wrong order.
So, we tell the LLM in the system prompt to read back important values before using them: "Just to confirm, your order number is 4 5 2 1, right?" We also make our function check the format of the value and return a clear error if it is wrong, so that the LLM can ask the user again.
Problem 2: Some functions must not run without a clear yes.
Some functions change something in the real world, like booking a table, cancelling an order, or making a payment. If the agent acts on a misheard word, it can book or cancel the wrong thing.
For such functions, the agent first asks the user to confirm, and our code runs the function only after the user clearly agrees. This is why our restaurant agent asked, "Shall I book it for you?" before calling book_table.
Problem 3: The user hears silence while the function runs.
In a chat, the user sees a loading spinner. On a call, there is no spinner. If the booking system takes three seconds, the user hears three seconds of silence and can think that the call has dropped.
To solve this, the agent often speaks a short line, like "Let me check that for you," while the function runs. Many LLMs can return a short text and a function call in the same reply, so the agent can speak the text and run the function at the same time. We have a detailed blog on how a voice agent hides the delay of a slow tool call that covers this end to end.
Problem 4: The user can interrupt while the function runs.
The user can start talking again while the function is still running. For example, "Actually, make it 9 PM." The agent must decide what to do with the old call. If the function only reads data, like check_table, our code can drop the old result and run it again for 9 PM. If the function changes data, like book_table, our code must make sure that it does not create two bookings, one for 8 PM and one for 9 PM.
Problem 5: The result must be easy to say out loud.
A function can return a table, a long list, or a link. These look fine on a screen, but they are hard to follow when spoken. So, we tell the LLM to turn the result into short spoken sentences. For example, instead of reading out ten free time slots, the agent says, "We have tables at 7, 8, and 9 PM. Which one works for you?"
To learn Tool use in Agents, Agent Architecture, and how to Design a Real-Time Voice AI Agent, and to build an AI Coding Agent from scratch, check out our AI and Machine Learning Program at Outcome School.
Function calling with a speech-to-speech model
Some voice agents do not use the three-step pipeline. They use a speech-to-speech model, which takes the user's audio directly and produces the reply audio directly, without a separate STT and TTS.
Function calling still works in the same way here. The only difference is that the model listens to audio and speaks audio, instead of reading and writing text.
Let's take the OpenAI Realtime API as an example. The flow works like below:
- We send the function descriptions in the session settings at the start, with a
session.updateevent. - The user speaks, and the model hears the audio directly.
- When the model decides to call a function, it sends events with the function name and the arguments. The event
response.function_call_arguments.donetells us that the arguments are complete. - Our code runs the function.
- Our code sends the result back with a
conversation.item.createevent, as an item of typefunction_call_outputwith the samecall_id. - Our code then sends a
response.createevent to ask the model for a new response, and the model speaks the answer in its own voice.
Here, we can notice that the same six steps happen in this flow too.
Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.
Putting it all together in code
Now, let's see the code for the function calling part of our restaurant voice agent. For the sake of understanding, we will keep it short, and we will use helpers named stt, llm, and tts_speak for the STT, LLM, and TTS parts.
First, we write the real functions and a map from names to functions as below:
import json
def check_table(date, time, guests):
# ask the restaurant's booking system
return {"available": True, "time": time, "table": "T12"}
def book_table(date, time, guests, name):
# create the booking in the booking system
return {"booked": True, "booking_id": "B-7781"}
FUNCTIONS = {
"check_table": check_table,
"book_table": book_table,
}
Here, FUNCTIONS lets our code find the real function from the name that the LLM returns.
After that, we write the loop that runs every time the user finishes speaking:
def handle_user_turn(audio, conversation):
text = stt(audio)
conversation.append({"role": "user", "content": text})
while True:
reply = llm(conversation, tools=TOOL_DESCRIPTIONS)
conversation.extend(reply.items)
if not reply.function_calls:
tts_speak(reply.text)
return
for call in reply.function_calls:
args = json.loads(call.arguments)
result = FUNCTIONS[call.name](**args)
conversation.append({
"type": "function_call_output",
"call_id": call.call_id,
"output": json.dumps(result),
})
Here, we have:
stt(audio)turns the user's voice into text, and we add it to the conversation.llm(...)gets the whole conversation andTOOL_DESCRIPTIONS, which is the list of function descriptions from Step 1.conversation.extend(reply.items)saves the LLM's reply, including any function calls, so that the LLM can see its own calls in the next round.- If the reply has no function calls, it is a normal text reply. We speak it with
tts_speakand stop. - If the reply has function calls, we find each real function in
FUNCTIONS, run it with the arguments, and add the result with the samecall_id. - The
whileloop then calls the LLM again, so that it can read the results and either call another function or write the final reply.
Here, I have kept only the important lines of the code and removed things like error handling, confirmation before booking, and filler speech for brevity.
Now we must have understood how function calling works in a voice AI agent, from describing the functions, to running them in our code, to speaking the result back to the user.
Frequently Asked Questions
Does the LLM run the function itself in a voice AI agent?
No. The LLM never runs the function itself. It only recommends which function to run and with what inputs. Our code, which we call the agent, reads that recommendation, runs the real function, and sends the result back to the LLM, which then writes the reply that the TTS speaks.
Is function calling the same as tool calling?
Yes. Function calling is also called tool calling, and the functions are often called tools. Both names mean the same thing: we tell the LLM about the functions our code has, and the LLM replies with the name of the function to run and the inputs to pass to it.
Why is the LLM called twice for one question?
The LLM is called once to decide which function to call and with what inputs. After our code runs the function and sends the result back, the LLM is called again to turn that result into a reply. If it needs more data, it can make another function call before it writes the final reply.
Is the function call spoken to the user?
No. A function call goes to our code and is not sent to the TTS, so nothing is spoken at that point. Only a normal text reply goes to the TTS. Many LLMs can return a short text and a function call in the same reply, so the agent can say "Let me check that for you" while the function runs.
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
You can connect with me on:
Follow Outcome School on:
