How do you write a system prompt for a voice agent, not a chat bot?

Authors
How do you write a system prompt for a voice agent, not a chat bot?

A system prompt for a voice agent is a set of instructions that tells the LLM its replies will be spoken out loud, so it asks for short, plain, spoken answers that sound natural to the ear, unlike a chat bot prompt that is written for answers people read on a screen.

In this blog, we will learn about how to write a system prompt for a voice agent, not a chat bot. We will also see what a system prompt is, why text written for a screen fails on a call, how to keep replies short and free of formatting, how to make the agent speak numbers and names correctly, how to handle mishearing, tool waits, and the end of the call, and how to put all of it together into one complete prompt.

We will cover the following:

  • What is a system prompt?
  • How is a voice agent different from a chat bot?
  • Why does a chat bot prompt fail on a voice call?
  • Rule 1: Keep every reply short
  • Rule 2: No formatting, only plain spoken text
  • Rule 3: Write numbers, dates, and symbols the way we speak them
  • Rule 4: Ask one question at a time
  • Rule 5: Plan for mishearing
  • Rule 6: Confirm the important details
  • Rule 7: Fill the silence while a tool runs
  • Rule 8: Know how to end the call
  • Putting it all together: a complete voice agent system prompt

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is a system prompt?

A system prompt is the set of instructions that we give to an LLM (Large Language Model) before the conversation starts. It tells the model who it is, what its job is, and how it must behave.

The user never sees the system prompt. But the model gets it along with the conversation every time it writes a reply.

Let's say we are hiring a new person for the front desk of a clinic. On the first day, we give them a page of instructions: "You work at City Clinic. Greet every caller politely. Help them book appointments. Never give medical advice." The system prompt is that page of instructions for the model. We have a detailed blog on Prompt Engineering that explains how the system prompt and the user prompt work together.

How is a voice agent different from a chat bot?

A chat bot talks with us through text. We type a message, it types back. We read its answer on a screen.

A voice agent talks with us through sound. We speak, it listens, and it speaks back. There is no screen. Very often, it is answering a phone call.

Inside a typical voice agent, three parts work together like below:

+-------------------------------+
|        Caller speaks          |
+-------------------------------+
                ↓
+-------------------------------+
|  Speech-to-Text (STT)         |
|  turns voice into text        |
+-------------------------------+
                ↓
+-------------------------------+
|  LLM + System Prompt          |
|  decides what to say          |
+-------------------------------+
                ↓
+-------------------------------+
|  Text-to-Speech (TTS)         |
|  turns text into voice        |
+-------------------------------+
                ↓
+-------------------------------+
|        Caller hears           |
+-------------------------------+

Here, we can see that the LLM sits in the middle. It still reads and writes text. But a person does not read its text. It is read aloud by the Text-to-Speech engine. Some newer voice models handle speech directly without separate parts, but the same rule holds: whatever the model produces, the caller hears it instead of reading it.

Means, in a voice agent, the caller hears the output of the LLM instead of reading it. This difference changes how we write the system prompt.

To learn Prompt Engineering, Context Engineering, and how to Design a Real-Time Voice AI Agent in depth, check out our AI and Machine Learning Program at Outcome School.

Why does a chat bot prompt fail on a voice call?

The best way to learn this is by taking an example.

Suppose we take a normal chat bot prompt like below:

You are a helpful assistant for City Clinic.
Answer the user's questions about appointments.

Now, a caller asks: "What are your timings?"

The LLM writes a reply that looks great on a screen:

**Our Clinic Timings:**
- Mon-Fri: 9:00 AM - 6:00 PM
- Sat: 10:00 AM - 2:00 PM
- Sun: Closed

For more details, visit https://cityclinic.com/timings 😊

Now, imagine the Text-to-Speech engine reading this out loud. Some engines read the symbols out, like "asterisk asterisk Our Clinic Timings". Some skip them. The web address gets read piece by piece, and the emoji is either skipped or read by its name, like "smiling face". Different engines handle these symbols differently, but none of them sound like a human receptionist.

The following issues arise:

  • The bold marks, bullets, and emoji make no sense in sound.
  • "Mon-Fri" and "9:00 AM - 6:00 PM" can be read in strange ways.
  • A web link is useless on a phone call. The caller cannot click it.
  • The reply has too many pieces to hold in memory after hearing it once.

The LLM did what it usually does for text. We never told it that it is speaking. So, the system prompt must tell it.

Now, let's learn the rules that fix these problems, one by one.

Rule 1: Keep every reply short

When we read, we can skim, jump back, and read again. When we listen, we hear each word only once. If the agent speaks for 40 seconds, the caller often forgets the first half.

Also, a phone call is a back-and-forth conversation. A long reply feels like a speech, and the caller has to wait for their turn.

So, we add an instruction like below:

Keep every reply to one or two short sentences.
Give the most important information first.
If the caller wants more detail, they will ask.

Here, we are telling the model to speak like a person on the phone. The most important part comes first, so even a caller who stops listening early gets the answer.

Rule 2: No formatting, only plain spoken text

A screen can show headings, bold text, bullet points, tables, links, code, and emoji. A voice cannot show any of these. LLMs usually write this formatting using markdown, which is a set of symbols like ** for bold and - for bullets.

So, we forbid them like below:

Your replies will be converted to speech.
Never use markdown, bullet points, numbered lists,
headings, bold text, emojis, or links.
Write only plain sentences that sound natural when spoken aloud.

Here, the first line gives the reason behind the rule. When the model knows the reason, it often follows the rule better, even in situations that our rule did not list.

But what about a list, like the clinic timings? We tell the model to turn it into a spoken sentence like below:

If you need to share several items, say them in one
natural sentence using words like "and" and "then".

Now, the earlier timings reply becomes:

"We are open from nine in the morning to six in the evening on weekdays, from ten to two on Saturday, and we are closed on Sunday."

A caller can understand this in one hearing.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Rule 3: Write numbers, dates, and symbols the way we speak them

On a screen, "$1,250.50" is clear. But when spoken, a Text-to-Speech engine can read it in odd ways. The same is true for dates like "12/04", phone numbers like "9876543210", and short forms like "Dr." or "St.".

The safest way is to ask the LLM to write these in words, the way a human would say them. Let's see the instructions as below:

Write all numbers, prices, dates, and times in words,
the way a person would say them out loud.
- Say "one thousand two hundred fifty dollars and fifty cents",
  not "$1,250.50".
- Say "the fourth of December", not "12/04".
- Say "three thirty in the afternoon", not "15:30".
- Read phone numbers and booking codes one digit at a time,
  in small groups, like "nine eight seven, six five four".
- Never use short forms like "Dr." or "St.". Say "Doctor" and "Street".

Here, we can notice that phone numbers are a special case. "9876543210" read as one big number is "nine billion eight hundred seventy six million...", which is useless. When it is read digit by digit in small groups, a caller can write it down.

Note: Some Text-to-Speech engines already convert many numbers to words on their own. But the conversion is not always what we want, so writing them in words in the LLM output gives us control.

Rule 4: Ask one question at a time

In a chat, a chat bot can ask:

"Please share your full name, phone number, and preferred appointment day."

The user reads it and types all three. But on a call, the caller hears three questions, answers the first one, and forgets the rest.

So, we add:

Ask only one question at a time.
Wait for the answer before asking the next question.

Now, the agent asks "May I have your full name?", waits, then asks "And your phone number?", and so on.

Rule 5: Plan for mishearing

In a chat bot, the text the user typed is exactly what the LLM gets. In a voice agent, the LLM gets what the Speech-to-Text engine thinks the caller said.

Phone lines are noisy. People have different accents. Names are hard to recognize. So, the text can be wrong. For example, the caller says "I want to book for Friday," but the transcript, which is the text that the Speech-to-Text engine produces, says "I want to look for Friday."

The LLM must know this. Otherwise, it often treats every transcript as perfect and acts on a wrong word with full confidence.

So, we add instructions like below:

The caller's words come from a speech recognition system
and may contain mistakes.
If something sounds odd or does not make sense,
guess the most likely meaning from the context.
If you still cannot understand, politely ask the caller to repeat.
Never mention "transcript" or "speech recognition" to the caller.
Say something natural like "Sorry, I did not catch that."

Here, we are telling the model that the input is not always correct, and how to ask for a repeat in a human way.

In our example, the agent works for a clinic that books appointments. So, when the transcript says "I want to look for Friday", the model can guess that the caller means "book for Friday".

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Rule 6: Confirm the important details

Because mishearing happens, some details are too important to trust after hearing them once. A wrong spelling of a name, a wrong date, or a wrong phone number can ruin a booking.

So, we tell the agent to repeat these back like below:

Before booking, read back the caller's name, phone number,
date, and time in one sentence, and ask if it is correct.
For names that are hard to hear, ask the caller to spell them.

Now, the agent says: "So, that is Rahul Verma, phone number nine eight seven, six five four, three two one zero, on Friday the fourth of December at three thirty in the afternoon. Is that correct?"

If the caller says "No, it is Varma, V A R M A," the agent fixes it before anything goes into the system.

But we must not confirm everything. If the agent repeats every small answer, the call becomes slow. We confirm only the details that matter.

Rule 7: Fill the silence while a tool runs

Many voice agents use tools, like checking the calendar for free slots or looking up an order. Here, the LLM recommends which tool to use and with what input, and our code runs that tool and gives the result back to the LLM.

This takes time, often a second or more. In a chat, the user sees a "typing" indicator. On a call, the caller hears only silence, and silence on a phone call can feel like the line has dropped.

So, we add:

Before using a tool that may take time, first say a short
phrase like "Let me check that for you" or "One moment please".
When the result comes back, share it in one short spoken sentence.

Now, the caller hears the agent speak while the tool runs, and then the answer comes. We have a detailed blog on how a voice agent hides the delay of a slow tool call that covers this end to end.

The second line is also important. A tool often returns data in a format meant for code, with field names and long lists. The agent must turn it into a sentence like "We have a slot at four in the afternoon", not read out the raw data.

If we want to go deep into Tool use in Agents, and to build an AI Coding Agent from scratch, we have it all in our AI and Machine Learning Program at Outcome School.

Rule 8: Know how to end the call

A chat stops when the user closes the window. A phone call must be ended on purpose. If the agent does not know when to stop, it can keep asking "Is there anything else?" again and again.

So, we tell it like below:

When the caller's request is done, ask once if they need
anything else. If they say no, thank them, say goodbye,
and end the call.
If the caller asks for a human, or you cannot help them,
tell them you are transferring them and transfer the call.

Here, ending the call and transferring the call are usually tools that our code provides. The system prompt tells the model when to recommend them.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Putting it all together: a complete voice agent system prompt

Now, let's combine all the rules into one complete system prompt for our clinic. We will organize it in clear sections.

# Role
You are Asha, the AI phone receptionist for City Clinic.
If a caller asks, say honestly that you are an AI assistant.
You speak with callers on a live phone call.
Your job is to help callers book, change, or cancel appointments.

# Clinic details
Open Monday to Friday, nine in the morning to six in the evening.
Open Saturday, ten in the morning to two in the afternoon.
Closed on Sunday.

# How you speak
Your replies will be converted to speech.
Sound warm, calm, and friendly, like a real receptionist.
Keep every reply to one or two short sentences.
Give the most important information first.
Never use markdown, lists, headings, bold text, emojis, or links.
If you need to share several items, say them in one natural sentence.

# Numbers and names
Write all numbers, prices, dates, and times in words.
Read phone numbers and booking codes one digit at a time, in small groups.
Never use short forms. Say "Doctor", not "Dr.".

# Conversation flow
Ask only one question at a time and wait for the answer.
To book, collect the caller's full name, phone number, and preferred day.
Before booking, read back the name, phone number, day, and time,
and ask if it is correct.
Ask the caller to spell names that are hard to hear.

# Mishearing
The caller's words come from speech recognition and may contain mistakes.
Guess the likely meaning from context.
If you still cannot understand, say "Sorry, I did not catch that"
and ask them to repeat.

# Tools
Before checking the calendar, say "Let me check that for you".
Share the result in one short spoken sentence.

# Limits
Never give medical advice. If asked, say that a doctor will
help them during the visit.
If the caller asks for a human, transfer the call.

# Ending
When the request is done, ask once if they need anything else.
If not, thank them, say goodbye, and end the call.

Note: The section headings with # are inside the system prompt only. They help the model read the instructions. The model does not speak them, because we have told it to reply in plain sentences.

Now, when the caller asks "What are your timings?", the agent says:

"We are open from nine in the morning to six in the evening on weekdays, from ten to two on Saturday, and we are closed on Sunday. Would you like to book a visit?"

Finally, the best way to improve this prompt is to call the agent ourselves and listen. Many problems are easy to miss on a screen and obvious when we hear them. When something sounds wrong, we add or fix one rule and call again.

Now we must have understood how to write a system prompt for a voice agent, not a chat bot, and why every rule in it comes from one fact: the caller hears the answer instead of reading it.

Frequently Asked Questions

Can we use the same system prompt for a chat bot and a voice agent?

No. A chat bot prompt is written for answers people read on a screen, so the LLM replies with bold text, bullets, links, and emoji. In a voice agent, the Text-to-Speech engine reads the reply aloud, and none of this makes sense in sound. The voice agent prompt must tell the model that its replies will be spoken.

Does the Text-to-Speech engine not convert numbers to words on its own?

Some Text-to-Speech engines do convert many numbers to words on their own. But the conversion is not always what we want. A phone number read as one big number is useless to the caller. So, asking the LLM to write numbers, prices, dates, and times in words gives us control over how they are spoken.

Should a voice agent confirm every answer the caller gives?

No. The agent must confirm only the details that matter, like the caller's name, phone number, date, and time, because a mistake in them can ruin a booking. If the agent repeats every small answer, the call becomes slow. So, we confirm the important details in one sentence and move on.

Why should a rule in the system prompt include its reason?

When the model knows the reason behind a rule, it often follows the rule better, even in situations that the rule did not list. For example, the line "Your replies will be converted to speech" tells the model why it must avoid formatting, so it can apply the same idea to cases we did not write down.

How do we test and improve a voice agent system prompt?

The best way is to call the agent ourselves and listen. Many problems are easy to miss on a screen and obvious when we hear them. When something sounds wrong, we add or fix one rule and call again, and we keep repeating this until the agent sounds right.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.