How does a voice agent handle structured inputs like dates, numbers, and addresses?

Authors
How does a voice agent handle structured inputs like dates, numbers, and addresses?

A voice agent handles structured inputs like dates, numbers, and addresses by converting the spoken words into a fixed format, checking that the value is valid, and reading it back to the caller to confirm it before using it.

In this blog, we will learn about how a voice agent handles structured inputs like dates, numbers, and addresses. We will also see why these inputs are hard to capture over a call, how spoken words are turned into written values, how the LLM fills the right fields, how the agent checks and confirms each value, how it speaks these values back clearly, and the tricks that make dates, phone numbers, spellings, and addresses work reliably.

We will cover the following:

  • What is a structured input?
  • Why structured inputs are hard for a voice agent
  • The journey of a spoken value
  • Turning spoken words into written values
  • Filling the fields with the LLM
  • Checking the value
  • Confirming with the caller
  • Speaking structured values back
  • Handling dates
  • Handling numbers and phone numbers
  • Handling names, spellings, and addresses

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is a structured input?

A structured input is a piece of information that must follow a fixed format, so that a computer can use it directly.

A date, a phone number, an order ID, an amount of money, a zip code, and an address are all structured inputs. Each one has rules. A date has a day, a month, and a year. A phone number has a fixed number of digits. An order ID has an exact set of letters and digits.

Let's say a caller wants to book a doctor's appointment. The booking system does not understand "sometime next Tuesday afternoon". It needs a value like below:

{ "date": "2026-10-27", "time": "15:00" }

So, the voice agent must listen to the caller, who speaks freely, and produce this exact value. If even one digit is wrong, the appointment is booked on the wrong day.

Why structured inputs are hard for a voice agent

In a chat app, the user types "27/10/2026" and the text is already there. On a phone call, the agent only gets sound. Many things can go wrong between the sound and the final value.

  • Many ways to say the same thing: "October twenty seventh", "the twenty seventh of October", "next Tuesday", and "27th" can all mean the same date.
  • Sounds that are close to each other: "fifteen" and "fifty", "B" and "D", "M" and "N" sound very similar, and that too over a noisy phone line.
  • Phone audio is poor: a phone call carries less sound detail than a normal recording, so the speech recognition makes more mistakes.
  • No screen to look at: the caller cannot see what the agent heard. A wrong digit stays hidden unless the agent says it back.
  • People correct themselves: "My number is 98... sorry, 89..." The agent must keep the corrected value, not the first one.

One wrong word in a normal conversation is usually fine, because the meaning still comes through. One wrong digit in a phone number is not fine, because the whole value becomes useless. This is why structured inputs need special handling.

The journey of a spoken value

Before jumping into each step, we must see the full path that a spoken value takes inside a voice agent.

Caller speaks into the phone
        ↓
Speech-to-Text (STT)
        ↓
Spoken form: "my birthday is march fifth nineteen ninety"
        ↓
Inverse Text Normalization (ITN)
        ↓
Written form: "my birthday is March 5th 1990"
        ↓
LLM extracts the field
        ↓
{ "date_of_birth": "1990-03-05" }
        ↓
Our code checks the value
        ↓
Agent confirms: "That is March fifth, nineteen ninety. Is that right?"
        ↓
Caller says: "Yes"
        ↓
Value is saved

Here, we can see five steps: hearing, converting, extracting, checking, and confirming. We have a detailed blog on how to design a real-time voice AI agent that covers the full pipeline end to end.

Now, let's understand each step one by one.

Turning spoken words into written values

The first step is Speech-to-Text (STT), which turns the caller's voice into text.

When we speak, we never say digits or symbols. We say words. So, the raw output of speech recognition is often in spoken form like below:

five hundred dollars on october twenty seventh at three thirty pm

But our code needs the written form like below:

$500 on October 27 at 3:30 PM

The process of converting the spoken form into the written form is called Inverse Text Normalization (ITN).

For example, ITN turns "twenty seven" into "27", "three thirty pm" into "3:30 PM", and "at gmail dot com" into "@gmail.com".

It is called "inverse" because Text Normalization is the opposite direction. Text Normalization turns "$500" into "five hundred dollars" so that a voice can speak it. We will see that direction later in this blog.

Many speech recognition services provide ITN as an option, often called smart formatting.

Note: ITN also has to guess. "One twenty" can be "120" or "1:20". "Twenty twenty" can be the year "2020" or the two numbers "20 20". So, ITN gives us a better starting point, but it does not give us a final, trusted value.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Filling the fields with the LLM

Now, we have text. The next step is to pull the exact value out of that text and put it in the right place.

The process of picking out each required value from what the caller says is called slot filling. Each required value is a slot. For a booking, the slots are the date, the time, and the patient name.

Here comes the LLM (Large Language Model) into the picture. We describe the slots to the LLM as a tool. A tool is a function in our code, and the LLM can recommend calling it with the right input. We describe the tool with a clear format, like below:

{
  "name": "book_appointment",
  "parameters": {
    "date": { "type": "string", "description": "Date in YYYY-MM-DD format" },
    "time": { "type": "string", "description": "Time in 24-hour HH:MM format" },
    "patient_name": { "type": "string" }
  }
}

Here, we have told the LLM exactly what each slot looks like. The description field tells the LLM to write the date as "2026-10-27" and not as "Tuesday the twenty seventh".

When the caller says "Book me for Tuesday the twenty seventh at three in the afternoon", the slots that the LLM has filled so far look like below:

{ "date": "2026-10-27", "time": "15:00", "patient_name": null }

Here, we can notice that the patient name is missing. We instruct the LLM to leave a slot empty when the caller has not said it, instead of inventing a value. So, before recommending the tool, the agent asks the next question: "Sure. May I have the patient's name, please?"

The LLM also sees the full text of what the caller said. So, when the caller says "My number is 98... sorry, 89...", the LLM can keep only the corrected digits.

This is how the agent collects the slots one by one, in a natural conversation, until all the required slots are filled.

Note: The LLM never runs the booking itself. The LLM recommends the tool and its input, and our code runs it. This gives our code a chance to check the value first.

To learn AI Agents, Tool use in Agents, and Prompt Engineering, and to build an AI Coding Agent from scratch, check out our AI and Machine Learning Program at Outcome School.

Checking the value

The LLM can still make mistakes. It can write "2026-02-30", which is not a real date. It can drop a digit from a phone number. So, before we use any value, our code must check it.

We can write the check as below:

from datetime import date

def check_appointment_date(value):
    try:
        booking_date = date.fromisoformat(value)
    except ValueError:
        return "invalid_format"
    if booking_date < date.today():
        return "date_in_past"
    return "ok"

Here, we have done two checks:

  • date.fromisoformat checks that the value is a real date in the YYYY-MM-DD format. "2026-02-30" fails here.
  • The second check makes sure the date is not in the past, because we cannot book an appointment for yesterday.

If the check fails, our code sends the reason back to the LLM. The LLM then asks the caller again in a polite way: "Sorry, that date does not look right. Could you tell me the date again?"

Similarly, we can check other values:

  • A phone number must have the right number of digits for its country.
  • An order ID must match the pattern our system uses, for example two letters and six digits.
  • An amount must be a positive number within our allowed limit.
  • An address can be checked with an address lookup service to see that it exists.

This way, a wrong value is caught by our code before it reaches the booking system.

Confirming with the caller

A value can pass every check and still be wrong. "March fifth" and "March fifteenth" are both valid dates. Only the caller knows which one is correct.

So, the agent needs a confirmation step. The agent reads the value back and asks the caller to confirm it.

There are two common ways to confirm:

  • Explicit confirmation: the agent asks a direct yes or no question. "I have your phone number as nine eight seven six five, four three two one zero. Is that correct?"
  • Implicit confirmation: the agent repeats the value inside the next sentence and moves on. "Great, booking you for Tuesday, October twenty seventh at three PM. And what is the patient's name?" If the value is wrong, the caller can correct it right away.

For values where a mistake is costly, like a phone number, an amount of money, or a card number, we use explicit confirmation. For values where a mistake is cheap and easy to fix, we use implicit confirmation, so that the call does not feel slow.

When the caller says "No, it is fifteenth, not fifth", the agent updates only that one slot and confirms it again. If the caller only says "No", the agent asks for that one value again. It does not start the whole process from the beginning.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Speaking structured values back

Now, it's time to learn about the opposite direction. When the agent reads a value back, it uses Text-to-Speech (TTS), which turns text into voice.

If we give the TTS the text "Your appointment is on 10/27 at 3:30", it has to guess how to read "10/27". It can say "ten slash twenty seven" or "ten twenty seven". Neither one sounds natural.

This is where Text Normalization comes into the picture. It turns the written form into the spoken form before the voice speaks it. Most TTS engines try to do this on their own, but they can guess wrong for dates, IDs, and phone numbers. A few examples of the correct spoken form:

Written formSpoken form
10/27October twenty seventh
3:30 PMthree thirty P M
$1,250.50one thousand two hundred fifty dollars and fifty cents
9876543210nine eight seven six five, four three two one zero
221B Baker St.two twenty one B Baker Street

One common way to do this is to tell the LLM to write its reply in spoken form already. We can add an instruction in the system prompt like below:

When you say dates, times, amounts, or phone numbers,
write them as words, the way a person would say them aloud.
Read phone numbers and IDs one digit at a time, in small groups.

Here, we are asking the LLM to write "October twenty seventh" instead of "10/27" in its reply. Now, the TTS has much less to guess. Reading long numbers in small groups also makes it easier for the caller to check the number in their head.

Many TTS engines also support SSML (Speech Synthesis Markup Language), which lets us tell the voice how to read a value, like below:

<speak>
  Your order ID is <say-as interpret-as="characters">AB12</say-as>.
</speak>

Here, the say-as tag with interpret-as="characters" tells the voice to read "A B one two" letter by letter, instead of trying to read "AB12" as one word. Support for these tags is different for each TTS engine, so we must check what our engine supports.

Handling dates

Relative dates: callers often say "tomorrow", "next Friday", or "the day after". The LLM cannot know what "tomorrow" means unless it knows today's date. So, we must always put the current date, the day of the week, and the caller's time zone in the system prompt, like below:

Today is Sunday, October 18, 2026. The caller's time zone is Asia/Kolkata.

Without this line, the LLM has no reliable way to know today's date, and "next Friday" can turn into the wrong date.

Unclear dates: "next Friday" can mean the coming Friday or the Friday of the week after. Instead of guessing, the agent must say the exact date back: "That is Friday, October 23rd. Is that right?"

Different formats: "05/03" is the 3rd of May in some countries and the 5th of March in others. This is one more reason to always store dates in one fixed format like YYYY-MM-DD and to read the month back as a word, never as a number.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Handling numbers and phone numbers

Long numbers are where many mistakes happen. A phone number, a card number, or an account number has many digits, and missing one digit breaks the whole value.

The following tricks make it reliable:

  • Ask in groups: the agent can ask for the number in parts. "Please tell me the first five digits." Then, "And the last five digits." Shorter parts mean fewer mistakes.
  • Handle the way people speak: people say "double seven" for "77", "triple zero" for "000", and "oh" for "0". The LLM can turn these into digits, and we can mention this in the system prompt.
  • Count the digits: our code checks the digit count. If a 10-digit number comes with 9 digits, the agent asks for the number again.
  • Use the keypad: for sensitive numbers, like a card number or a PIN, the agent can say "Please enter your card number using the keypad." The keypad sends exact digits as tones, called DTMF tones, so speech recognition is not needed for it.
  • Use the caller ID: if the caller is calling from the same phone number we need, the agent can ask "Can we use the number you are calling from?" Then there is nothing to recognize.

Similarly, for amounts, "fifteen" and "fifty" sound very close. So, the agent confirms the amount and also says its digits: "That is fifteen dollars, one five. Correct?"

Handling names, spellings, and addresses

Names, email addresses, and street names are often the hardest, because many of them are not normal dictionary words. The speech recognition can turn the name "Kavya" into "cave ya" or "Gavya".

The following tricks help here:

  • Ask the caller to spell: "Could you spell your last name for me, please?" Then the LLM joins the letters into one word.
  • Use a spelling alphabet: letters like "B", "D", "E", "P", and "T" sound almost the same over the phone. So, many agents accept or offer words like "B as in Bravo" and "D as in Delta". The LLM turns "S as in Sierra, H as in Hotel" into "SH".
  • Give hints to speech recognition: many STT services let us pass a list of expected words, often called keyword boosting or custom vocabulary. If we pass the street names of a city or the product names of our company, the STT is more likely to hear them correctly.
  • Collect addresses in parts: instead of asking for the full address in one go, the agent asks for the zip code first, then the street and house number. The zip code usually tells us the city and the state, so the caller does not have to say them.
  • Look up the address: our code can send the collected address to an address lookup service. If the service finds a close match, the agent confirms it: "I found 221B Baker Street, London. Is that your address?"

Let's say a caller is giving an email address. The agent hears "kavya dot sharma at gmail dot com". The LLM turns it into kavya.sharma@gmail.com, our code checks that it has a valid email format, and then the agent reads it back letter by letter for the part before the "@": "K A V Y A dot S H A R M A at gmail dot com. Is that correct?"

Now we must have understood how a voice agent handles structured inputs like dates, numbers, and addresses, from the spoken words all the way to a value that our system can trust.

Frequently Asked Questions

What is the difference between Inverse Text Normalization and Text Normalization?

Inverse Text Normalization turns the spoken form into the written form, like "twenty seven" into "27". Text Normalization goes the other way and turns the written form into the spoken form, like "$500" into "five hundred dollars", so that a voice can speak it. A voice agent uses the first after Speech-to-Text and the second before Text-to-Speech.

Should a voice agent confirm every value the caller gives?

No. For values where a mistake is costly, like a phone number, an amount of money, or a card number, the agent uses explicit confirmation with a direct yes or no question. For values where a mistake is cheap and easy to fix, it uses implicit confirmation, so that the call does not feel slow.

How does a voice agent understand relative dates like "tomorrow" or "next Friday"?

The LLM cannot know what "tomorrow" means unless it knows today's date. So, we put the current date, the day of the week, and the caller's time zone in the system prompt. For unclear dates like "next Friday", the agent says the exact date back and asks the caller to confirm it.

Why should a voice agent ask for a card number or a PIN on the keypad?

The keypad sends exact digits as DTMF tones, so speech recognition is not needed for them. This avoids mistakes like hearing "fifteen" as "fifty" or dropping a digit. For sensitive numbers, like a card number or a PIN, the agent can ask the caller to enter them using the keypad.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.