Voice UX is not Chat UX

November 22, 2026·12 min read

aillmvoiceuxprompt-engineering

A few weeks ago I was at a Conversational AI peer group with a 5-6 people from different companies/industries (DB, Flix, Zalando etc) and this post is more-or-less triggered based on the discussions we had there (and the failures I see in production every day).

The majority of the things we know about and around building LLM based products come from chatbots, so its natural that everything we know about improving these products (evals, prompt optimiziation, UX etc) to come from chat based experiences, even our gut feeling of what is a good answer comes from chat. Its natural to try to take the same products and attach voice to it, but unfortunatelly many of the good parts start failing apart, and it's not obvious why until you actually listen.

Take for example this answer from an assistant that books appointments.

"I found three free slots: Tue 14 Nov, 09:15 · Wed 15 Nov, 10:30 · Thu 16 Nov, 16:45. Which one works for you?"

On a screen, three options are three buttons. One tap and it’s done.

In a chat window this answer is perfectly fine. The three slots are on the screen as three buttons, the user looks at them, taps one, done, nobody thinks twice about it. The same thing said by an AI voice assistant can be completely different, e.g. the assistant says first "Tue 14 Nov", when it reaches Thursday the user has forgotten what time was the Tuesday one. Some ask to hear it again, some say "the first one" and then they are not sure which one was the first, and a few just say "yes", and good luck figuring out what the yes is for.

A voice agent isn't a chatbot with a speaker attached. A lot of voice bugs start as a sentence that would be totally fine on a screen.

I found three free slots.Tue 14 Nov, 09:15Wed 15 Nov, 10:30Thu 16 Nov, 16:45Tue 14 Nov,09:15,Wed 15…10:30…
A voice agent isn’t a chatbot with a speaker attached.

What you lose when you lose the screen

ScrollbackButtonsTheir own wordsPatienceTuesday9:15,Wednesday10:30,Thursday16:45…
No scrollback, no buttons, no exact words, no patience for silence.

In chat the user can scroll back and read again what the assistant said, they can have the options as buttons, what they typed is exactly what the model gets, and while they wait there is an indicator (spinner, fake typing etc) that tells them that something is happening. On voice agents you don't have that, the user hears every sentence once, the options exist only for the few ms they are spoken, and the model never sees what the user said. You also have silince, which can mean various things, from hesitation to "I am working on it" to "it broke/doesn't work".

The failures can depend on the architecture you decide to pick (classic STT -> TTS or voice-2-voice) but no matter the architecture, voice is not chat. As of the time of writing this post voice-2-voice technology is impressive on demos but hard to stear the agents to do what you want on production, so for anything that takes irreversible actions (e.g. booking an appointment) I would still go with a classic setup although it may feel less natural.

The model reads a transcript, not the user

speech-to-textReid?Reed?Read?
The model never hears the caller. It reads a guess at what they said.

The input of a voice agent is noisy, and most of the time the agent has no idea which parts are the noise.

Chatbots are trained to show that they understood by echoing back what the user said, e.g. "So you'd like to move your appointment to next week?" On a call, if STT transcribed the words into nonsense, the TTS will repeat the nonsense back with full confidence, and for the user it sounds like nobody was listening. If part of what the user said is unclear, the agent should not quote it back or even worse saying it "in its own words". A simple clarifying question based on a confidence level usually helps.

Noisy input also matters for misclassification. Think of a yes/no question where "yes" does something hard to undo. The answer comes garbled, the model guesses, and sometimes it guesses towards the expensive side. Better accuracy helps of course, but you can also write the rule so the model prefers the cheaper mistake, and a garbled answer is never taken as the expensive one.

Names are the worst. "Reid", "Reed" and "Read" sound the same, and they can be three different people in the same customer list. Names are rare words, so STT guesses and the model guesses too. Asking for users to do spelling helps but doesn't solve the problem. Users spell with code words ("R as in Romeo, E, I as in India…"), they add filler in the middle, stop halfway, pause to breathe, and STT sometimes merges a few letters into one token. Asking the LLM to just "extract the spelled name" kinda works, but when it fails it's hard to see, e.g. one letter missing or a code word read as a letter.

Text that is fine to read and wrong to hear

Some strings are correct on a screen and broken out loud, some examples

  • LLMs learned language mostly from text, so they ask questions like a form does. "Please indicate your preferred method of payment." is correct English but nobody says it like that, people say "Cash or card?" and then the voice has to get the tone of a question right.
  • Shorthand like "Dr.", "approx.", "Mon-Fri" or "and/or" is normal in writing, but a TTS voice may read it literally, the slash too.
  • "09:15" should be "nine fifteen" or "quarter past nine", never "zero nine colon fifteen", and "14:30" should be "half past two", not "fourteen thirty".
  • A phone number that the model made up sounds exactly like a real one when it's read aloud, and the user has no way to check it.
  • The voice has to guess how to pronounce a surname, and when it guesses wrong it sounds like the agent never heard the name before.
  • Tool results and prompts are full of internal terms, "slot", "resource", "request status"... On the phone the model is the UI, and it will happily say "I found an available slot for the requested resource.", for many people these terms will sound weird.

Lists don't work by ear

WeberDuboisLopezBeckerMartinKellerRossi
By the fourth name, the caller has forgotten the first.

A screen can show twelve options at once, as buttons or in a dropdown, a voice has to read them out one by one.

On a screen all twelve options are there at once. The user scans them and picks one.

"We have Weber, Dubois, Lopez, Becker, Martin, Keller, Rossi, Novak…", somewhere around the fourth name the user has lost the first one, and now they have to ask again, or they say "the second one" and you hope they mean the same second one as the agent. Three options read out loud is more-or-less the limit, above that it is better if the agent asks an open question ("Who is the appointment with?") and lets the user say the name themselves, in their own words, whatever they remember of it. The old phone menus (press 1 for this, press 2 for that) went up to seven options, with voice there is room for even less.

Same thing with sentences, a chat answer can be skimmed and a spoken one can't, so the agent has to keep its turns short, one thing to say and one question to ask and then it stops, and this includes the politeness, because LLMs love to thank you for every single answer, "great question", "perfect", "happy to help", which in text you skip without even noticing, but spoken, every "thank you so much for that!" costs a second and pushes the actual question one second further away.

Silence is a failure mode

0.2 speople0.8 sagent target4 schat users waitHello? Are you still there?
People leave about 0.2 seconds between turns. Four seconds of nothing sounds like a dropped call.

In chat, "I'd like to come in on Thursday" followed by four seconds of "Assistant is typing…" is totally fine, on the phone it's four seconds of nothing and then "Hello? Are you still there?", because people leave something like 0.2 seconds between turns when they talk and most voice agents try to answer in under 0.8 seconds, while in chat people wait several seconds without complaining (think how long you wait for a reasoning model to answer), but on a call dead air sounds like the call dropped.

In chat, four seconds of “typing…” is fine. Something is clearly happening.

A promise with nothing behind it is even worse than a slow answer. A booking agent says "One moment please, let me check…" and then hands the turn back without calling the availability tool. The user waits in silence for something that will never happen. Often the model simply has no valid next step so it stalls.

When a search really takes a while, have the agent say so, then the user knows it's working and not gone. Small delays matter too, a backend that freezes for a second goes unnoticed in chat, but on a live call it's enough to break the conversation.

The latency budget also decides which model you can use. A chat product can afford a big reasoning model that thinks for a few seconds before it answers. On a call the whole turn (speech-to-text, the model, tool calls and text-to-speech) has to fit in more-or-less a second, so you end up with a smaller/faster model that is not as smart.

Taking turns is part of the interface

“Shall I book Tues—”“Tuesday… no, Wednesday”
Nobody presses Enter, and anyone can talk at any time.

A chat user presses Enter when they are done and the assistant can't be cut off mid-message, on a call there is no Enter and anyone can talk at any time. The first problem is to know when the user is done talking, the system guesses it from the audio and it can only guess, so if it guesses too early the agent jumps in exactly where people pause the most, like when they read a number from a card or spell a name, and if it guesses too late every single turn ends with this awkward gap.

"Yes yes, Tuesday, I know, Tuesday at ten", callers talk over the agent all the time, usually because they already know what it is going to ask, so the agent has to stop, but not for every cough, train announcement or voice in the background, and it also has to remember where exactly it stopped, otherwise it will happily refer to things the user never heard, because in chat you can assume the user read everything the model wrote and on a call you really can't.

"Sorry, what was that?" is the only scrollback a user has, and when they say it the agent should say it again but shorter, not repeat the same long sentence, and the same goes for when a user corrects themselves ("Tuesday… no, Wednesday"), both words reach the model and it has to take the last one, which is not as obvious to the model as it is to you.

In chat, the answer is still there. Scroll up and read it again.

And people take turns differently, ask "Your first name and date of birth?" and one user gives you both at once while another says "Laura" and waits, so if you split it in two questions the first user is stuck waiting for a question they already answered, and there is no design that is fastest for everyone, the agent has to accept both answers in one turn and only ask for what is missing.

Not every voice is talking to the agent

“Tuesday or Thursday?”for me?…call now…“Wednesday’s better!”“Thursday.”
The microphone hears everyone in the room. Not every voice is talking to the agent.

In chat every message is typed by the user and it is meant for the assistant, a phone microphone on the other hand hears the whole room, the user asking their partner "does Thursday work for you?", a kid in the background (if you have kids at home you know exactly what I mean), the TV, and speech-to-text writes all of it down as if the user said it to the agent.

So the agent asks "Tuesday or Thursday?", the user turns away to check, someone in the room says "Wednesday's better" and the agent books Wednesday. Nothing was misheard, the words were just not meant for the agent.

Confirming without a "Confirm" button

“Wednesday at ten. Shall I book it?”“mm-hmm…”“Yes.”booked
On the phone, the only Confirm button is the caller's yes.
In chat, Confirm is a button. Nobody has to guess what “mm-hmm” meant.

The only button a user has is their "yes", and voice design has known for a long time two ways to use it, implicit confirmation, where the agent says back what it understood and moves on ("Wednesday, then. Morning or afternoon?"), and explicit confirmation, where it stops and waits ("Wednesday at ten. Shall I book it?"), and if you do everything explicit the call feels like someone reading a form to you, while if you do everything implicit mistakes go through, so I'd go with implicit when the agent understood clearly and a mistake is cheap, and explicit for anything that commits someone to something, and if the agent is not sure what it heard it doesn't confirm anything, it asks again. A vague "mm-hmm" in the middle of another sentence is not a yes, no matter how much the model wants it to be.

Summary

So, moving an LLM product from chat to voice is a bit more than putting STT in front and TTS at the end, the caller has no screen and hears every sentence once, so whatever the agent says has to work the first time it's heard. And here we only covered the basics. Adjusting to how fast the caller talks, picking up the emotion in their voice (angry, in a rush, etc.), pitch, prosody, dialect, changing languages mid conversation, speech impairments and stutters, and so on are topics for another article.