Designing an AI voice agent that holds the conversation

An AI-based voice agent listens, understands a request made in natural language, decides what to do with it and answers out loud, in real time, in front of someone who can neither reread nor go back. This page is for the teams who design one or hand it to a provider.

It helps you decide what the agent says, in which order, with what sentence length, and what it does when it does not know. A voice project rarely fails on the model: it fails on scripts written like a web page, an out-of-date knowledge base and three seconds of silence nobody had planned for.

A voice agent is not a synthetic voice

The first decision in a voice project is a matter of vocabulary: three very different objects circulate under the same words. If your question is wider (which support requests to automate, how to measure resolution), it belongs to the AI customer service agent.

Synthetic voice, callbot, real-time agent: three objects not to confuse

Each of these notions designates a distinct scope of work:

• Synthetic voice / voice generator: producing audio (TTS) from a text, without understanding or holding a dialogue. It is a building block, not a system.
• Voicebot / callbot: call automation on marked-out paths, with a finite number of planned routes.
• Real-time voice agent: a complete pipeline (listening → understanding → generation → delivery) with turn-taking, latency and stability management.

Only the third belongs to AI agents, whose seven families and four permission levels are described in our reference guide. A fine “AI voice” is therefore not enough: performance is won on understanding, context, execution and the ability to recover.

From the tender stage onwards, remember that an impressive demo on audio delivery alone says nothing about the third turn.

The habits of writing that speech rules out

In writing, the reader rereads, compares two options, goes back. In speech, the information passes once, in the order it is spoken, and is lost if it was not retained. Written conversation is covered by the AI conversational agent; real time, for its part, imposes three constraints before the first line:

• One memorable piece of information per turn. Three figures said one after the other come back as “sorry, could you repeat that?”.
• No bullet lists, no tables. Beyond three items, an enumeration becomes a closed question.
• No going back. Every correction must be possible by voice: “no, that is not it” is a path to plan for.

In return, speaking is often faster than filling in a form, especially on mobile or while multitasking. A clumsy design destroys that gain first.

The pipeline: listen, understand, decide, speak

A voice agent assembles four building blocks, and each one can bring down the whole. Understanding rests on a technology that is common without being universal: natural language processing (NLP) shows an enterprise adoption rate of 33% (Hostinger, 2026), a benchmark taken from our set of AI statistics. The block can be bought; the work remains what you give it to understand and to do.

The four blocks, and what breaks in each

Each layer has failures of its own:

• 1. ASR / speech-to-text: turning voice into text. What breaks: accents, noise, overlapping speech, end of sentence detected wrongly.
• 2. Understanding: detecting the intent and extracting the entities (case number, date, product). What breaks: an unmapped intent, a badly extracted entity that contaminates the call.
• 3. Orchestration: applying the rules, calling the tools, handling confirmations and escalations. What breaks: a business tool that does not respond, and no line written to say so.
• 4. Generation + TTS: producing the answer and making it intelligible aloud. What breaks: an answer that is correct but too long, a proper name mispronounced.

This breakdown is a diagnostic tool: a TTS latency and an understanding error are fixed by different people. As long as a team says “the agent answers badly” without naming the layer, it is changing settings at random: require every anomaly to be tied to a block before it is worked on.

This separation of layers also makes a fault locatable: each layer can be measured and replaced without touching the others. Installing the agent on a line is a separate project: connection to the phone system, call routing and transfer with context belong to the AI phone agent.

Answer or act: the right to execute without asking

Two needs coexist in the same call: retrieving reliable information, and carrying out an operation in a business tool. The action itself is handled differently depending on whether it can be undone:

These three tiers match levels 1 to 3 of the four permission levels in the guide cited above. “Reversible” is the key word: a draft ticket can be undone, a confirmation sent to the customer cannot.

In speech, confirming means repeating what you are about to do and waiting for a clear agreement: you never take a silence for a yes. On an ambiguous request, clarifying or escalating is worth more than improvising.

Latency: where it is manufactured, what can be taken out of it

Perceived performance comes down to the delay before the first answer and the ability to hold the exchange without a break. Most of the delay felt is manufactured somewhere other than the model: know where the time is consumed, block by block, before touching anything.

The five sources of latency

The delay is spread across five sources, rarely just one:

• ASR: end of sentence detected too late, noise, hesitations from the caller.
• Generation: compute time, prompts that are too long, access to bulky documents.
• TTS: audio synthesis and buffering before playback.
• Network: round trips between the services called during the turn.
• Orchestration: calls to business tools, timeouts and retries.

Writing fixes two of them without any technology: a prompt loaded with three pages of context lengthens every generation, and a two-sentence answer is synthesized faster than a paragraph. Before negotiating response times with a vendor, reread what your scripts ask the system to produce.

Four levers, and the limit they do not cross

The effective levers come from real-time production:

• 1. Streaming: start speaking without waiting for the complete answer.
• 2. Splitting: answer in two stages (“let me check…” then the result) rather than in one long monologue.
• 3. Cache: serve stable answers immediately (opening hours, address, status).
• 4. Pre-warming: prepare contexts and connections before periods of heavy traffic.

These levers share one limit: the user accepts a “let me check” if they perceive progress, not mechanical repetition. An agent that says “one moment” three times seems stuck, even if it answers faster. The writing rule that follows:

• plan several waiting phrases;
• make them progress (“I am looking at your file”, then “I have your order, I am checking the date”);
• set a duration beyond which the agent announces a real delay or hands over rather than filling time.

Writing for speech: scripts, confirmations, silences

This is where a voice project is won or lost, and it is the most underestimated part, because it looks like copywriting without being it. One rule governs the rest: one idea per sentence, and one objective per turn. A long message gets the user cutting in and degrades both the ASR and the context.

A script is judged aloud, at normal pace, stopwatch in hand: fifteen seconds maximum without the caller speaking again. Beyond that, it is too long. This ceiling serves as the reference in the language guidelines and the risk grid.

Confirming, rephrasing, making people wait, saying no

A successful turn rests on four moves, and each one is written:

• Confirm the critical entities. A name, a date, a reference or an amount is read back before use, one entity at a time. A date is said in full words, an identifier is broken into groups.
• Rephrase before deciding. “If I understand correctly…” costs two seconds and avoids a call run on a false intent; it is placed at the switch towards an action.
• Handle silence according to its length. A short silence is respected; a medium silence calls for a reworded prompt, never the same sentence; a long silence ends with an announced exit.
• Say no explicitly. A short sentence refuses, says why and offers the next step; otherwise the caller rephrases their request indefinitely.

A list of more than three items, a web address, a long reference or a text of terms and conditions is never read aloud: the agent announces that it is sending it another way and moves on.

Mapping the intents before writing a single line

Voice penalizes grey areas heavily: 5 very well mastered paths are better than 50 approximate ones. The mapping that precedes the writing rests on three inventories:

• the main intents, the ten to twenty reasons that cover most requests;
• the exceptions: urgency, unidentified caller, missing information;
• the escalation paths, with their trigger rule and the summary passed on.

The priority scenarios (reason and collection of two to five pieces of information, follow-up on a request, recurring questions, appointments, factual confirmation) each become a script, not a chapter: a few turns, a planned exit, a failure path.

Write from real transcribed calls: authentic wording is shorter and more elliptical than anything a team produces in a meeting room.

Knowledge and tone: what the agent is allowed to say

A language model answers probabilistically and cannot sort the true from the out of date: contradictory or obsolete content produces distorted answers. Speech makes the bill worse: a wrong spoken answer costs more than a web page to correct, because it has been heard, leaves no trace and commits the company on the spot.

Building the base: sources, short units, timestamping, control

Knowledge is therefore held as a living reference base: owners, dates, versions, exceptions. Proceed as you would with a quality system:

• 1. Identify the sources (procedures, terms, frequently asked questions, internal documentation) and their business owner, by name.
• 2. Structure into short units: questions and answers, rules, decision tables. A unit must be sayable in two sentences.
• 3. Timestamp and version, above all the “time-sensitive data”: offers, laws, procedures. They are the ones that go stale without warning.
• 4. Control through conversation tests and regular sampling, not through an annual review.

That is where the bottleneck of a voice project lies. A base three months out of date produces wrong answers with the same assurance as right ones. It is a continuous load: it is planned, it is assigned, and it does not disappear when a tool is bought.

The language guidelines, in a few applicable lines

Defining the brand personality means setting rules a writer applies without consulting you and a reviewer checks line by line. A handful is enough, provided they are settled:

The same guidelines say what the agent is allowed to ask for: what it may request, what it never asks for out loud, and when it hands over rather than collecting a sensitive piece of data.

Across sites or countries, keep a common “core” (values, structure of the answers) and localize opening hours, legal constraints and terminology. In speech, an inconsistency in tone is heard before it is understood.

Judging a conversation and improving the agent

A team can say that a call went badly, rarely what a good call is; without that definition, quality drifts. A short grid, applied every week to a sample, fixes this:

• Intelligibility: did the caller have to ask for something to be repeated?
• Accuracy: does the answer match the up-to-date source?
• Brevity: was the objective reached with no pointless turn?
• Clean exit: when the agent stopped, did the caller know what was going to happen next?

With one yes-or-no answer per conversation, these questions compare two versions of a script and settle an internal disagreement. The analysis then identifies the failure reasons; the missing intents, readable in the conversations that ended in a transfer, are the most useful material.

The failure still has to be clean. The recovery plan is defined once and for all, and its branches are written as lines of dialogue, not as specifications:

The improvement cycle is then repeated in a short loop:

• extract the twenty main reasons for transfer;
• correct the scripts and the knowledge;
• retest on a batch of calls already handled;
• deploy with close monitoring over the first few days.

Governance makes that cycle sustainable over time:

• versioning of prompts, scripts and sources;
• business sign-off on sensitive paths;
• a record of who changed what, when and why;
• a weekly quality review.

Without a record, a drop in quality cannot be tied to any change. Three drifts deserve a written safeguard from the design stage, and two others appear after a few months:

Continue reading

Depending on the stage your voice project has reached, three articles in the cluster take over:

• Your scripts are written and the question becomes one of retrieval: finding the right passage in your sources is set out in the RAG AI agent.
• You have designed the dialogue but not yet the agent that will carry it: the method, from scoping to go-live, is in the guide to create an AI agent.
• The design is settled and you have to choose what to build it with: models, vendors and tools are compared on the page devoted to AI agent platforms.

FAQ on AI-based voice agents

What is an AI-based voice agent?

An AI-based voice agent is software that holds a dialogue by voice in natural language, understands the intent, answers out loud and handles simple requests or hands over to a human. It chains speech recognition, understanding, orchestration and speech synthesis, which sets it apart from a simple voice generator.

How does an AI-based voice agent work?

A voice agent works in four stages: transcription of the voice, extraction of the intent and the entities, orchestration of the rules and tools, then generation and synthesis of the answer. An escalation to a human agent can occur at any layer, and each layer has failures of its own.

How does an AI-based voice agent differ from a chatbot and from an interactive voice response system?

Compared with a chatbot, the voice agent adds real time: turn-taking, interruptions, latency, no rereading. Compared with an interactive voice response system, it understands free sentences instead of numbered choices and transfers with the context collected. So you write lines of dialogue, not a tree of keypresses.

Which use cases are most relevant for an AI-based voice agent?

The most relevant use cases have a stable answer and a reversible consequence: identifying the reason, following up on a request, recurring questions, appointments, factual confirmation. They are written one by one as scenarios: 5 very well mastered paths are better than 50 approximate ones.

How do you reduce latency and improve the stability of a real-time AI-based voice agent?

Latency is reduced by treating the conversation as a stream: streaming, answers in two segments, a cache of stable answers, pre-warming before peaks, lighter prompts. Stability rests on a written recovery plan for failed recognition, the silent tool and the hesitating model.

How do you create effective scripts and a knowledge base for an AI-based voice agent?

Effective scripts start from a map of the intents and the exceptions, with one idea per sentence and one objective per turn. The base relies on approved sources with a named owner, broken into short units, timestamped, versioned and tested in conversation.

How do you define the brand personality and tone of an AI-based voice agent?

The personality of a voice agent is defined by applicable rules: register, style according to the situation, length of a turn, the way to say “I do not know”, forbidden promises, information it may ask for. Across sites, a common core remains and only what is necessary is localized.

Which is the best voice AI?

There is no universally best voice AI: the right solution holds your real scenarios with a controlled escalation rate. Compare offers on your own data and on observable criteria: delay before the first answer, stability, context retained, quality of transfers, control of the data collected.