Build an ElevenLabs Voice Agent That Handles Real Calls
A voice agent lives or dies on latency. Under 800ms round trip feels like a conversation; over 1.5 seconds feels like a machine and callers hang up.
The latency budget
Split your budget across four stages: speech-to-text, model response, text-to-speech, and network. Stream every stage you can. Start speaking the first sentence while the rest is still generating — perceived latency is what the caller judges.
Scripting rules for voice
Text written for the screen sounds wrong when spoken aloud.
- Cap responses at two sentences unless the caller asked for detail.
- Confirm names, dates, and numbers back to the caller.
- Handle interruptions — barge-in support is non-negotiable.
- Always offer a human handoff path.
Telephony and going live
Twilio connects a real phone number to your agent over a media stream. Before launch, run at least ten synthetic calls covering silence, background noise, an angry caller, and a wrong number. Add a recording disclosure — it is a legal requirement in most markets.
Frequently asked questions
How much does an AI voice agent cost to run?
Costs stack across speech-to-text, the model, text-to-speech, and telephony minutes. A short booking call typically lands in the cents range; long support calls cost meaningfully more.
Can an ElevenLabs voice agent book appointments?
Yes — connect a calendar tool to the agent loop so it can check availability and write the booking during the call.
Go deeper with the handbook
Chapter 1 is free to read — no sign-in required.