Ch 5: ElevenLabs Voice Agents

Vertical phone agents are the highest-margin product you can build with voice AI. Every small business you know pays $2,000/month for a receptionist and misses half their calls anyway. This chapter is a playbook for seven of those businesses, all built on the same skeleton — swap the prompt, swap the tools, ship the agent.

Section 1

1. The Shape of a Phone Agent

diagramclaude-code

Twilio number ──▶ WebSocket ──▶ your server
                                    │
                        ┌───────────┼───────────┐
                        ▼           ▼           ▼
                    Deepgram     Claude     ElevenLabs
                        │           │           │
                        └───────────┴───────────┘
                                    │
                              audio back to caller

Section 2

2. Twilio Setup

Sign up at twilio.com, verify a personal number.
Buy a US number ($1/month) — Phone Numbers → Buy a number.
Under Voice → A call comes in, point to your webhook URL.

Claude Code — Terminal

Screenshot placeholder

Twilio number config page with the incoming-call webhook pointing at https://agent.example.com/incoming.

Figure — Twilio number config page with the incoming-call webhook pointing at https://agent.example.com/incoming.

Section 3

3. Media Streams (real-time audio)

xmlclaude-code

<Response>
  <Start>
    <Stream url="wss://agent.example.com/twilio" />
  </Start>
  <Pause length="600" />
</Response>

tsclaude-code

ws.on("message", async (msg) => {
  const evt = JSON.parse(msg.toString());
  if (evt.event === "media") {
    const audio = Buffer.from(evt.media.payload, "base64");
    deepgram.send(audio);
  }
});
deepgram.on("final", async (text) => {
  const reply = await claude(text);
  const speech = await elevenStream(reply);
  for await (const chunk of speech) {
    ws.send(JSON.stringify({ event: "media", media: { payload: chunk.toString("base64") } }));
  }
});

Section 4

4. Restaurant Booking Agent

textclaude-code

SYSTEM: You are Marco, host at Osteria Luna.
Confirm party size, date, time, name, phone.
Tools: checkTable, bookTable. Wrap in <= 4 turns.

Section 5

5. Dental Clinic Agent

textclaude-code

SYSTEM: You are Priya at Bright Smile Dental.
Triage emergency vs routine. Emergencies transfer immediately.
Tools: nextAvailableSlot, bookAppointment, transfer.

Section 6

6. Salon Booking Agent

Same skeleton, add service duration lookup so slots reflect a 90-min color vs a 30-min trim.

Section 7

7. HVAC Booking Agent

Add urgency triage — "no heat", "no AC in a heatwave" → same-day slot. Collect zip code to route the right technician.

Section 8

8. Real Estate Agent

Inbound: qualify lead (budget, area, timeline). Book a viewing on the listing agent's calendar. Text the caller a link after hangup.

Section 9

9. Lead Qualification Agent

textclaude-code

SYSTEM: Friendly qualifier for Acme B2B.
Ask 4 questions max: role, team size, current tool, timeline.
Score 1–5. If >=4, book a demo. Otherwise, send nurture email.

Section 10

10. Generic Appointment Booking Agent

Reusable prompt with placeholders for business name, services, and calendar tool.

Section 11

11. Call Transfer

xmlclaude-code

<Response>
  <Say voice="Polly.Joanna">Connecting you to a teammate now.</Say>
  <Dial>+15551234567</Dial>
</Response>

Section 12

12. Voicemail

xmlclaude-code

<Response>
  <Say>Please leave a message after the tone.</Say>
  <Record maxLength="60" transcribe="true" />
</Response>

Section 13

13. FAQ Handling

Give Claude a searchFAQ(query) tool that returns top-3 snippets. Force citations in the reply.

Section 14

14. CRM Integration

tsclaude-code

await supabase.from("calls").insert({
  phone: from,
  duration_s: ended - started,
  transcript, intent, outcome,
});

Section 15

15. Google Calendar

tsclaude-code

import { google } from "googleapis";
const cal = google.calendar({ version: "v3", auth });
await cal.events.insert({
  calendarId: "primary",
  requestBody: {
    summary: "Cleaning — Sara Ali",
    start: { dateTime: "2026-07-24T10:00:00-05:00" },
    end:   { dateTime: "2026-07-24T10:30:00-05:00" },
  },
});

Section 16

16. Twilio Integration Tips

Use signed webhook validation — verify X-Twilio-Signature on every POST.
Set machine_detection=Enable for outbound calls to skip voicemail greetings.
Log every CallSid — it's your join key across Twilio, DB, and support tickets.

Section 17

17. Ship Checklist

One number, one webhook, one server — end-to-end tested.
Latency < 1.5s per turn.
Transfer path works to a real human.
Every call logged with transcript + outcome.
Google Calendar event created for at least one test booking.

← Chapter 4 · Tools Back to chapters Chapter 6 →