Ch 3: ElevenLabs Voice Agents

You have a key, a voice, and a dashboard. Time to make them talk. In this chapter we build an AI Receptionist: a caller says "Hi, do you have appointments tomorrow morning?" and our agent answers, in a warm human voice, within two seconds. Same architecture powers every voice agent in this book — nail it once, reuse it forever.

Section 1

1. Project Brief: AI Receptionist

Answers general questions about the business.
Understands appointment intent and confirms availability.
Handles interruptions gracefully.
Escalates to a human when it cannot help.

Section 2

2. The Flow

diagramclaude-code

Customer Speaks
     │
     ▼
Speech to Text  (Whisper / Deepgram / AssemblyAI)
     │
     ▼
Claude  (understand → plan → reply)
     │
     ▼
ElevenLabs  (synthesize voice)
     │
     ▼
Customer Hears Response

Claude Code — Terminal

Screenshot placeholder

Whiteboard drawing of the four-block pipeline with latency annotations at each arrow.

Figure — Whiteboard drawing of the four-block pipeline with latency annotations at each arrow.

Section 3

3. STT Options — Compared

Whisper (OpenAI) — best accuracy, batch. ~$0.006/min. Latency 1–2s.
Deepgram Nova-2 — real-time streaming, sub-300ms partials. ~$0.0043/min.
AssemblyAI Universal — great multilingual, speaker diarization built in.

4. Whisper

pythonclaude-code

from openai import OpenAI
client = OpenAI()

with open("caller.wav", "rb") as f:
    r = client.audio.transcriptions.create(
        model="whisper-1",
        file=f,
        language="en",
    )
print(r.text)

5. Deepgram

tsclaude-code

// npm i @deepgram/sdk
import { createClient } from "@deepgram/sdk";
const dg = createClient(process.env.DEEPGRAM_API_KEY!);

const live = dg.listen.live({ model: "nova-2", smart_format: true, interim_results: true });
live.on("Results", (data) => {
  const text = data.channel.alternatives[0].transcript;
  if (data.is_final && text) console.log("USER:", text);
});

6. AssemblyAI

pythonclaude-code

import assemblyai as aai
aai.settings.api_key = "YOUR_KEY"
tx = aai.Transcriber().transcribe("caller.wav")
print(tx.text)

Section 7

7. Claude as the Brain

Claude is where intent becomes action. We give it a tight system prompt, the running conversation, and a couple of tools (lookupAvailability, bookAppointment).

textclaude-code

SYSTEM
You are Nova, the receptionist for Bright Smile Dental.
Rules:
 - One short sentence unless the caller asks for detail.
 - Warm tone. Never say "as an AI".
 - If the caller wants to book, call bookAppointment(name, phone, slot).
 - If unsure, offer to transfer: "Let me connect you to a teammate."

Section 8

8. ElevenLabs Voice

Reuse the Chapter 2 code path — stream the response back so audio starts within ~400ms.

tsclaude-code

const speech = await fetch(
  `https://api.elevenlabs.io/v1/text-to-speech/${VOICE_ID}/stream?output_format=mp3_44100_128`,
  {
    method: "POST",
    headers: { "xi-api-key": KEY, "Content-Type": "application/json" },
    body: JSON.stringify({ text: reply, model_id: "eleven_turbo_v2_5" }),
  },
);

Section 9

9. n8n — Visual Orchestration

n8n is a self-hostable Zapier for developers. For voice agents, it's the fastest way to wirewebhook → transcribe → Claude → ElevenLabs → return audio without writing a server.

Claude Code — Terminal

Screenshot placeholder

n8n canvas showing five nodes chained: Webhook → OpenAI Whisper → Claude → ElevenLabs → Respond to Webhook.

Figure — n8n canvas showing five nodes chained: Webhook → OpenAI Whisper → Claude → ElevenLabs → Respond to Webhook.

Section 10

10. Project Setup

bashclaude-code

mkdir receptionist && cd receptionist
npm init -y
npm i openai @anthropic-ai/sdk dotenv express multer

Section 11

11. Capture the Caller

For local dev, accept an uploaded .wav. In Chapter 5 we'll swap this for Twilio.

tsclaude-code

import express from "express";
import multer from "multer";
const upload = multer({ dest: "/tmp" });
const app = express();

app.post("/turn", upload.single("audio"), async (req, res) => {
  const audioPath = req.file!.path;
  const text      = await transcribe(audioPath);
  const reply     = await think(text);
  const mp3       = await speak(reply);
  res.type("audio/mpeg").send(mp3);
});
app.listen(3000);

Section 12

12. Transcribe (Whisper)

tsclaude-code

import OpenAI from "openai";
import { createReadStream } from "node:fs";
const openai = new OpenAI();

export async function transcribe(path: string) {
  const r = await openai.audio.transcriptions.create({
    file: createReadStream(path),
    model: "whisper-1",
  });
  return r.text;
}

Section 13

13. Reason (Claude)

tsclaude-code

import Anthropic from "@anthropic-ai/sdk";
const claude = new Anthropic();
const history: { role: "user" | "assistant"; content: string }[] = [];

export async function think(userText: string) {
  history.push({ role: "user", content: userText });
  const r = await claude.messages.create({
    model: "claude-sonnet-4-5",
    max_tokens: 200,
    system: SYSTEM_PROMPT,
    messages: history,
  });
  const reply = (r.content[0] as { text: string }).text;
  history.push({ role: "assistant", content: reply });
  return reply;
}

Section 14

14. Speak (ElevenLabs)

tsclaude-code

export async function speak(text: string): Promise<Buffer> {
  const r = await fetch(
    `https://api.elevenlabs.io/v1/text-to-speech/${process.env.ELEVENLABS_VOICE_ID}`,
    {
      method: "POST",
      headers: {
        "xi-api-key": process.env.ELEVENLABS_API_KEY!,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        text,
        model_id: "eleven_turbo_v2_5",
        voice_settings: { stability: 0.35, similarity_boost: 0.8 },
      }),
    },
  );
  return Buffer.from(await r.arrayBuffer());
}

Section 15

15. The Turn Loop

Every call is a series of turns: caller speaks → agent listens → agent speaks. Keephistory in memory per call ID; discard after hangup.

Section 16

16. Local Test Drive

bashclaude-code

sox -d -c 1 -r 16000 caller.wav trim 0 3
curl -F "audio=@caller.wav" http://localhost:3000/turn --output reply.mp3
open reply.mp3

Section 17

17. Same Agent in n8n

Add a Webhook node → POST /receptionist.
OpenAI node → Whisper transcription of the incoming file.
Anthropic node → Claude with the same system prompt.
HTTP Request node → ElevenLabs TTS, response as file.
Respond to Webhook → return the audio.

Section 18

18. Exercises

Swap Whisper for Deepgram streaming — measure the latency drop.
Add a bookAppointment tool that appends to a JSON file.
Detect silence > 1.5s and prompt: "Are you still there?"

Section 19

19. Quiz

Which STT is best for real-time phone audio, and why?
What does eleven_turbo_v2_5 optimize for over eleven_multilingual_v2?
Where does history live and when do you discard it?

← Chapter 2 · Setup Back to chapters Chapter 4 →