Foundations of Voice
Automation AI

Voice AI is any system where a computer converts human speech into meaning, reasons about it, and responds with generated speech — in near real-time. It is not one model. It is a pipeline of specialised models chained together: a listener, a thinker, and a speaker.

• Listener (STT): turns your microphone audio into text.
• Thinker (LLM): reads the text, plans a response, may call tools.
• Speaker (TTS): turns the model's reply back into natural audio.

The magic is not any single block — it's the orchestration: how fast the audio flows, how the model interrupts, how the voice sounds human. That orchestration is what you will learn to build.

1. 1952 — Audrey (Bell Labs): recognised spoken digits 0–9. One speaker only.
2. 1970s — Harpy (CMU): 1,011-word vocabulary, DARPA-funded.
3. 1990s — Dragon NaturallySpeaking: the first mainstream dictation software.
4. 2011 — Siri: voice assistants reach the phone in every pocket.
5. 2014–2018 — Alexa, Google Assistant, Cortana: voice becomes a UI layer.
6. 2022 — Whisper (OpenAI): open-source STT good enough for production.
7. 2023 — ElevenLabs: TTS finally crosses the "sounds human" threshold.
8. 2024 — GPT-4o & Realtime API: single-model, sub-second, speech-to-speech.
9. 2025+ — Agentic Voice: voice agents that book, buy, escalate, transfer.

The old world of phone menus ("Press 1 for billing…") is called IVR — Interactive Voice Response. It is a decision tree. It cannot handle a sentence it wasn't scripted for. An AI voice agent is the opposite: a general-purpose conversational brain.

Side-by-side

• Input: IVR listens for DTMF tones or a fixed keyword; AI listens for open speech.
• Understanding: IVR matches strings; AI understands intent, entities and context.
• Flow: IVR follows a tree; AI plans dynamically and can backtrack.
• Tone: IVR is robotic and rigid; AI is warm, interruptible, human-like.
• Cost: IVR is cheap per minute; AI is more expensive but replaces agents entirely.

Every production voice system, no matter the vendor, is some flavour of this loop:

microphone  →  [STT]  →  text  →  [LLM + tools]  →  reply text  →  [TTS]  →  speaker
                                    ↑                                              │
                                    └──── memory / context / interruption ─────────┘

Two things separate a toy demo from a real product: latency (how fast the round trip is) and turn-taking (who is allowed to speak when). We will return to both.

STT — sometimes called ASR (Automatic Speech Recognition) — converts raw audio samples into a text transcript. Modern STT models are neural networks trained on hundreds of thousands of hours of human speech.

What makes an STT model "good"

• Word Error Rate (WER): lower is better. State-of-the-art is under 5% on clean English.
• Latency: ideally under 300ms from end-of-speech to final transcript.
• Streaming: emits partial transcripts as you talk, not only at the end.
• Diarization: can tell speakers apart (Speaker A, Speaker B).
• Language coverage: Whisper covers 99+ languages, including Urdu.

Popular STT options

• OpenAI Whisper — great multilingual quality, batch or streaming.
• Deepgram Nova-2 — very low latency, telecom-grade.
• AssemblyAI — strong diarization and formatting.
• ElevenLabs Scribe — high accuracy, tightly integrated with their TTS.

Once you have text, an LLM reads it and decides what to say back. In a voice pipeline the LLM must be more than a chatbot — it must be fast, must be able to call tools (look up an order, schedule a callback), and must produce speakable text (short sentences, no bullet points, no markdown).

What matters for voice

• Time to first token (TTFT): anything above 800ms feels laggy on a phone call.
• Streaming: stream tokens straight into TTS so the caller hears speech before the model finishes.
• Tool use: function calling is how your agent actually does things.
• Instructability: the model must obey a strict persona and style guide.

TTS turns the model's text into audio. Modern neural TTS is nearly indistinguishable from a real human. The three properties you evaluate are naturalness (does it sound human?), latency (how long from text to first audio byte?), and steerability(can you control emotion, pace, whisper, laugh?).

Top TTS engines today

• ElevenLabs — industry benchmark for naturalness and voice cloning.
• OpenAI tts-1 / gpt-4o-audio — great quality, tight OpenAI integration.
• Google Chirp / Cloud TTS — huge language coverage.
• Azure Neural TTS — enterprise pricing, hundreds of voices.

Voice cloning creates a synthetic voice from a short sample of a real person. With ElevenLabs, 30 seconds of clean audio is enough for instant cloning; a few hours produces a professional clone nearly indistinguishable from the original.

Legitimate uses

• Founders cloning their own voice for a personal assistant.
• Podcasters generating corrections without re-recording.
• Accessibility — restoring the voice of someone who lost theirs to illness.
• Localised marketing where the brand voice must stay consistent across languages.

On a phone call, humans expect a response within about 500 milliseconds. Anything above one second feels awkward; above two, broken. Your budget for the entire round trip is small, and every hop eats into it.

Typical budget for a natural conversation
────────────────────────────────────────
End-of-speech detection ....  120 ms
STT final transcript .......  200 ms
LLM time to first token ....  350 ms
TTS first audio byte .......  250 ms
Network + jitter ...........  180 ms
────────────────────────────────────────
Total ......................  ~1.1 s

Streaming means data flows continuously instead of waiting for a complete block. Every serious voice stack streams at three points:

1. Mic → STT: audio is chunked every 100ms and sent live.
2. STT → LLM: partial transcripts trigger early planning; the final transcript triggers the actual response.
3. LLM → TTS → speaker: tokens flow into TTS which streams audio bytes back to the caller.

A real-time voice agent is one that supports barge-in (the user can interrupt the AI mid-sentence and the AI stops immediately), backchannel ("uh-huh", "mhm" while the user speaks), and emotion. The industry breakthrough came with OpenAI's Realtime API and models like gpt-4o-realtime, which handle speech-to-speech inside a single model — skipping the STT and TTS hops for lower latency.

ElevenLabs is the reference platform for production voice. Three products matter for us:

• TTS API — 5000+ voices, 32 languages, streaming with sub-400ms first byte.
• Voice Cloning — instant and professional tiers.
• Conversational Agents — hosted STT+LLM+TTS with a React SDK; drop-in phone agents.

OpenAI provides three complementary pieces: Whisper for STT, the tts-1 / gpt-4o-audio family for TTS, and the Realtime API for end-to-end speech-to-speech. If you want a single vendor and the shortest possible pipeline, this is it.

• Great English quality; multilingual is good but not always native-sounding.
• Realtime API supports function-calling mid-conversation.
• Fewer voices than ElevenLabs but tighter integration if you already use GPT.

Claude (Anthropic) does not, at time of writing, ship its own STT or TTS. But Claude is a first-class brain for the LLM slot — especially for agents that must reason carefully, follow long system prompts, and call tools reliably. A common production stack is:

Deepgram (STT)  →  Claude 3.5 Sonnet (LLM + tools)  →  ElevenLabs (TTS)

Use Claude when your voice agent must handle nuance, refuse gracefully, or drive a complex tool chain (booking, refunds, medical triage). Use GPT-4o when you want the shortest possible latency via a single Realtime model.

Zoomed in, a production pipeline has more moving parts than the three-block picture suggests. Here is the real thing:

┌──────────────┐   audio    ┌────────────┐  text   ┌────────────┐
│  Microphone  │ ─────────▶ │  VAD +     │ ──────▶ │    STT     │
│  (browser or │            │  Noise     │         │ (streaming)│
│   telephony) │            │  Reduction │         └─────┬──────┘
└──────────────┘            └────────────┘               │
                                                         ▼
                                                  ┌────────────┐
                                                  │   LLM      │
                                                  │ + memory   │
                                                  │ + tools    │
                                                  └─────┬──────┘
                                                        │ tokens
                                                        ▼
┌──────────────┐   audio    ┌────────────┐   text  ┌────────────┐
│   Speaker    │ ◀───────── │ Playback + │ ◀────── │    TTS     │
│              │            │ Barge-in   │         │ (streaming)│
└──────────────┘            └────────────┘         └────────────┘

• VAD (Voice Activity Detection) tells you when the user started/stopped speaking.
• Memory holds recent turns and long-term user facts.
• Tools are functions the LLM can call (lookup order, send SMS, transfer call).
• Barge-in monitors mic energy during playback and cuts TTS instantly.

Here is the canonical architecture we will build in this course. Memorise it — every project in the next nine chapters is a specialisation of this diagram.

                        Input (spoken words)
                                 │
                                 ▼
                     ┌───────────────────────┐
                     │   Speech Recognition  │  ← Whisper / Deepgram
                     └──────────┬────────────┘
                                │ text
                                ▼
                     ┌───────────────────────┐
                     │          LLM          │  ← GPT-4o / Claude 3.5
                     │   (prompt + tools)    │
                     └──────────┬────────────┘
                                │ reply text
                                ▼
                     ┌───────────────────────┐
                     │      ElevenLabs       │  ← streaming TTS
                     └──────────┬────────────┘
                                │ audio bytes
                                ▼
                          Voice Output

Explaining every component

• Input: the mic (browser getUserMedia) or a phone via Twilio/Vonage.
• Speech Recognition: streams partial + final transcripts to your server.
• LLM: receives the transcript plus a system prompt, may call tools, streams tokens.
• ElevenLabs: receives streamed text, returns streamed audio (MP3 or PCM).
• Voice Output: an <audio> element in the browser or an RTP stream on the phone leg.

Below is the smallest possible working pipeline: browser records a clip, we send it to a server function that transcribes with Whisper, asks the LLM for a reply, and returns TTS audio from ElevenLabs. In Chapter 2 we will make this streaming and real-time.

// server: transcribe + reply + speak
import { createServerFn } from "@tanstack/react-start";

export const speakBack = createServerFn({ method: "POST" })
  .handler(async ({ data }: { data: { audio: string } }) => {
    // 1) STT — Whisper via Lovable AI Gateway
    const form = new FormData();
    form.append("model", "openai/gpt-4o-mini-transcribe");
    form.append("file", await (await fetch(data.audio)).blob(), "in.webm");
    const stt = await fetch("https://ai.gateway.lovable.dev/v1/audio/transcriptions", {
      method: "POST",
      headers: { Authorization: `Bearer ${process.env.LOVABLE_API_KEY}` },
      body: form,
    }).then(r => r.json());

    // 2) LLM reply — short, speakable
    const chat = await fetch("https://ai.gateway.lovable.dev/v1/chat/completions", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${process.env.LOVABLE_API_KEY}`,
      },
      body: JSON.stringify({
        model: "openai/gpt-4o-mini",
        messages: [
          { role: "system", content: "Reply in one short spoken sentence. No markdown." },
          { role: "user", content: stt.text },
        ],
      }),
    }).then(r => r.json());
    const reply = chat.choices[0].message.content as string;

    // 3) TTS — ElevenLabs
    const audio = await fetch(
      "https://api.elevenlabs.io/v1/text-to-speech/EXAVITQu4vr4xnSDxMaL",
      {
        method: "POST",
        headers: {
          "xi-api-key": process.env.ELEVENLABS_API_KEY!,
          "Content-Type": "application/json",
        },
        body: JSON.stringify({ text: reply, model_id: "eleven_turbo_v2_5" }),
      },
    ).then(r => r.arrayBuffer());

    return { reply, audioBase64: Buffer.from(audio).toString("base64") };
  });

// client: record → send → play
const rec = new MediaRecorder(await navigator.mediaDevices.getUserMedia({ audio: true }));
const chunks: Blob[] = [];
rec.ondataavailable = (e) => chunks.push(e.data);
rec.onstop = async () => {
  const url = URL.createObjectURL(new Blob(chunks, { type: "audio/webm" }));
  const { audioBase64 } = await speakBack({ data: { audio: url } });
  new Audio("data:audio/mpeg;base64," + audioBase64).play();
};
rec.start(); setTimeout(() => rec.stop(), 4000);