Advanced Voice
Automation

Streaming means the caller hears the first syllable before the model has finished the last word. ElevenLabs' /stream endpoint returns audio as it's generated.

const res = await fetch(url + "/stream", { method: "POST", ... });
for await (const chunk of res.body!) writeToCaller(chunk);   // first byte ~ 400ms

When Deepgram fires a speech_started event while the agent is speaking, stop the TTS stream immediately, clear the outbound buffer, and start listening. Callers try to interrupt on average every 6 seconds.

Target end-to-end round trip: 1200 ms
  Deepgram partial → final:  ~250 ms
  Claude first token:        ~350 ms
  Claude full reply (short): ~400 ms
  ElevenLabs first audio:    ~400 ms
  Network + jitter:          ~100 ms

• Short-term — last N turns kept verbatim in the prompt.
• Long-term — key facts summarised into a "profile row" in Supabase.
• Episodic — the previous call's summary passed in as system context.

Cap history at ~20 turns. On overflow, summarise the oldest 10 into 2 sentences.

Same brain, different masks: switch voice_id + system prompt tone together.

const PERSONAS = {
  warm:      { voice: "EXAVITQu4vr4xnSDxMaL", tone: "warm, brief, human" },
  formal:    { voice: "21m00Tcm4TlvDq8ikWAM", tone: "professional, precise" },
  playful:   { voice: "AZnzlk1XvdvUeBnXmlld", tone: "witty, upbeat, 1 emoji max" },
};

Push style up (0.4–0.7) for excitement or sadness. Keep stability low (0.2–0.35) so the emotion actually lands. Test with real speakers before you ship.

Your order <break time="300ms"/> ships tomorrow.
Track it here <emphasis level="strong">acme.co/track</emphasis>.
Call us at <say-as interpret-as="telephone">18005551234</say-as>.

• ElevenLabs text-to-speech/{id}/stream — HTTP chunked.
• ElevenLabs WebSocket — send text chunks, receive audio chunks. Best for LLM → TTS pipelines with token-level streaming.
• Deepgram WebSocket — streaming STT with partials.

Non-real-time jobs — audiobook generation, batch dubbing — belong in a queue. Push a job to Supabase or Redis, process it with a worker, notify by webhook when done.

• Use eleven_turbo_v2_5 for real-time (cheaper, faster). Keep multilingual_v2 for premium narration.
• Cache greetings and menu prompts as static MP3s — don't regenerate every call.
• Cap Claude to max_tokens: 200 for phone turns.
• Bill per-minute; alert at 80% of monthly budget.

• Stateless server → horizontal autoscale on Railway/Fly/AWS.
• State (calls, transcripts) → Postgres with connection pooling (pgbouncer).
• Concurrency limit per API key — ElevenLabs plans cap you; queue overflow.

log.info("turn.done", {
  call_id, turn_no, ms_stt, ms_llm, ms_tts, tokens_in, tokens_out, cost_usd,
});

Ship metrics to Grafana / Axiom / Datadog. Alert on p95 latency > 2s.

               ┌─── Twilio ───┐
               ▼              │
      ┌── Load Balancer ──┐   │
      ▼                   ▼   │
  Agent Node          Agent Node  ← horizontal
      │                   │
      └──────┬────────────┘
             ▼
      ┌──── Redis ────┐   ← call state
      ┌── Postgres ───┐   ← durable log
      ┌── Object Storage ── ← recordings

• Call center co-pilot — barge-in + emotion + real-time transcript.
• News radio agent — SSML + persona switching hourly.
• Language tutor — long-term memory + adaptive difficulty.