Every voice agent you'll ever build fails in one of ten ways. This chapter is the symptom → cause → fix reference you keep open next to the terminal. Bookmark it.
Section 1
1. Bad Audio In → Garbage Out
- Symptom: STT returns "…" or wildly wrong words.
- Cause: sample rate mismatch, wrong codec, or noisy source.
- Fix: force 16 kHz mono PCM/WAV. Denoise with rnnoise. Reject calls where input RMS < -50 dBFS.
Section 2
2. Latency > 2s
- Measure each block: ms_stt, ms_llm, ms_tts. Fix the biggest first.
- Switch to streaming for both STT (Deepgram) and TTS (ElevenLabs /stream).
- Cap Claude max_tokens to 200 for phone turns.
- Deploy in the same region as your telephony provider.
Section 3
3. Poor Voice Quality
- Robotic → stability too high. Drop to 0.35.
- Slurred → text has no punctuation. Add commas and periods before TTS.
- Wrong pronunciation of a brand → use SSML <phoneme> or <sub>.
Section 4
4. Wrong Voice Settings
Copy the presets from Chapter 2 and stop tuning. Only revisit if a real user complains.
Section 5
5. Streaming Issues
- Choppy audio → your event loop is blocked. Move CPU work off the main thread.
- Silence in the middle → back-pressure. Consume the response body without buffering.
- WebSocket closes at 60s → add a heartbeat ping every 20s.
Section 6
6. API Errors — Reading Them Right
tsclaude-code
if (!res.ok) {
const body = await res.text();
log.error({ status: res.status, body }, "elevenlabs.fail");
throw new Error(`ElevenLabs ${res.status}: ${body}`);
}Section 7
7. Rate Limits (429)
- Exponential backoff: 1s, 2s, 4s (max 3 retries).
- Queue overflow on ElevenLabs concurrent request cap.
- Split traffic across two API keys if a plan caps you.
Section 8
8. Authentication Errors (401)
- Copy-paste added a space. Trim the key.
- Key rotated in the dashboard but not in the platform vault.
Section 9
9. Voice Clone Problems
- Muffled clone → training WAV had reverb.
- Wrong accent → source had < 60s. Provide at least 90s.
- Cloning fails on upload → convert with ffmpeg -ar 22050 -ac 1.
Section 10
10. The Debugging Playbook
- Reproduce with a saved WAV — remove the phone.
- Log every request/response pair for the failing call ID.
- Replay the exact prompt in the Anthropic console.
- Curl the ElevenLabs endpoint with the exact JSON you sent.
- Compare a working call to a failing one — diff the payloads.
Section 11
11. Monitoring — What to Actually Watch
- p95 turn latency (alert > 2s).
- Error rate per provider (alert > 2%).
- Cost per call (alert > 2× yesterday's median).
- Barge-in success rate.
Section 12
12. Best Practices
- One system prompt file per persona, versioned in git.
- Cache greetings as static MP3s.
- Every call has a UUID that joins Twilio + logs + DB + Sentry.
- Ship a synthetic call in CI.
- Keep a "known failures" doc.
Section 13
13. Exercises
- Force a 429 in dev; implement backoff and prove it recovers.
- Introduce a 3-second CPU spike; observe choppy audio; fix with a worker.
- Rotate the ElevenLabs key mid-call; add automatic re-read of secrets on 401.