I'm ripping out my AI phone stack — here's what actually failed.

For a while the pitch was clean: put an AI on an inbound business phone line, let it qualify, answer FAQs, and warm-transfer to a human when it should. On paper the stack looked solved. In production it accumulated small failures that compounded into something I no longer want to maintain.

This is a teardown note — what broke, what I'm evaluating, and what I'd tell myself six months ago. Specifics stay generic on purpose.

The problem: inbound agents on business phone lines

The use case is boring in the best way. Callers ask about availability, scheduling, hours, policies, and “can someone just call me back.” Volume is spiky. Staffing is uneven. Missed calls are expensive in a quiet way.

An agent that answers, stays calm, and routes correctly is worth more than a clever demo. The bar isn't “sounds like GPT.” The bar is: does the team answering the line trust it on a busy day?

What we were running

Telephony and media through Telnyx. Conversation layer through Vapi. Tools for calendar / CRM hooks, a thin prompt layer per line or property, and a transfer path to on-call humans. Fine for a pilot. Fragile once you care about every extra 400ms and every clumsy handoff.

What actually failed

Latency that callers can feel

End-to-end delay wasn't catastrophic in isolation. It was inconsistent. A half-second pause after a simple yes/no is enough for someone to talk over the agent — then you get barge-in weirdness, repeated questions, and the caller deciding the line is “broken.”

Telephony hop + STT + LLM + TTS + back out is a chain. Optimizing one link while the others wander doesn't fix the experience. Callers don't care which vendor owns the lag.

Transfers that drop the plot

Warm transfer was the feature everyone wanted and the one that kept failing the vibe check. Context sometimes arrived incomplete. Timing on the bridge felt abrupt. Humans would pick up mid-sentence or without a crisp summary, then re-ask everything the agent already collected.

Once staff stop trusting the summary, they treat the AI like a fancy hold music. At that point you're paying for automation theater.

Naturalness that doesn't survive real speech

Demo calls with a patient speaker sound great. Real inbound is overlapping speech, accents, background TV, kids, cars, and people who refuse to wait for turn-taking. “Natural” voices still produce unnatural conversation when interruption handling and silence detection are off by a little.

The uncanny part wasn't the voice — it was the rhythm. Too slow to confirm. Too eager to fill silence. Too polite when the caller wanted a yes or a transfer.

What I'm evaluating next

• Tighter media path. Fewer hops between carrier audio and the model loop. Measure p50/p95 from ring to first useful audio, not just LLM TTFT.
• Transfer as a first-class product. Structured handoff packets humans can skim in two seconds. Explicit “agent failed gracefully” paths.
• Line-specific evals. Scripted golden calls plus live shadow scoring on interruption recovery, not just transcript BLEU fantasies.
• Routing discipline. Smaller / faster models for intake; escalate model size only when the turn needs it. OpenRouter-style routing where it earns its keep.
• Kill criteria. If latency or transfer trust doesn't move in a fixed window, rip again — don't polish a stack that already lost staff confidence.

Lessons that stuck

1. Pilots optimize for “wow.” Production optimizes for “don't make the front desk angry.”
2. Transfer quality is the product for inbound business phone lines. Everything else is supporting cast.
3. Inconsistent latency feels worse than slightly higher but stable latency.
4. Generic names in writeups keep the post useful and the operators safe — specifics belong in private postmortems.
5. Willingness to tear the stack down is a feature. Attachment to last quarter's architecture is how you keep paying for a bad phone experience.

Closing

I'm not anti-vendor. I'm anti-pretending the stack worked because the slide deck looked modern. The next version has to win on feel: answer fast, transfer clean, sound like it belongs on a real business phone line.

If you're shipping inbound voice agents and you've hit the same walls — latency, transfers, rhythm — I'd rather compare notes than compare logos. Say hello.