OpenAI Agents API + GPT-Live: what changes for production phone agents

I've been tearing down a Telnyx + Vapi-style inbound stack for the usual reasons: latency callers can feel, transfers that drop the plot, and “natural” voices that still produce unnatural conversation. The question now isn't “which TTS sounds nicer.” It's whether a tighter model loop — Agents API for tools/handoffs, GPT-Live for full-duplex voice — actually shrinks the failure modes that made the old stack unmaintainable.

This is a practitioner note, not a launch recap. Draft status means numbers and vendor claims will get sharpened as I re-run evals.

What “full-duplex” has to buy you on a phone line

Inbound business calls are overlapping speech, accents, background noise, and people who will not wait for polite turn-taking. Half-duplex pipelines (STT → LLM → TTS) can demo well and still lose on rhythm: barge-in weirdness, silence-fill that sounds needy, confirmations that arrive a beat late.

GPT-Live-style full-duplex only matters if it cuts end-to-end delay and improves interruption recovery. If p95 from caller utterance to useful agent audio doesn't move, you've swapped vendors without fixing the product.

Agents API: tools and handoffs without another glue layer

The Agents surface is interesting less for “multi-agent theater” and more for structured tool use and handoff packets — the same transfer problem that killed trust in the previous stack. Warm transfer is the product on inbound lines. If the human picks up without a two-second skim summary, staff treat the AI like fancy hold music.

What I want to measure: can an agent call tools, decide to escalate, and emit a handoff that a front-desk human actually uses — without a custom orchestration maze between telephony and the model.

What I'm not assuming

• That “native voice” automatically beats a well-tuned STT/TTS path on every line.
• That Agents API removes the need for line-specific evals and kill criteria.
• That fewer vendors equals fewer failure modes — sometimes it just concentrates them.

Eval checklist (same bars as the teardown)

1. Latency. Ring → first useful audio; utterance → response. Track p50/p95, not demos.
2. Transfer trust. Structured handoff; human re-ask rate; “agent failed gracefully” path.
3. Naturalness under stress. Interruptions, accents, TV in the background — not patient speakers.
4. Ops cost. Prompt/tool drift, debugging time, how fast you can kill a bad path.

Tie-back to the stack I'm ripping out

The previous note — I'm ripping out my AI phone stack — was about accumulated small failures. Agents + GPT-Live only earns a slot if it attacks those failures directly: fewer media hops, better duplex rhythm, and transfers that don't require the human to restart the call.

If it doesn't move those metrics in a fixed window, I rip again. Attachment to this quarter's architecture is how you keep paying for a bad phone experience.

Closing

I'll update this draft as audio samples and call traces land. Until then: treat GPT-Live and Agents as hypotheses against latency, transfers, and naturalness — not as a slide that says the phone problem is solved.

Comparing notes beats comparing logos. Say hello.