Model eval as a Slack message: Descript’s Underlord loop on OpenRouter

OpenRouter’s Sep 21, 2026 write-up on Descript is not another “we multi-homed providers” case study. It is a queue story. Before OpenRouter, Descript’s Underlord team kept three direct integrations — OpenAI, Anthropic, Google — plus DIY fallback logic. Adding a model was a couple of engineer hours. Getting those hours onto a calendar took days. Promising models sat for a week before anyone knew whether they cleared the bar.

That is the same failure mode I keep hitting on phone and ops desks: model choice collapses into a wiring project. The useful fix is not “more providers.” It is making the model a parameter and putting eval into a harness the team can fire without booking eng time.

The expensive part was the waiting

Descript builds Underlord, its video-editing agent. When model access is wired provider by provider, the person who wants to try a model is rarely the person who can add it. The request has to be explained, scheduled, and waited on. The team tried models worth interrupting an engineer for and skipped the rest. Throughput of experiments — not raw API latency — set how many models they actually tried.

Aleks Mistratov, Head of AI Product at Descript, describes the loop now:

“I’m on a walk, I’m very online to my own detriment, and I see a tweet that a new model is out. I go into Slack and say: run evals on this model on OpenRouter, and link to it. An hour or two later we’ve run evals in our harness on that model.”

The mechanism is Claude Tag — Anthropic’s Slack integration — configured to run the evaluations and open pull requests. A human still reviews and approves. What made the workflow automatable is that trying a model no longer requires building a provider integration first. There is still work in it. Nobody has to be asked for calendar time to start it.

What weekly evaluation changed

Descript now evaluates models multiple times a week. Most of what they test does not ship. Three consequences follow — and they map cleanly to production agent desks:

Model choice is empirical. New releases get measured against Descript’s own harness, not against a launch blog. The old default — standardize early, revisit when something forces you — is what phone stacks do when “swap the LLM” means a sprint.
Releases stop being events. On a recent Anthropic drop, Descript went from announcement to live in a couple of hours: run evals, decide whether it beats the incumbent, add it to the picker, deploy. The remaining steps are internal structure, not vendor wiring.
The field widened. Descript never built a direct xAI integration. Grok 4.5 runs in production through the same path as everything else. That is the point of a router that keeps the model id stable while the provider behind it can move.

Provider swap, model constant

None of this means “route everything through a shared endpoint and accept whatever comes back.” Descript brings its own Baseten key for a dedicated open-weight deployment. OpenRouter routes to Baseten first; other providers sit as fallbacks for the same model. Failover swaps the inference host and keeps the model constant — so a degraded provider does not silently change which brain answers the call.

Mistratov’s advice to engineers weighing the same change is short:

“It’s far easier than trying to figure out all these different connections. You do it once and everything else is a parameter.”

Same integration for testing new models and for running the ones that stay. Testing finds a model. Routing configuration runs one.

How this maps to phone and ops agents

Production phone agents already fail when secrets, tools, and model identity collapse into one process. The Descript loop is the model half of the same architecture family as OpenRouter shell/Files blast-radius and Muse Secure VM credential surrogates:

Treat model swap as a parameter + harness eval, not a wiring project. If “try DeepSeek Flash instead of Luna” needs a ticket and a calendar slot, your desk is still in Descript’s before state.
Own the harness, not the provider friendship. Eval cases for booking, cancellation, FAQ, and warm-transfer should live in your repo. Slack → agent → PR is optional; a green eval gate is not.
Provider fallbacks must keep the model constant. Same lesson as “agent never sees the password”: the boundary owns failover. A Baseten-first open-weight route with same-model backups is closer to production phone ops than “whatever is cheapest today.”
Rankings are a routing signal, not a religion. OpenRouter weekly through Sep 20, 2026: DeepSeek V4.1 Flash #1 (~15.8T, +219%), GLM 5.3 Flash #2 (~14.1T), Hy4 preview #3 (~12.5T), GPT-5.6 Luna #4 (~9.72T), DeepSeek V4 Flash 0731 #5 (~9.44T). Apps still Hermes / Claude Code / Kilo / Cline heavy. Volume moved; your eval bars did not.

Foil worth naming: Claude’s voice-mode update (Opus/Sonnet/Haiku, connected tools, more languages) is still turn-based chat with a mic — useful for thinking out loud, not a speech-native phone agent. Descript’s lesson is about eval throughput for the models behind agents; it does not replace full-duplex telephony bars from the GPT-Live draft.

What I’d copy into a desk this week

One OpenRouter (or equivalent) integration; stop adding direct provider SDKs for every trial.
A harness the team can trigger without eng calendar — Slack bot, CI label, or a one-command script.
Human approve on the PR that changes the production picker; auto-eval is not auto-ship.
Dedicated keys for open-weight hosts you care about; configure same-model provider fallbacks.
Weekly multi-eval budget; expect most candidates not to ship — that is the point.

Closing

Descript cut model evaluation from “engineer hours + days of queue” to “Slack message → evals in one to two hours → human-approved PR.” The architectural bet is the same one as hosted shell with default-deny network and Secure VM surrogates: name the boundary, keep the expensive coupling outside the model, and let the model become a parameter you measure instead of a project you schedule.

Related drafts: OpenRouter shell + Files API, Muse Secure VM credentials, OpenRouter rankings in production, GPT-Live voice, phone stack teardown. Comparing notes beats comparing logos. Say hello.

Sources: OpenRouter — How Descript Cut Model Evaluation to Hours (Sep 21, 2026); OpenRouter rankings (usage through Sep 20, 2026); Claude — Think through hard problems in voice mode (foil: turn-based voice, not phone-native).