Voice agent testing
A voice agent can read back a clean transcript and still fail the call. It talks over the caller, leaves three seconds of dead air, fires the wrong tool, or says a refund posted that never reached your backend. Text-only tests never see any of it, because the failure lives in the timing and the two audio channels, which a transcript flattens away. Voice agent testing has to measure the call, not a paraphrase of it.
Why a passing transcript is not a passing call
When you evaluate a voice agent from its transcript, you throw away the two things that make voice hard. The first is time. A transcript records that the caller spoke and the agent spoke, but not that the agent started 40 milliseconds into the caller's sentence, or waited two seconds before it yielded the floor. The second is separation. Real calls are two channels, caller and agent, and once they are merged into one column of words, an interruption and a clean handoff look identical.
The result is a test suite that goes green while callers hang up. A model judge reading the transcript reports that the agent sounded correct, which is a claim about wording, not about floor control, latency, or whether the action the agent narrated actually happened. Voice regressions hide in exactly the gap between what the transcript shows and what the audio contains.
Measure the call: talk-over, time-to-yield, say-do
Hotato scores a recording on separate dimensions and never blends them into one number. Turn-taking timing is deterministic and model-free: talk-over duration, response gap, and time-to-yield are computed from two-channel audio at the millisecond, so the same recording produces the same verdict every run. Speech and transcript checks read latency, required phrases, and PII exposure. Outcome and policy checks read the trace spans and your system of record, so "the refund went through," spoken on the call, is graded against whether a refund exists, not against how confident the agent sounded. It measures timing and say-do, not intent.
Point it at one call and read the report:
hotato autopsy ./call.wav # zero-install first run on bundled calls hotato autopsy ./call.wav # score one recording, ranked moments
The loop: observe, catch, pin, test, prove
A single report is a diagnosis. The durable value is closing the loop so a voice failure you found once becomes a check that runs on every change. Hotato runs the same loop end to end.
- Observe. Pull recent calls from your stack and render them into one self-contained dashboard:
hotato sweep --since 7d. No account, offline by default. - Catch. Rank the moments in a recording and label the one that failed, for example a missed barge-in or a talk-over:
hotato investigate label. - Pin. Freeze that labeled moment into a content-addressed contract with an explicit expectation:
hotato contract create --expect yield. - Test. Re-measure the captured evidence under the pinned policy:
hotato contract verify contracts/, exit 0/1/2. - Prove. Emit a claim-scoped proof for the catch you fixed:
hotato prove --contracts contracts/.
One command drives the whole loop over a folder of recordings against your pinned fixtures:
hotato loop ./recordings --fixtures tests/hotato
What it grades
Turn-taking timing and say-do are measured from audio channels and trace evidence. The advisory model judge runs on a separate lane and never stands in for a deterministic measurement, so a model opinion cannot flip a timing gate.
Read the scorecard, not one number
A run yields a per-dimension scorecard, never a single blended score, because the dimensions fail for different reasons and want different fixes. Outcome asks whether the task succeeded. Policy asks whether it followed the rules, including required disclosures and PII handling. Conversation holds the turn-taking timing. Speech covers latency, phrasing, and audio delivery. Reliability reports how often the agent holds up across repeated runs. A red Conversation score with a green Outcome tells you the agent did the right thing while stepping on the caller, which is a floor-control fix, not a prompt rewrite. Collapsing that into one figure would hide the very signal you came for.
Simulate before you ship
You do not need a live caller to test a change. Hotato renders a scenario into deterministic simulated conversations, seeded from a hash, so a talk-over or dead-air persona replays identically for everyone who runs it:
hotato simulate barge-in-missed --out ./sim hotato test run conversation-test.yaml --agent my-agent
The simulated caller drives your agent, the same scorer grades the result, and the run yields a per-dimension scorecard you can gate on. Every simulated artifact is stamped origin=simulated so it is never confused with a captured call.
Wire it into CI
Once a catch is pinned, the contract belongs in continuous integration next to your other tests, so a prompt edit or a model swap that reintroduces the talk-over turns the build red before it reaches a caller. See CI for AI agents for the gate, and docs: CI for the workflow. Capture from your stack is per-provider: docs: capture.
Start here
Run the loop on one recording, then keep the catch:
pip install hotato hotato autopsy ./call.wav
Docs: getting started, turn-taking timing. Repo: github.com/attenlabs/hotato.