How do I test voice agent interruptions?

Score a recorded two-channel call and read three numbers: did_yield, seconds_to_yield, and talk_over_sec. Hotato measures them from audio energy alone, with no model in the path. One command runs the check on a recording: hotato run --stereo call.wav --onset 12.4 --expect yield . Each number is re-derivable by hand from the waveform.

The agent talks over the caller

talk_over_sec measures it: how long both channels carried energy at once. The overlap shows up as a window on the dual-channel timeline, in milliseconds. Each channel already belongs to one party, so the measurement needs no words and no transcript. The scorer flags the moment as a candidate; you label whether it was an interruption, a backchannel, or a clean handoff.

The agent interrupts customers mid-sentence

did_yield answers it: once the caller started, did the agent stop? seconds_to_yield times the stop, from the caller’s onset to the agent’s stop. Run the recording with --expect yield and a call where the agent kept talking comes back as a fail, with the onset marker, the overlap window, and the yield line drawn on the timeline.

The agent ignores real barge-ins

Mark the caller’s onset and check did_yield. When the caller takes the floor and the agent keeps going, did_yield reads false and talk_over_sec carries the length of the collision. Energy is not intent: a flagged moment starts as a question, and it becomes a finding by climbing from a candidate you label to a card showing both channels and the overlap window in milliseconds.

Home / Incident

The agent kept speaking 1,810 ms after the caller interrupted.

One recording, one command, one incident: a measured moment where the call went wrong, with a timestamp and an id you can hand to someone who does not trust you. This is what hotato autopsy writes for the example in the repo, and for yours. How the number is validated →

the command that produced everything below

$ hotato autopsy examples/autopsy/audio/autopsy-01-barge-in-say-do.example.wav

incident apx-cc33f46fad58#2 · autopsy-01-barge-in-say-do.example.wav

CRITICAL BARGE-IN apx-cc33f46fad58#2

The agent kept speaking for 1,810 ms after the caller interrupted.

observed: t=2.99s into a 12.0s call, 2 channels, stereo
measured: overlap 1.81 s; the agent did not go silent within the 3.0 s limit
evidence: dual-channel audio: caller and agent on separate channels
pin: apx-cc33f46fad58, this call's id; #1..#2

The same fields land in the self-contained HTML report each run writes to hotato-output/.

What the run printed.

Every line names an incident kind, the second it happened, and the number that made it one.

hotato autopsy: autopsy-01-barge-in-say-do.example.wav  (12.0s, 2 channels, stereo)
  [CRITICAL] BARGE-IN       t=2.99s  overlap=1.81s  agent did not go silent within 3.0s
      the caller took the floor and the agent kept talking over them for 1.81 s without going quiet within 3.0 s
  1 incident: 1 critical, 0 warning
  report: hotato-output/autopsy-apx-cc33f46fad58.html
  pin: apx-cc33f46fad58  (incidents apx-cc33f46fad58#1..#1)

Generated from Hotato 1.20.0 on the example recording in the repo. Timing comes from audio energy over time, one channel per party, so each number recomputes from the waveform. What every verdict rests on →

Reported in the wild

The same moment, filed by other people.

The builders living this are already asking for the missing layer in public. Every link below is someone else’s report, on their own stack, of the failure this page measures.

A LiveKit community thread on 2026-07-09 asks how teams profile per-turn latency: user-stop through playback, every tool call in between, p50 and p95 across runs (LiveKit community: per-turn latency).
A separate report describes instrumenting ASR, LLM, and TTS, watching the services total under one second, and still hearing a three to four second gap before the agent speaks (LiveKit community: services 1s, reply 3-4s).
One HN comment describes a barged-in appointment confirmation that still triggered the downstream booking pipeline: the caller cut in to stop it, and the booking went through anyway (Hacker News: a barge-in that booked anyway).
One production report estimates an agent interrupting itself through its own echo in about ten percent of calls, where lowering the sensitivity risks the agent never yielding to the caller (Retell community: an agent echoing itself).
One developer reports 1,242 calls with zero conversions, blames a three to four second response gap, and wants sub-second replies to recover the funnel (Retell community: no conversions). Another reports prospects hanging up during a three second silence before the agent speaks (Retell community: opening silence).

The turn-taking is the product surface, and right now it is the untested one.

And the other half: what the agent said it did.

One commenter states the constraint: judges “can’t reliably judge LLM outputs without access to the environment where those actions are executed” (Hacker News: judging without the environment).

One reported case has a refund tool firing when the caller only asked for order status; the same comment reports a verification flow that never checked the caller’s phone against the account (Hacker News: a refund tool firing on a status check).
A maintainer issue records interrupted tool results dropped from the history, producing duplicate reservations and duplicate rows: the agent said it once and the backend did it twice (livekit/agents: tool results dropped from history).
A related class shows up where the trace goes quiet: a tool runs, the reply stalls, and the trace omits the cause, so even the log cannot tell you what happened (openai-agents-python: a stalled reply, no trace of the cause).
Passing a prompt test does not establish that a caller was authenticated (Hacker News: a prompt test cannot establish authentication), and releasing sensitive data cannot depend on the model choosing to comply with an instruction (Hacker News: data release left to model compliance).

Why it lands hereWhen the agent’s word is the only record, a wrong action and a right one look identical. The incident above is the same moment with a timestamp, an id and a number on it.

Send it. Then pin it.

Both start from the printed id, apx-cc33f46fad58.

01 / SHARE
One file, whoever owns the fix

The run writes this report as a self-contained HTML file. Attach it to the ticket and the timeline travels with it.
02 / PIN
The incident becomes a check

hotato pin apx-cc33f46fad58#2 writes a portable .hotato bundle: the audio, the timing evidence, the check.
03 / PROVE
CI reads the exit code

hotato prove --contracts contracts/ re-runs the stored checks and fails closed: 0 pass, 1 fail, 2 could not tell.

How the timing is measured → · how say-do is graded → · full walkthrough →

The payoff

The transcript passed. The call failed.

Every voice team has shipped this call. The transcript reads clean, the tool log says success, the dashboard is green, and the customer got the wrong outcome. Here is the walk from a call to a gate.

You start with one recording:

console

$ hotato investigate ./call.wav
hotato investigate [run 1]: call.wav
  input health: eligible for scan
  verdict path: eligible (a labeled event here can carry a yield/hold verdict)
  most likely failure (top-ranked candidate):
    [1] t=7.63s agent_stop_no_caller  trailing_silence_sec=0.37, caller_proximity_sec=0.5
  next: label it (use --expect hold instead if the agent was right to keep talking):
    hotato investigate label '.hotato/investigate-state.json#1' --expect yield

The scan ranks the candidate moments, marks the top one, and prints the command that comes next. You label that moment as a yield the agent owed or a hold it was right to keep, and that writes a contract named by its evidence hash:

console

$ hotato investigate label '.hotato/investigate-state.json#1' --expect yield
created hotato contract: call-8s-yield
  dir:      contracts/call-8s-yield.hotato
  expect:   yield
  passed:   False
  measured: did_yield=False seconds_to_yield=n/a talk_over=0.00s
next:
  hotato contract verify contracts

Then the gate scores the stored evidence, deterministically, in CI:

console

$ hotato contract verify contracts
hotato contract verify: contracts (1 contract)
  [FAIL] call-8s-yield (expect yield): did_yield=False seconds_to_yield=n/a talk_over=0.00s | integrity: intact
  0/1 contracts pass; exit_code=1
  These contracts pin known failures. Each stays red until you fix the agent and recapture the call, the same way a snapshot test stays red until you update the snapshot.
  Path to green: fix the agent, then recapture with `hotato drive <bundle>` (vapi/twilio), or the manual path in docs/RECAPTURE.md.

That exit 1 is the part worth slowing down on. A committed contract is a pinned bad call, and it is meant to stay red. The frozen audio never changes, so the gate goes green only after you fix the agent and recapture the call, the same way a snapshot test stays red until you update the snapshot. The red names the contract, its expectation, and the numbers. Nothing recaptures automatically: the recorded behavior is the source of truth in CI, and updating it is a deliberate, local, reviewed act.

Say-do is what turns a green transcript into a red gate. A tool_call assertion counts only the spans in your ingested trace, so an agent’s words claiming a tool ran do not count. An outcome assertion combines those spans with a query against your system of record, so a refund the agent announced is checked against whether a refund row exists.

hotato prove folds the checks you ran into one release proof, green only if all pass. Wire the gate into CI →

Run it on your recording.

One call in, every measured incident out, with the report already written.

$ hotato autopsy ./call.wav

pip install hotato is the whole setup. Get started →

The agent kept speaking 1,810 ms after the caller interrupted.

What the run printed.

The same moment, filed by other people.

And the other half: what the agent said it did.

Send it. Then pin it.

One file, whoever owns the fix

The incident becomes a check

CI reads the exit code

The transcript passed. The call failed.

Run it on your recording.