Hotato catches the moments your voice agent talks over a caller, or grabs a turn nobody offered.
barge-in · talk-over · backchannel · endpointing, scored offline on your own recordings.
$ uvx hotato capture --stack vapi --call-id <id>
No install, no account. With your Vapi key set, capture pulls that one recording, scores it offline, and leaves your audio on your box.
$ uvx hotato run --suite barge-in # built-in self-test (synthetic floor) hotato [suite] stack=generic offline=True 8/8 events pass (failed=0) [PASS] 01-hard-interruption: did_yield=True seconds_to_yield=0.50s talk_over=0.50s [PASS] 02-backchannel-mhm: did_yield=False seconds_to_yield=- talk_over=1.57s ... exit_code=0
The zero-input self-test. It runs synthetic batteries to prove the plumbing and guard against regressions. A verdict about your agent comes from capture scoring your own call, above.
The caller cuts in. The agent keeps talking. A real interruption, ignored, and people hang up.
The caller says “mhm” to show they are listening. The agent stops mid-sentence and blurts.
Your agent has opinions. So does the caller. Just one of them should be talking.
the whole product, honestly
One command replays what went wrong, says which kind of wrong, and hands you the knob.
Each battery is a moment the caller speaks over the agent. Hotato measures what the agent did with it.
A pass means stopped for the right reason. Hotato tells a genuine interruption from a “mhm” the agent should have talked through.
Most failures map to one setting: a threshold, a buffer, an endpoint window. Hotato names it. You change it, re-run, watch the battery flip.
Turn-taking bugs are usually one dial set a notch off. Hotato ships the map from failure to setting, so you turn knobs instead of guessing.
| When Hotato flags | the knob it points at |
|---|---|
| Talks over the callerignored a real interruption · clutch | lower the barge-in threshold, or enable interrupt-on-speech |
| Stops for “mhm”yielded to a backchannel · fumble | raise VAD sensitivity, or add a backchannel hold-off |
| Slow to actually stophigh talk_over after yield | shorten the output buffer, or flush the interrupt faster |
| Cuts the caller off earlyendpoints before they are done | lengthen the endpoint silence window |
| Fails on both axes at oncemisses a real interruption and false-triggers on a backchannel | one threshold cannot win both directions. This one needs a discriminating layer, not a different number. |
Some calls fail both ways at once: the agent misses a real interruption and stops for a backchannel. Raise the threshold and one gets better while the other gets worse. Lower it and they swap.
Telling a real interruption from a backchannel takes a learned engagement-control layer, not a config knob.
Attention Labs builds and licenses one, consent and focus gated. Hotato just flags the case. Hit this wall? Get in touch →
Point it at any dual-channel recording from Vapi, Retell, Bland, LiveKit, or Twilio. It scores the audio timing itself, on your hardware.
Read every line, fork it, vendor it into your own harness.
The scoring path makes no network call. Your recordings stay put.
Same recording, same verdict. Verify the timing frame by frame.
One command over any dual-channel WAV, whatever produced it.
One tool, hotato.run, lets an assistant score a recording, read back the failing batteries, and propose the knob. Machine-readable in and out.
# one tool, machine-readable in and out tool hotato.run suite="barge-in" recording="call.wav" → { passed: 8, failed: 0, failures: [] }
A prompt tweak or config change can quietly break how your agent handles interruptions. Hotato exits non-zero on a failing battery, so CI catches it like any unit test.
# fails the job on a turn-taking regression uvx hotato run --suite barge-in \ --fixtures ./calls --fail-under all echo $? # 0 green, 1 red
What it reads, what it publishes, and how it fits the stack you already ship on.
It reads timing and energy: who held the floor, and for how long. That audio-only view is what keeps it deterministic and local.
Energy marks who was talking, not who was right. Words and speaker identity are a separate layer.
You get reproducible timing you can verify frame by frame. Self-reported F1 numbers come from private benchmarks and do not agree.
An independent study, Testing the Testers (arXiv:2511.04133), found a widely-cited tool ranked well below its own reported score.
They set a floor and guard against regressions. Your real verdict comes from scoring your own call recordings.
Point it at any dual-channel recording you already have: Vapi, Retell, Bland, LiveKit, or Twilio. It scores offline and exits non-zero in CI.
Yes. Every threshold is a published parameter and every frame is inspectable. Run it with no network and get the same answer we do. How it works →
Nothing. MIT open core, zero-install through uvx, no account and no phone-home. Fork it and vendor it if you like.
Catch the turn, then pass it, fast and clean. Drop it and you missed a real interruption. Clutch it and you are talking over someone who just said “wait.”
Hotato watches the hand-off, times the toss, and flags when your agent will not let go.
Yes, it is a potato. It has made peace with that.
One command, your own recordings, an answer you can reproduce.
$ uvx hotato capture --stack vapi --call-id <id>