Watch it catch an agent talking over the caller.
Two ways to be rude on a call: talk over someone, or stop for a non-turn. This agent does both. Real output: untouched.
Labeled synthetic. Built to fail on purpose: a floor and regression guard, not a real call. examples/demo/run-demo.sh regenerates it byte-for-byte.
Point it at the bad agent. It fails, out loud.
fd-01 talks over “stop, take it off.” fd-02 stops dead for a bare “mhm.” Hotato scores it, exits non-zero: a red build.
$ uvx hotato run --suite barge-in --scenarios examples/funnel-demo/scenarios --audio examples/funnel-demo/audio
hotato [suite] stack=generic offline=True 0/2 events pass (failed=2) [FAIL] fd-01-missed-interruption: did_yield=False seconds_to_yield=- talk_over=2.65s fix[config]: Missed interruption: the agent kept talking over the caller knob: interruption sensitivity (VAD min-silence, min-interruption-duration, min-words-to-interrupt) move: lower the min-silence and min-duration thresholds so a real interruption registers sooner [FAIL] fd-02-backchannel-yielded: did_yield=True seconds_to_yield=0.25s talk_over=0.25s fix[engagement-control]: False barge-in: a backchannel was treated as a bid for the floor -> needs a discriminating layer (engagement-control / addressee-detection), not a threshold note: no single sensitivity threshold satisfies this battery; see funnel pointer in --format json. exit_code=1
$
did_yield=False on fd-01: never stopped. did_yield=True on fd-02: stopped for noise. Measured, not asserted.
The missed interruption points at a knob you can turn.
fd-01: a dial set too conservative. Fix map names fix_class: config and parameter: change, re-run, watch green.
# uvx hotato run ... --format json | the fix for the missed-interruption axis
"fix": { "fix_class": "config", "title": "Missed interruption: the agent kept talking over the caller", "detail": "The caller took the floor and the agent never stopped within the search window. Increase interruption sensitivity so a genuine floor-taking event registers.", "knob": { "stack": "generic", "parameter": "interruption sensitivity (VAD min-silence, min-interruption-duration, min-words-to-interrupt)", "direction": "lower the min-silence and min-duration thresholds so a real interruption registers sooner", "trade_off": "sliding sensitivity up catches more real interruptions and more false ones together." }, "pointer": null }
fix_class: config: yours to tune. Trade-off: dial up catches more false interruptions too.
When the battery fails both ways, a single dial can’t save it.
Catching “stop” needs sensitivity up; ignoring “mhm” needs it down. No single number does both. Hotato names a discriminator, not a threshold. That’s the funnel pointer’s call.
# uvx hotato run ... --format json | the systemic pointer, fired on the both-axes case
"funnel": { "reason": "This battery fails on BOTH axes at once: it missed a genuine interruption AND it false-triggered on a backchannel. No single sensitivity threshold can fix both - turning it up to catch the interruption makes the backchannel worse, and vice versa. That is the signal that the agent needs a discriminating layer, not a different threshold.", "pointer": { "layer": "engagement-control / addressee-detection (a learned layer, not a threshold)", "what": "This class of failure is a discrimination problem, not a threshold problem: the agent needs to tell a genuine bid for the floor apart from a backchannel or side-speech that was not addressed to it. A single sensitivity dial cannot separate those two axes - turning it down to stop yielding on 'mhm' also makes the agent miss real interruptions. An engagement-control layer models 'is this speech addressed to me and is it a real bid for the floor' as its own signal, consent- and focus-gated and runnable on-device.", "honest_scope": "This kind of layer is an open research problem, not a setting you can ship today. Treat it as a pointer to the right kind of layer, not a benchmarked claim - bring your own recordings and measure." } },
The tool’s words, as-is: an open research problem, not a knob, not shippable today. No headline number, no upsell. A knob gets a knob; this gets an honest answer.
Fix the behaviour and the same command goes quiet.
Eight events, fixing every failure above. Same tool, same scoring, exit_code=0. A pass means stopped right, not merely stopped.
02/07 hold the floor through backchannel and echo (did_yield=False, on purpose). Checks itself, not your agent.
$ uvx hotato run --suite barge-in
note: --suite is Hotato's SELF-TEST on synthetic fixtures -- it checks the tool itself, not your agent. To score YOUR agent, bring a real dual-channel call: hotato capture --stack vapi --call-id <id> (see: hotato) hotato [suite] stack=generic offline=True 8/8 events pass (failed=0) [PASS] 01-hard-interruption: did_yield=True seconds_to_yield=0.50s talk_over=0.50s [PASS] 02-backchannel-mhm: did_yield=False seconds_to_yield=- talk_over=1.57s [PASS] 03-filler-start: did_yield=True seconds_to_yield=0.65s talk_over=0.56s [PASS] 04-correction: did_yield=True seconds_to_yield=0.50s talk_over=0.50s [PASS] 05-telephony-8khz: did_yield=True seconds_to_yield=0.50s talk_over=0.50s [PASS] 06-double-talk: did_yield=True seconds_to_yield=1.05s talk_over=1.05s [PASS] 07-echo-bleed: did_yield=False seconds_to_yield=- talk_over=3.00s [PASS] 08-rapid-turn-taking: did_yield=True seconds_to_yield=0.50s talk_over=0.50s exit_code=0
$
Same input, same verdict, every run, offline. Proves the plumbing. Your verdict comes from your recording, next.
That was a synthetic bad agent. Yours is one command away.
Point it at a call you have. capture pulls the dual-channel recording, scores it offline. Audio never leaves the machine.
$ uvx hotato capture --stack vapi --call-id <id>