Twelve maintainer-recorded live probe calls, scored against one default configuration
Twelve calls on one stock assistant, factory settings, recorded 2026-07-06 and scored offline with Hotato.
Scored with Hotato 1.0.0 · artifact captured 2026-07-09 · scripted against corpus revision 33985d1, tagged v0.8.0 · reproduce: score the corpus/probe-defaults clips with hotato
Maintainer-recorded scripted calls: one human caller worked from a written script against a synthetic voice assistant, not customer production failures. The labels (what the script called for) come from that script, not a model. Hotato measures timing from the two audio channels only. A field note, named throughout this page, is the operator’s own per-moment observation of what happened on the call, logged separately from that timing measurement; where the two disagree, this page says so, and the measurement is what gets scored. Clips below are MIT licensed, in the repository.
Summary: sixteen scored moments, one page
| Moment | What happened | Scored result | Jump to |
|---|---|---|---|
| Backchannel (“mhm”, “yeah”), 4 moments | Three of four soft acknowledgements stop the agent as fast as a real interruption, 0.32-0.37s; the fourth, a quieter mhm, is correctly held. | Fail · 3 of 4 false yields | Summary chart → |
| “Stop.” command | Agent talks 1.26s past the command to its own sentence-end pause; the yield is scored at 1.46s, 0.46s past the 1.0s bound. | Fail · 1.46s vs 1.0s bound | Full timeline → |
| Quiet interrupt (half-volume), moment 10a | Never crosses the default energy threshold; agent keeps talking through the whole attempt. | Fail · missed entirely | Detail → |
| Quiet interrupt, louder retry, moment 10b | Same caller, same call, retried louder: crosses the threshold and yields. | Pass · 0.59s talk-over | Summary chart → |
| Thinking pause (4s) | Caller falls silent mid-sentence intending to keep the floor; agent reads the silence as turn-end and starts 3.44s in. | Not scored pass/fail · latency-only | Detail → |
| Correction retry | First correction attempt not picked up, dead air until the caller re-prompts; the retry lands and yields in 1.16s. | Fail · needed a retry | Detail → |
| Hard interrupts, double-talk, greeting overlap (7 moments) | Clean floor holds or a genuine interruption is caught, 0.00-0.40s. | Pass · 7 of 7 | Summary chart → |
| Call 11, clean floor | 82.5 seconds, no caller-during-agent onset anywhere: the baseline every measurement above departs from. | Not scorable · nothing to score | Funnel verdict → |
Every number above is a timing measurement, not an accuracy percentage. Full derivation per moment: manifest.json in the corpus directory linked above.
The measured results, one moment at a time
Bounds, applied uniformly to every should-yield moment: 1.0 second to yield, 1.0 second of talk-over. Sixteen scored moments: nine pass, six fail, one latency-only reading. Call 11, 82.5 seconds with no caller-during-agent onset anywhere, is the clean-floor baseline every measurement below departs from: nothing to score, not counted in the sixteen.
Twelve of the sixteen scored moments
Nine correct handles, three false yields. The default is not broken all the time: most interruptions land in about a third of a second, but three soft backchannels hit that same reflex speed, and no threshold tells them apart from a real interruption.
Two of the twelve, in audio
The same reflex speed, one correct and one not: a genuine interruption caught in 0.35 seconds, a backchannel mistaken for one in 0.34.
The moments that expose the funnel
Four of sixteen, each a different shape of failure: a command ignored, a quiet interrupt that never trips the threshold, a thinking pause mistaken for a turn ending, and a correction that needed a retry through dead air.
Here is the whole “Stop.” timeline it is scored from, zeroed at the caller’s command:
The funnel verdict
The battery fails on both axes at once: it misses a genuine interruption (moment 10a) and false-yields on backchannels (moments 4 and 5) in the same run. Run against the battery, hotato plan reports exactly that, and refuses to propose a threshold change:
hotato plan [stack] finding=threshold_funnel decision=do_not_tune_single_threshold config_only_safe=false production_apply=false (approval: manual) hypothesis: The battery missed a genuine interruption and also stopped for a backchannel. One sensitivity threshold cannot satisfy both: raising it to hold through backchannels drops real interruptions, lowering it to catch interruptions yields to backchannels. The failure class is discrimination, not calibration. recommended fix class: engagement-control platform mutation: performed=false (hotato plan is read-only)
Output from the battery, trimmed to the decision and hypothesis. All sixteen scored moments are measured above; the full untrimmed JSON, with per-event evidence, and the manifest (every onset derivation, the two measurement/field-note disagreements named above) are in the repository: the corpus on GitHub →
One assistant, one configuration, one date, one scripted caller: enough to demonstrate the threshold funnel, not enough to rank platforms.