What Hotato validates.
A turn handoff is not one number, so Hotato never reports a single accuracy percentage. Instead it is validated on three separate jobs, each measured on its own terms. If you are deciding whether to trust Hotato, judge these three one at a time. Every block below is verbatim output from hotato 0.5.0, offline.
Timing reproducibility
Does the same recording produce the same timing measurements every run, on any machine? No learned weights, no sampling, no RNG. The energy VAD and the reference framing are deterministic, so the numbers are byte-stable. A red build means the audio changed, not that the scorer drifted.
$ hotato run --stereo 01-hard-interruption.example.wav --expect yield 1/1 events pass (failed=0) [PASS] did_yield=True seconds_to_yield=0.51s talk_over=0.51s $ hotato run --stereo 01-hard-interruption.example.wav --expect yield [PASS] did_yield=True seconds_to_yield=0.51s talk_over=0.51s # identical across runs: the property a regression test needs
did_yield, seconds_to_yield, and talk_over_sec per event, plus the exact thresholds and the frame grid behind them. What it does not establish: that 0.51s is the "true" latency in some absolute sense, only that the measurement is stable and re-derivable by hand.
Candidate-discovery usefulness
Does scan surface the moments a human reviewer would want to look at, ranked by salience? Each candidate is a timing fact, an overlap or a gap, not a verdict and not an intent label. The bar is recall of human-notable moments at a workable count, not precision against a ground-truth intent (there is none at scan time, by design).
$ hotato scan --stereo 02-backchannel-mhm.example.wav --top 5 hotato scan: 02-backchannel-mhm.example.wav (6.0s, 3 candidate moments) Candidates are timing events. You decide the expected behavior. [ 1] t=2.09s overlap_while_agent_talking overlap=1.58s agent did not go silent [ 2] t=3.19s overlap_while_agent_talking overlap=1.07s agent did not go silent [ 3] t=4.29s overlap_while_agent_talking overlap=0.56s agent did not go silent
A ranked list of candidate moments as timing facts; you label intent with hotato fixture create. What it does not establish: that every candidate is a real bug, or that a quiet region is guaranteed clean. Scan widens the net; you make the call.
Fixture pass-fail agreement
Once you label a moment's expected behavior, does the PASS/FAIL verdict agree with that label on the audio? Both labelled failures below are caught, and the battery-level note refuses to name one threshold when a missed interruption and a false stop fail in the same run. That refusal is part of the validated behavior.
$ hotato demo --no-open --format text 0/2 events pass (failed=2) [FAIL] fd-01-missed-interruption: did_yield=False talk_over=0.25s fix[config]: the agent kept talking over the caller [FAIL] fd-02-backchannel-yielded: did_yield=True seconds_to_yield=0.34s talk_over=0.32s fix[engagement-control]: a backchannel was treated as a bid for the floor note: no single sensitivity threshold satisfies this battery exit_code=1
Per fixture: the verdict, the measured signals behind it, and the named fix class when the failure maps cleanly to a config family. What it does not establish: that the label was correct (you own it), or that a passing fixture means the agent is good in general.
A hard boundary, drawn on purpose.
These are out of scope on purpose. Naming them is how a Hotato number stays honest.
No semantic intent
Hotato measures timing. It does not know whether a caller sound meant "stop" or "mhm, go on." You supply that as a label.
No task success
Whether the call booked the appointment or resolved the ticket is out of scope. Use a QA platform for that.
No satisfaction or sentiment
Hotato has no opinion on tone or CSAT.
No root cause
A slow yield can be TTS buffering, transport, or VAD. diagnose names a likely layer and stays unknown_root_cause when one recording cannot separate them.
No vendor ranking
A provider-default example demonstrates the threshold funnel on one assistant, one config, one date. It is not a benchmark of the vendor.
No single accuracy score
There is no headline percentage anywhere in Hotato, on purpose. The three jobs above are the whole claim.