hotato
GitHub
Three jobs, no single accuracy score

What Hotato validates.

Hotato reports three separate jobs, each evaluated on its own: timing reproducibility, candidate-discovery usefulness, and fixture pass-fail agreement.

Verbatim output below, generated from Hotato 1.0.0, captured 2026-07-09, run on Hotato's own bundled corpus and demo fixtures, not customer calls.

one corpus · three commands
hotato's bundled corpus 01 hotato run timing reproducibility 02 hotato scan candidate discovery 03 hotato demo fixture pass-fail Same corpus in. Three unrelated questions out.
Read this before any status on this site

Five levels of evidence. Never call the weakest one a verdict.

Every status that judges whether a flagged behavior is real names one of these levels and claims only what it shows. Job 1's reproducibility check names none of them on purpose, since it proves the measurement is stable, not that a flagged behavior is real. A higher level just means a narrower claim.

Level 1 Candidate Hotato found a timing event worth human review. Not a bug yet, not a verdict. Level 2 Human-labeled failure A reviewer confirmed the recording violated an explicit yield-or-hold expectation. Level 3 Stored-evidence check The historical audio still produces the expected result under the pinned policy and scorer. Level 4 Fresh-recapture comparison A newly captured call passed the same contract; no submitted paired guard regressed.
Level 5 External proof An outside team confirmed Hotato caught a regression, or a fresh recapture passed after their change.
Three jobs, evaluated separately

Timing reproducibility, candidate discovery, fixture agreement.

Job 1 · the question

Timing reproducibilityOutside the levels

Does the same recording produce the same timing measurements every run, on a fixed hotato version? Yes: no learned weights, no sampling, no RNG. With version, audio, and scoring config pinned, results are byte-stable in the verified environments (CI on Linux x86_64), and a changed result means a pinned input changed, not scorer drift.

What is reported

did_yield, seconds_to_yield, and talk_over_sec per event, plus the exact thresholds and frame grid behind them: proof the measurement is stable and re-derivable by hand, not that 0.51s is the true latency.

Job 2 · the question

Candidate-discovery usefulnessLevel 1

Does scan surface the moments a human reviewer would want to look at, ranked by salience? Each candidate is a timing fact, an overlap or a gap, never a verdict or intent label. The bar is recall of notable moments at a workable count, not precision against ground truth, since none exists at scan time by design.

hotato scan --stereo 02-backchannel-mhm.example.wav --top 5
$ hotato scan --stereo 02-backchannel-mhm.example.wav --top 5
hotato scan: 02-backchannel-mhm.example.wav  (6.0s, 3 candidate moments)
Candidates are timing events. You decide the expected behavior.
  [ 1] t=2.09s  overlap_while_agent_talking  overlap=1.58s  agent did not go silent
  [ 2] t=3.19s  overlap_while_agent_talking  overlap=1.07s  agent did not go silent
  [ 3] t=4.29s  overlap_while_agent_talking  overlap=0.56s  agent did not go silent
ranked by overlap size
t 0s 1s 2s 3s 4s 5s 6s [1] t=2.09s overlap=1.58s [2] t=3.19s overlap=1.07s [3] t=4.29s overlap=0.56s
Shareable card: Hotato found a candidate talk-over of 1.58 seconds of overlap while the agent was talking, at t equals 2.09 seconds in the recording. Footer reads: Hotato reports timing candidates, not intent.
Every candidate exports as a card.
What is reported

A ranked list of candidate moments as timing facts: a quiet region is not proven clean, and intent gets labeled separately with hotato fixture create.

Job 3 · the question

Fixture pass-fail agreementLevel 3

Once you label a moment's expected behavior, does the PASS/FAIL verdict agree with that label on the audio? Both failures below are caught. This is a stored-evidence check: it re-scores frozen, labeled recordings under a pinned policy, so a pass means evidence and scorer still agree, not that a live agent behaves the same way today. No single sensitivity threshold fixes both: a missed interruption and a false stop need opposite repairs.

hotato demo --no-open --format text
$ hotato demo --no-open --format text
  0/2 events pass  (failed=2)
  [FAIL] fd-01-missed-interruption: did_yield=False talk_over=0.25s
         fix[config]: the agent kept talking over the caller
  [FAIL] fd-02-backchannel-yielded: did_yield=True t=1.28s trailing_silence=0.46s agent_stop_no_caller
         fix[engagement-control]: a backchannel was treated as a bid for the floor
  note: no single sensitivity threshold satisfies this battery
  exit_code=1
two fixtures, two different fixes0/2 pass
fd-01-missed-interruption FAIL · kept talking talk_over=0.25s fd-02-backchannel-yielded FAIL · stopped too soon silence=0.46s 0.0s 0.6s
Shareable card: Hotato found a candidate false-stop of 0.46 seconds of silence after the agent stopped, with no caller nearby, at t equals 1.28 seconds in the recording. Footer reads: Hotato reports timing candidates, not intent.
Scan finds this as a candidate. Labeled fd-02-backchannel-yielded, it becomes the Level 3 verdict shown above.
What is reported

Per fixture: the verdict, the measured signals behind it, and a named fix class when the failure maps cleanly to a config family, not a claim the label was correct or that a passing fixture holds on calls it has not heard.

What we do not claim

A hard boundary, drawn on purpose.

One line is measured today: timing and turn-taking. The five below it are boundaries Hotato refuses to cross, from semantic intent up to vendor ranking, each one the scorer does not have the input to decide.

measured
timing / turn-taking: who held the floor, and for how long, to the frame.
not measured
semantic intent: whether a caller sound meant "stop" or "mm-hm, go on." You supply that as a label.
not measured
task success: whether the call booked the appointment or resolved the ticket. A QA platform's job.
not measured
satisfaction / sentiment: tone and CSAT carry no signal here.
named, not proven
root cause: a slow yield can be TTS buffering, transport, or VAD. diagnose names a likely layer and stays unknown_root_cause when one recording cannot separate them.
out of scope
vendor ranking: a provider-default example is one assistant, one config, one date, not a benchmark.
More evidence

The rest of the evidence, one link each.

Live probe calls

Twelve probe calls

One stock configuration, scored offline, with full timelines and audio.

 Demo report

A generated artifact

Two events scored offline, zero pass: what a Hotato report looks like.

 Trust and limits

The input contract

Ten input conditions, each mapped to score, caveat, or refuse.

 Corpus

Every clip, licensed

The consent and licensing ledger, MIT and CC BY 4.0 only.