Every number here re-derives from one command.
On 13 human-labeled clips from a published corpus, hotato's measure of when the caller takes the floor lands a median 10 ms from the label. The clips, their labels and the scorer ship in one MIT repository, so the evidence behind a verdict outlives the tool that wrote it.
- 20 msmedian caller-onset error vs human labels
- 13AMI meeting clips, human-labeled
- 80 msmedian time-to-yield error
- CC BY 4.0published corpus, attributed
Every clip is in the repo.
The turn-taking timing a transcript never sees, scored against a published corpus of human-labeled meeting audio. Run the same command on the same clip and read the same millisecond.
benchmark-report/measurement-error.json, re-derived by
PYTHONPATH=src python3 -m hotato.benchmark.Table view — every fixture’s error, in milliseconds
| fixture | caller onset | time to yield |
|---|---|---|
| ami-en2002b-bc-0859 | 20 | |
| ami-en2002b-bc-1049 | 20 | |
| ami-en2002b-bc-1114 | 20 | |
| ami-en2002b-bc-1416 | 100 | |
| ami-en2002b-take-0149 | 0 | 0 |
| ami-en2002b-take-0772 | 390 | 30 |
| ami-en2002b-take-0913 | 10 | 80 |
| ami-en2002b-take-0930 | 30 | 2270 |
| ami-en2002b-take-1069 | 10 | 780 |
| ami-es2002a-bc-0526 | 0 | |
| ami-es2002a-bc-0687 | 10 | |
| ami-es2002a-bc-1049 | 0 | |
| ami-es2002a-take-0677 | 1430 | 80 |
| What is measured | clips | median error | range |
|---|---|---|---|
| Caller onset, the moment the caller takes the floor | 13 | 20 ms | 0 to 1430 ms |
| Time to yield, how fast the agent stops talking | 6 | 80 ms | 0 to 2270 ms |
Those are the errors when the scorer and the label agree that something happened. The other half of the result is where they disagree at all: of the 13 clips, 7 are labelled as a turn the agent should have held through, and the scorer reports a yield on 6 of them. It errs toward calling a yield rather than toward missing one, which is the direction that matters when a measured overlap is a candidate for a human to label rather than a verdict.
should_yield fixture was measured as a yield and none was missed;
6 of the 7 should_not_yield fixtures were also measured as
yields. On this corpus the scorer errs toward reporting a yield rather than toward missing one. Source:
benchmark-report/measurement-error.json, re-derived by
PYTHONPATH=src python3 -m hotato.benchmark.hotato check corpus/real/audio/ami-en2002b-take-0930.example.wav hotato autopsy: ami-en2002b-take-0930.example.wav (9.0s, 2 channels, stereo) [WARNING] BARGE-IN t=4.37s overlap=0.69s agent went silent after 1.04s the caller took the floor and the agent kept talking over them for 0.69 s before going quiet [WARNING] TALK-OVER t=4.57s overlap=0.69s caller kept talking 2.67s the agent started talking while the caller still had the floor, overlapping them for 0.69 s [WARNING] TALK-OVER t=3.45s overlap=0.27s caller kept talking 0.42s the agent started talking while the caller still had the floor, overlapping them for 0.27 s ... 5 incidents: 0 critical, 5 warning report: hotato-output/autopsy-apx-3b69ea8fc6d9.html pin: apx-3b69ea8fc6d9 (incidents apx-3b69ea8fc6d9#1..#5)
9.0 seconds of a 2-channel recording,
5 incidents: 0 critical, 5 warning, one pin id. 2 further incidents measured overlap=0.00s and are cut here for length. The clip is
ami-en2002b-take-0930 from the AMI Meeting Corpus, CC BY 4.0, and ships in
the repo under corpus/real/audio/.
$ PYTHONPATH=src python3 -m hotato.benchmark --scenarios corpus/real/scenarios --audio corpus/real/audio
$ hotato autopsy corpus/real/audio/ami-es2002a-take-0677.example.wav
This is measurement error against human labels, not an accuracy score. The errors have a long tail: the widest caller-onset clip lands 1430 ms out and the widest time-to-yield clip 2270 ms out, so the median, not the mean, is the summary a single outlier cannot drag. Errors quantise to the 10 ms frame hop, and on both signals the closest clips land on the label exactly. Every clip is shown, tail included, with no rounding.
Each clip, its human label, and its provenance sit in the repo at corpus/real/ →
Every claim is capped by the evidence under it.
Each finding prints the tier of its input: a four-step ladder from separated audio down to refusal. Only tier 1 decides a pass or fail.
-
Dual-channel audio
Caller and agent separated at capture. Overlap and timing are facts of the recording, exact to the sample.
Licenses a verdict. Only this tier gates a build:
pinfreezes one caught moment,provere-runs it in CI, and only these calls enter the Voice Stability denominator. -
Mono, plus provider metadata
A speaker-attributed transcript, a pipeline trace, tool-call logs, or diarizer turn labels.
Licenses an attributable finding. Each one names the source that attributed the speaker, and never merges into tier 1’s counts.
-
Raw mixed mono
No metadata. A mono gap says everything stopped, not who stopped.
Licenses symptom detection: dead air and latency gaps, each with a measured confidence and its derivation printed beside it.
-
Insufficient evidence
An unreadable file, a silent required channel, or a mixed export where a deterministic verdict was asked for.
Licenses nothing, and says so: exit
2, with the reason and the next step. A refusal leaves no artifact and never becomes a number.
The Voice Stability Score is the share of dual-channel calls with no critical incident, times 100; mono findings get their own block and counts. In the repo: EVIDENCE-CONTRACT.md → · how verify refuses a tampered proof →
Three properties hold every verdict up.
Determinism, a bounded scope, and a judge that never touches the exit code.
The same input, the same verdict.
The scoring paths run no model. On a pinned version, one recording produces one report, byte for byte, so a CI exit code means what it meant last week. Exit 0 pass, 1 fail, 2 refuse.
It measures turn timing and say-do, not intent.
Timing comes from energy over time, one lane per party: which channel was live, and when. Say-do checks a claim against your trace and system of record. It names no speakers and grades no words.
The model judge never gates.
A model verdict rides its own advisory lane, labelled as such in every report. It cannot certify an outcome and it cannot change an exit code; the deterministic checks decide.
Validation methodology, the labeled corpus with every fixture's consent status, and twelve maintainer-recorded calls scored offline all ship in the repo beside the code that produced them: VALIDATION.md, corpus/real/, and the evidence standard. Clone the repo and re-run any one of them. The capabilities these numbers back: turn-taking timing, say-do, and the CI gate that replays them.
hangover_sec after
its energy drops — between two words that bridges a gap, which is what it is for,
but after the last word it reports silence as speech and the other channel’s onset
lands inside it. Removing that put 25.7 s of
reported talk-over back to zero across 177 clips. No clip reports more overlap
than before, and every clip built around a barge-in still reports one. Source:
CHANGELOG.md, 1.20.0.Point it at your own calls.
Run the same commands on your own recording or a timestamped transcript, scored offline.
$ hotato autopsy ./call.wav
One recording is enough to start. Get started → · self-host →