Home/Docs/Tiered suites

Tiered corpus suites

Four labelled suites, 112 scenarios, every one synthetic shaped noise rendered deterministically from its own labelled timings. The timings are the ground truth, and the defect suites exist to fail.

Suite	Scenarios	Conditions	Expected exit
`silver`	40	clean, 16 kHz, default noise floor	`0`
`silver-defects`	16	clean conditions, deliberate defect renders	`1`
`gold`	40	hard conditions: noise floors, 8 kHz, gain extremes, echo, edge timings, endurance	`0`
`gold-defects`	16	hard-condition defect renders, plus two labelled capture-defect cases	`1`

The families span the behaviours that matter on a real call: hard interruptions (onset, speed, duration, resume), backchannels (position, density, repeats, long acks), double-talk, one-word interruptions, stutter onsets, multi-turn exchanges, resume-then-reinterrupt, and latency prompts. corpus/suites/manifest.json is the machine-readable inventory.

Defect suites fail by design

Every scenario in a defect suite is rendered to fail on its labelled axis: an agent that keeps talking through a real interruption, yields to a backchannel, or misses its latency budget. Exit 1 there is the scorer catching what it claims to catch.

The negative control

A suite that can only pass proves little. The defect suites are the negative control that gives the passing suites their meaning: the same scorer, the same thresholds, and a known-bad render must come back red.

Run a suite

Any suite is a scenarios directory plus an audio directory, on every surface: run, report, export, and the pytest gate.

bash

hotato run --suite barge-in \
  --scenarios corpus/suites/gold/scenarios \
  --audio corpus/suites/gold/audio

Deterministic builder

The suites are generated, never hand-edited, and the generator is the proof. Each scenario renders from a seed derived from sha256(scenario_id), so the audio on disk is exactly what the labelled timings say it is, on any machine.

bash

python3 corpus/suites/build_suites.py           # rebuild in place
python3 corpus/suites/build_suites.py --check   # regenerate to a temp dir, byte-compare
build_suites --check: regenerated output is byte-identical to disk

Synthetic is the floor

These suites prove the plumbing and catch regressions. Validity on your system comes from your own labelled calls: the walkthrough is docs/SUBMITTING.md, and running suites against a live stack is docs/BENCHMARK-STACKS.md.

Contribute

New synthetic families are welcome through the normal PR path. The highest-value contribution stays a real, consented, labelled call: open the corpus-submission issue form and it walks you through record, label, validate, submit.