Home/Docs/The regression loop

Turn a bad call into a regression test

A production call where the agent talked over someone is your best test asset. Seven steps from that call to a CI gate that keeps the bug fixed.

Fix vs. prove

Your stack’s runtime controls fix turn-taking. Hotato proves the fix worked and fails CI when the bug comes back.

The loop this page walks through: capture the bad call, score it, tune your stack, compare before and after, keep the call as a fixture, gate CI on it.

1. Export the dual-channel recording

Pull the bad call with caller and agent on separate channels. The capture guides show the setting per stack.

bash

# auto-pull from your stack (the download is the only network call)
uvx hotato capture --stack vapi --call-id <id>
uvx hotato capture --stack twilio --recording-sid RE...

2. Score it

One command scores offline, writes the visual report, and opens it.

bash

uvx hotato doctor --stereo call.wav

3. Read the failure and the fix class

fix[config]: a stack setting is off. The output names the knob.
fix[engagement-control]: a discrimination case no single threshold settles.

4. Tune your stack’s controls

The fix lives in your runtime, in one of these control families:

Interruption settings: sensitivity, words-to-interrupt or minimum-duration thresholds.
Endpointing: min-silence and end-of-turn latency.
VAD sensitivity on the input path.
TTS buffering: how much queued speech plays after a stop signal.

Per-stack knob names: ADAPTER-STATUS.md in the repo.

5. Re-run and compare

Replay the same caller audio against your changed agent, record the new call, and diff the two runs.

bash

# before the change: save the envelope
uvx hotato run --stereo call.wav --expect yield --format json --no-fail > before.json

# after: score the new recording with deltas against the saved run
uvx hotato report --stereo call-after.wav --expect yield --base before.json --out compare.html

The report marks each signal better or worse: talk-over, time to yield, and any pass/fail flip.

6. Keep the recording as a fixture

Commit the audio plus one scenario JSON to your repo. That bad call is now a permanent test.

your/scenarios/prod-2031-talk-over.json

{
  "id": "prod-2031-talk-over",
  "title": "Caller says stop, agent keeps talking",
  "category": "should_yield",
  "expected": { "yield": true, "max_time_to_yield_sec": 0.9, "max_talk_over_sec": 1.0 }
}

Audio goes next to it as your/audio/prod-2031-talk-over.example.wav; the suite finds it by <id>.example.wav. Backchannel case? Use "category": "should_not_yield" and "yield": false.

Your suite, your repo

This is your own regression suite; nothing leaves your machine. Strip PII before committing audio anywhere shared. To donate a fixture to the public corpus, see Contributing.

7. Gate CI on it

Every failing fixture is a red build, with exit codes CI already understands.

bash

# run your fixtures: exit 1 on a regression
uvx hotato run --suite --scenarios your/scenarios --audio your/audio

# or gate the pytest run you already have
pytest --hotato-suite --hotato-suite-scenarios your/scenarios --hotato-suite-audio your/audio

The GitHub PR check posts one sticky results comment per pull request and fails the job on a regression.

The durable loop

Every bad call you keep makes the suite stricter. The moment that caused the incident can never regress quietly again.