Getting started
Three commands, offline, no account and no credential. A bundled call fails on screen before you need a recording of your own; then you score it, gate CI on it, and point the same command at your own call. Your audio and traces stay on your machine.
Install, then hotato start --demo (the bundled first run), then hotato investigate on the call it just wrote, then hotato contract verify contracts/ — the gate, which exits 1 by design. Everything below is a command to paste, in that order. Your own recording is step 6, after you have seen the loop run.
1. Install
uvx fetches and runs hotato from PyPI with no install step; pip, pipx and uv tool install it once for daily use. Working from a clone of the repository instead? Run the module out of src/ and every command on this page is the same, with PYTHONPATH=src python3 -m hotato in place of hotato.
# run it without installing uvx hotato --version # or install once for daily use pip install hotato # or: pipx install hotato, or: uv tool install hotato # or from a clone, no install and no index: git clone https://github.com/attenlabs/hotato && cd hotato PYTHONPATH=src python3 -m hotato --version hotato 1.20.0
2. Watch a bundled call fail
hotato start --demo sweeps the two calls bundled inside the install, ranks the failing moment, freezes it as a portable contract, re-scores it, and runs the say-do check on a bundled scripted conversation. Offline, no account, no key, and nothing of yours needed. It fails by design: the agent talks straight over a caller interruption. Setup finishing is what makes it exit 0; the verdict inside it is FAIL. Want the one-command version, without the contract and the say-do act? hotato autopsy --demo scores the same bundled WAV and writes one HTML report: [CRITICAL] BARGE-IN t=1.99s overlap=2.51s agent did not go silent within 3.0s.
hotato start --demo [... first line lists every file the run wrote; the say-do act is quoted in section 5 ...] hotato start: swept the 2 bundled demo calls offline. sweep result: hotato-sweep.json sweep dashboard: hotato-sweep.html funnel card: hotato-no-single-threshold.svg demo contract: contracts/demo-missed-interruption.hotato ------------------------------------------------------------------ Conversation failed: Agent did not yield; measured talk-over was 2.51 s. ------------------------------------------------------------------ In plain terms: after the caller took the floor, the agent kept talking over them for 2.51s instead of yielding. The measured signal on this one call: talk-over 2.51s seconds the agent kept talking while the caller held the floor Share in a PR: hotato-failure-record/failure-record.md Share as an image: hotato-failure-record/failure-record.svg Verify the record: uvx --from hotato==1.20.0 hotato record verify hotato-failure-record/failure-record.json re-scored FAIL, by design: this call talked over the caller. The contract freezes that moment as evidence, so a CI gate flags any later change to its evidence or policy. Proving the live agent improved uses a fresh recapture (docs/RECAPTURE.md). Setup finished, so start --demo exits 0. See the gate return exit 1: hotato contract verify contracts/ Your next step -- score a call now (the demo just wrote one you can score): hotato investigate contracts/demo-missed-interruption.hotato/audio/event.wav Run the same command on your own recording once you have one.
3. Score the call
investigate takes the WAV the demo just wrote, checks the file is scorable, ranks the candidate turn-taking moments, and prints the exact command to label the one you pick. The state path it remembers is absolute, under the directory you ran in. Labeling is your decision on the moment: --expect yield means the agent should have stopped for the caller, --expect hold means it was right to keep talking, and it writes the contract the gate below re-measures.
hotato investigate contracts/demo-missed-interruption.hotato/audio/event.wav hotato investigate [run 1]: event.wav most likely failure (top-ranked candidate): ---------------------------------------------------------------- [1] t=1.99s overlap_while_agent_talking overlap=2.51s agent did not go silent within 3.0s the caller took the floor and the agent kept talking over them for 2.51 s without going quiet within 3.0 s ---------------------------------------------------------------- turn it into a CI contract (use --expect hold instead if the agent was right to keep talking): hotato investigate label '/tmp/firstrun2/.hotato/investigate-state.json#1' --expect yield capture origin: operator-asserted local file (event.wav) you supplied this WAV path directly; hotato has not independently verified it against any vendor, so its origin is operator-asserted only input health: eligible for scan acoustic health (signal measures, not speech content): caller ch0: SNR 54.1 dB, 59.1% silence, 9.2/min energy bursts, clipping 0.00%, 6.5s agent ch1: SNR 54.3 dB, 8.3% silence, 9.2/min energy bursts, clipping 0.00%, 6.5s verdict path: eligible (a labeled event here can carry a yield/hold verdict) state remembered at: /tmp/firstrun2/.hotato/investigate-state.json
4. The gate that stays red
contract verify is the CI command. It re-measures the evidence stored in each contract and exits 1 while a pinned failure is still failing, so the build goes red. It never re-runs your live agent, which is why a regression cannot slip through as routine.
hotato contract verify contracts/ hotato contract verify: contracts/ (1 contract) [FAIL] demo-missed-interruption (expect yield): did_yield=False seconds_to_yield=n/a talk_over=2.51s | integrity: intact 0/1 contracts pass; exit_code=1 These contracts pin known failures. Each stays red until you fix the agent and recapture the call, the same way a snapshot test stays red until you update the snapshot. This step re-measures stored evidence and never re-runs your live agent, so a real regression cannot slip through as routine. Path to green: fix the agent, then recapture with `hotato drive <bundle>` (vapi/twilio), or the manual path in docs/RECAPTURE.md. exit=1
0 every contract passes · 1 a pinned failure is still red, the build fails · 2 a usage or input error, including an input that is not scorable. Scaffold the workflow for your repository with hotato init starter --stack generic --out . (--stack vapi, retell, twilio, livekit or pipecat tunes it), or wire it by hand from the CI page.
5. The say-do check
Turn-taking measures how the call sounded. The say-do check asks a separate question: did the agent do what it told the caller it did? Tool spans and post-call state decide it, never the agent's words. The bundled conversation carries a refund the agent claimed and never issued, and start --demo reports it in the same run.
# quoted from the same `hotato start --demo` run as section 2 conversation: saydo/ (transcript + tool trace + post-call state + test) test result: saydo/test-run.json say-do check: FAIL, by design: the agent said the refund was sent; the trace shows no such tool call succeeded (no issue_refund span), and the order's post-call refund_status stayed "none". # the gate form of the same check, exit 1: hotato test run saydo/test.json --agent demo-agent --transcript saydo/transcript.json --trace saydo/trace.jsonl --state saydo/state.json
6. Run it on your own call
Same command as section 3, pointed at your recording: hotato investigate ./call.wav. Everything above works without one, so bring yours when the loop already makes sense. Recording still sitting in your stack? hotato connect vapi (also retell, twilio and the other vendor-hosted stacks) runs one auth check and stores the credential in ~/.hotato/connections.json at mode 0600, on that machine only and sent only to the vendor's own API; then pull and score in one step with hotato investigate --stack vapi --call-id <id>. When you have a fix, close the loop: recapture and compare before and after → · see the whole loop run against twelve recorded calls →
Separate tracks give every overlap an owner: caller on channel 0, agent on channel 1, one party per channel, so you can see who was already talking and who cut in. A mixed mono file cannot carry that, and hotato scores it best-effort rather than with an owned overlap. If your stack exports one mixed file, switch on separated recording; the capture guides show the setting per stack.