# Hotato

> Portable, offline, deterministic, runtime-agnostic turn-taking / barge-in /
> endpointing eval for voice agents. Does your agent drop the turn, or hog it?
> Scores barge-in, talk-over, backchannel and endpointing latency from any call
> recording you already have (Vapi, Retell, Bland, LiveKit, Twilio) and points
> most failures at a concrete fix. Wire it into CI to catch a turn-taking
> regression before it ships.

Open-source (MIT), zero-install, offline CLI plus one MCP tool. Scores voice-agent
turn-taking from a call recording and returns a machine-readable pass/fail with a
non-zero exit code for CI, coupled to no runtime. Turn-taking eval is a populated
space (LiveKit and Pipecat ship native smart-turn models; Hamming and voicetest.dev
are cloud/paid; academic benchmarks exist). Hotato's edge is packaging, not novelty:
the portable, offline, deterministic, runtime-agnostic one you run on any recording,
in CI.

Score your own call (the real signal) - capture auto-pulls the dual-channel
recording and scores it offline (needs VAPI_API_KEY; the only network is the
direct recording download):
`uvx hotato capture --stack vapi --call-id <id>`
Already have a two-channel WAV? Score it directly:
`uvx hotato run --stereo call.wav --stack vapi --expect yield`
Built-in self-test (synthetic floor, zero input):
`uvx hotato run --suite barge-in`

## What it measures
- did_yield: whether the agent stopped for a real interruption
- seconds_to_yield: how fast it stopped
- talk_over: how long it kept talking after it should have stopped
- backchannel handling: whether it correctly talked through a listening noise ("mhm")
- endpointing latency: slow to stop, slow to start
All derived from audio energy and timing on your recording.

## What it does NOT do
- No speaker identification, no diarization
- No transcription, no sentiment/emotion inference
- No intent inference — energy is not intent; it knows who was talking, not who was right
- Synthetic fixtures are a floor and a regression guard, not a stand-in for production traffic

## The fix map
Most failures map to one config knob:
- Talks over the caller (ignored a real interruption) -> lower barge-in threshold / enable interrupt-on-speech
- Stops for "mhm" (yielded to a backchannel) -> raise VAD sensitivity / add a backchannel hold-off
- Slow to actually stop (high talk_over) -> shorten the output buffer / faster interrupt flush
- Cuts the caller off early -> lengthen the endpoint silence window
- Fails on BOTH axes at once (misses a real interruption AND false-triggers on a backchannel) ->
  no single threshold wins both directions; this needs a discriminating layer, not a different number.

## The both-axes case (needs a learned engagement-control layer)
When a recording fails both axes at once, no threshold fixes it: the two failures
pull the dial opposite ways. Telling a real interruption from a listening noise needs
a learned engagement-control / addressee-detection layer, not a config knob. Attention
Labs builds and licenses one, consent- and focus-gated. Hotato's own output stays
neutral: it flags the case and names the kind of fix, never a product and never a
number. Hit this wall? https://attentionlabs.ai.

## Honesty
No accuracy percentage, benchmark, or leaderboard is published, and there are no
traction claims. The edge is reproducibility: the same recording produces the same
verdict and the same fix pointer on every run, on any machine, offline. Open core,
MIT licensed, never relicensed.

## Links
- Site: https://hotato.dev
- Docs: https://hotato.dev/docs/
- Source: https://github.com/attenlabs/hotato
- License: MIT
- Built by: Attention Labs