Run it a hundred times. Report what held.
Hotato replays a scenario across seeds and repetitions, and reports whether it held every time.
$ hotato simulate scenario.yaml --out runs/pass^k demands a clean sweep.
Deterministic scripted and state-machine callers replay each scenario across its variables, seeds, and repetitions. The reliability read reports pass@1, pass@k, and pass^k (held on every run) with a Wilson 95% interval.
Per-run seeds are derived from sha256 of the scenario and its variation tuple: no wall-clock, no Math.random. A fixed scenario and seed reproduce the same conversation bytes every time, so a red run points at a change in behavior.
pass^k holds only if every run passed. One flake sends reschedule to 0.00. The hatched cell is SIMULATOR_INVALID: excluded from n, never a pass or a fail.
A broken simulation is a broken fixture, set aside.
When a run does not faithfully render its scenario, it is reported as SIMULATOR_INVALID and dropped into its own bucket: excluded from the count, never scored as an agent pass or fail.
Every produced conversation is stamped origin=simulated, and the writer refuses to save anything whose origin is not simulated. Synthetic runs are never merged into a bucket of recorded calls.
What ships today, what's gated, what's next.
Deterministic simulation runs right now with no credentials. Placing a live billed call is gated behind live credentials and an opt-in. A generative caller is on the roadmap, and labelled as such.
Deterministic simulate
Scripted and state-machine callers render a scenario with no model, no TTS, no network. Seeded, byte-reproducible, offline.
- hotato simulate
Drive a live call
Place a live billed call through Vapi or Twilio, double-gated behind your own creds and HOTATO_DRIVE_OPT_IN=1.
- gated · billed
Generative caller
An LLM-driven caller that improvises inside a scenario. Not in this release; the deterministic foundation lands first.
- roadmap · Phase 6