CI for AI agents
Agents regress quietly. You change a prompt, bump a model, or edit a tool, and a behavior that used to work stops working, but nothing goes red. There is no failing test, because the thing that broke was a conversation, not a function. Continuous integration for AI agents means turning a specific failure into a check that fails the build the next time it comes back.
Why agent regressions slip through CI
Your unit tests still pass, because the code is fine. What changed is the agent's behavior in a full exchange: it now interrupts the caller, skips a required disclosure, books the reservation twice, or claims a refund posted that never reached the backend. None of that is expressible as a normal assertion over a return value. So the regression ships, and you learn about it from a user, a support ticket, or a bad recording.
An LLM judge in the loop does not close the gap on its own, because its verdict varies run to run and cannot read the environment where the action executed. To gate a pull request you need something that produces the same answer every time and reads real evidence: the trace spans, the audio channels, and the row your system of record actually holds.
Turn a caught failure into a contract
Hotato's unit of regression testing is a contract: a specific caught moment, frozen with an explicit expectation, addressed by the hash of its evidence. You find the failure once, label it, and pin it. From then on the contract re-measures that evidence under the pinned policy and returns a clean exit code.
- Catch the moment in a recording or run and label it:
hotato investigate label. - Pin it into a content-addressed contract with the expectation it must hold:
hotato contract create --expect yield. - Verify the whole contract folder, exit 0/1/2:
hotato contract verify contracts/.
hotato contract verify contracts/ --junit contracts-junit.xml exit 1 # a pinned expectation no longer holds: the build is red
The gate in a GitHub Actions workflow
The verify command is built for CI. It emits JUnit XML for the test panel, a Markdown verdict for the step summary, and JSON for anything downstream. It runs offline, with no service to call and no data leaving the runner.
- name: Verify hotato contracts run: | pip install hotato hotato contract verify contracts --junit test-results/hotato.xml \ --step-summary "$GITHUB_STEP_SUMMARY"
A green run means every pinned expectation held under the fixed policy. A red run names the contract that moved and the dimension it moved on, so the failure points back at the specific behavior, not a blended score. The verdict is byte-for-byte reproducible, which is the property that lets it gate a merge.
Deterministic gate, advisory judge
The gate runs on deterministic checks: turn-taking timing, trace and state assertions, policy packs. The model judge is available on a separate advisory lane and only gates when you opt in with --gate, so a varying model opinion never fails a build on its own.
The loop, end to end
Observe, catch, pin, test, prove is one command over a folder of recordings and your committed fixtures, so the same flow you run by hand runs in CI:
hotato loop ./recordings --fixtures tests/hotato hotato prove --contracts contracts/ # claim-scoped proof for the fixed catch
If you drive a text or chat agent instead of voice, the same contracts work: a scripted deterministic caller runs against a local HTTP endpoint and records a timestamped transcript in the shape the scorer consumes, so a chat agent goes from scenario to gate with no audio.
Cluster failures across a run history
As the suite grows you want to see where the agent breaks most, not just that one contract went red. Hotato clusters failures across a folder of runs into deterministic fingerprints, ranked by count, so a recurring say-do gap or a repeated talk-over surfaces as one named cluster instead of ten scattered reds. Timing drift between two saved run envelopes gets its own gate under explicit tolerances, so a slow creep in response gap fails before it becomes a caller-visible pause.
hotato diagnose --fleet runs/ # rank failure clusters by count hotato baseline check tolerances.yaml baseline.json candidate.json
Trends over time, pass rate and latency percentiles, land in hotato team runs/, so the same evidence that gates a merge also shows whether the agent is getting better or worse across releases.
Deterministic, offline, reproducible
Continuous integration only works if the check gives the same answer on the developer's laptop and on the runner. Hotato's deterministic dimensions are computed from evidence with fixed seeds, contracts are addressed by content, and there are zero runtime dependencies to drift. Nothing is uploaded: traces and prompts stay on the machine that runs the check. That is what makes an agent regression suite you can actually trust to block a merge.
Start here
Pin your first catch, then let CI hold the line:
pip install hotato hotato autopsy ./call.wav hotato contract verify contracts/
Docs: CI, regression loop, pytest. Repo: github.com/attenlabs/hotato.