The agent said it issued the refund. Check the trace and the database.
Say-do grades what the agent claimed against what your systems record. Every deterministic check is pure regex, checksum or span lookup — never a model call — and a check with no evidence to read comes back INCONCLUSIVE rather than guessed.
# infer a starter assertions file from a trace's tool_call spans $ hotato assert init --from-trace voice_trace.jsonl # evaluate it against a call's transcript, spans and timing $ hotato assert run --assertions assertions.yaml --trace call.jsonl # the bundled, deterministic packs you can merge into a run $ hotato assert packs
tool_call assertion per distinct tool the trace saw, plus a require_order assertion once two or more tools were observed. The tool calls it a starter you edit, not a claim that these are the right assertions for the call.The deterministic lane
Five kinds, no model in any of them.
The split between what a machine can settle and what it cannot is structural here, not a setting.
| Kind | What it reads |
|---|---|
| outcome | Whether the task happened, resolved over its sub-predicates as a met/of fraction. |
| tool_call | The ingested trace spans. A span is the trace’s record that a tool ran; the transcript is not consulted. |
| phrase | The transcript, by pattern. Absent a transcript, the check is INCONCLUSIVE. |
| pii | The transcript, for identifiers that should not have been spoken. |
| policy | A named, versioned, offline pack of banned language and required disclosures. |
The summary always prints the deterministic pass, fail and inconclusive counts separately from the judge count, and emits no merged number. A model judge is a quarantined capability that runs on its own lane; it never wears the authority of a trace span or a database row.
INCONCLUSIVE is an outcome, and you choose what it costs.
Missing input leaves a check inconclusive rather than guessed. By default that never fails the run — but a compliance suite can make it gate.
- report.
The inconclusive count is printed and the run’s exit code ignores it. This is the default.
- fail.
An inconclusive check fails the run, so a missing trace is treated as a broken test rather than a shrug.
- refuse.
The run refuses outright.
Set it per run with --inconclusive-policy, or once in the assertions file so the policy travels with the checks.
no --transcript → phrase, pii, count checks no --trace → tool_call, tool_result, sequence, latency no --state → state, state_change (Authority 2)
$ hotato test run conversation-test.yaml --agent support-v3 \ --transcript turns.json --trace call.jsonl --state sandbox.json \ --audio call.wav --out ./conv-artifact
success.required conditions, and a per-dimension scorecard.- conversation-test.v1
- hotato.conversation.v1 artifact
- no overall_score
Success is a boolean over named conditions, never a blended score.
There is no overall_score in a conversation test result. Conditions are named, and the exit code honours the file’s own inconclusive policy exactly as assert run does, raised to non-zero when a required condition fails.
--repetitions N runs the deterministic lane N times and reports the per-run outcomes plus a reliability aggregate: pass@1, pass@k, pass^k and a Wilson confidence interval, threaded into the report’s Reliability dimension. Every run scores the same recording, so that lane has zero variance and pass^k equals pass@1 — the aggregate says so rather than dressing the repetition up as new information.
The model-judged rubric lane returns INCONCLUSIVE and is counted on its own. The evidence is bound into a hotato.conversation.v1 artifact, and --created-at lets you pin the timestamp for a byte-reproducible manifest.
A tampered artifact is refused, never quietly accepted.
conversation verify re-hashes every bound child against its recorded sha256 and exits 2 on any mismatch, missing child or malformed manifest.
$ hotato conversation verify ./conv-artifact $ hotato conversation verify ./conv-artifact --format json