Your agent said the refund went through. Only the backend knows if it did.
An LLM judge can read a transcript and tell you the agent sounded correct. It cannot tell you whether the refund posted, whether the reservation was made once or twice, or whether the caller was ever the account holder. One HN commenter states the constraint directly: judges "can't reliably judge LLM outputs without access to the environment where those actions are executed" (news.ycombinator.com item 44920368). The model's own account of what it did is not evidence that it did it. Verification has to come from the backend, not the transcript.
The say-do gap is not rare and it is not cosmetic. One reported case has a refund tool firing when the caller only asked to check order status; the same comment reports a verification flow that never checked the caller's phone number against the account (news.ycombinator.com item 47248766). A maintainer issue records interrupted tool results dropped from the conversation history, producing duplicate reservations and duplicate database rows: the agent said it once and the backend did it twice (github.com/livekit/agents, issue 3702). A related class shows up where the trace itself goes quiet: a tool runs, the reply stalls, and the trace omits the causal evidence, so even the log cannot tell you what happened (github.com/openai/openai-agents-python, issue 1156). When the agent's word is the only record, a wrong action and a right one look identical.
There is a sharper version of this, and it deserves its own name: the authorization and security-state sub-pattern. Passing a prompt test does not establish that a caller was authenticated (news.ycombinator.com item 43081663). Releasing sensitive data cannot depend on the model choosing to comply with an instruction (news.ycombinator.com item 43118410). Those are not tone or phrasing failures. They are questions about persisted state: was ownership verified, was the session authenticated, did the authorization change. A model that says the right words while the security state says otherwise is the failure, and only the backend holds the answer.
Hotato grades say-do from evidence, not from model output. A tool_call assertion reads only the spans in your ingested trace, so an agent narrating a tool call it never made does not count. A state assertion queries your system of record through a state adapter and resolves to one of three outcomes: the row is present and matches, the record was read and the value is absent, or the source of record was unreachable. That third outcome is kept distinct on purpose, because "we could not check" is not the same claim as "it did not happen," and collapsing them is how a security failure passes. An outcome assertion combines the trace spans with that state query, so "the refund succeeded," spoken on the call, is checked against whether a refund exists and whether it belonged to the caller who asked.
The refund on an unverified order, the reservation booked twice, the authentication a prompt test never established: each becomes a scored, offline regression that reads the backend, not the model's summary of it. The catch locks into a content-addressed contract that re-measures the captured evidence under the pinned policy on every machine, so the action your agent claimed once is checked against the state it left behind on every CI run.
Check what your agent did, not what it said:
hotato autopsy ./call.wav