Did the task happen, or just get claimed?
Whether it happened is a question for the trace and the system of record.
Eleven deterministic ways to prove a task ran.
Every outcome kind is regex, checksum, or a span or state lookup. Each reads a source of record; each carries deterministic: true, including when it returns INCONCLUSIVE.
| kind | what it reads | example verdict |
|---|---|---|
| outcome | a composite over the sub-predicates below, as a met / of fraction | met 3 / of 3 → pass |
| tool_call | the ingested trace spans, never the transcript | issue_refund span present → pass |
| tool_result | a tool’s returned value, from the trace | refund result ok → pass |
| tool_error | a tool’s raised error, from the trace | charge raised timeout → fail |
| state | a state adapter snapshot: the system of record | balance shows the credit → pass |
| state_change | a before / after delta (mock adapter) | balance moved by the refund → pass |
| handoff | a transfer span in the trace | escalated to a human → pass |
| dtmf | a keypad-tone span in the trace | 1-2-3-4 collected → pass |
| termination | how the call ended, from the trace | ended by caller hang-up → pass |
| sequence | an ordered set of steps across tools | lookup before refund → pass |
| count | a bounded number of occurrences | at most one refund → pass |
tool_call with no trace at all reports INCONCLUSIVE. A tool_call against a trace that was ingested and holds zero spans is a grounded FAIL. The distinction is structural.
$ hotato test run conversation-test.yaml --agent support-botWhat the agent did.
tool_result and tool_error read the authenticated voice_trace. The span is the evidence a tool ran.
What your data shows.
state and state_change query a post-call state adapter: your DB or API. Absent adapter → INCONCLUSIVE.
What a person confirmed.
An Ed25519-signed human label record. The final word for the call a machine cannot settle on its own.
Three authorities: the trace, your data, a human.
An outcome is only as good as what it reads. Hotato grounds each verdict in one of three sources, and structurally cannot read a fourth.
A state query has three outcomes: the record matches (PASS), the record was read and holds no such row (a grounded FAIL), or the record could not be reached (INCONCLUSIVE, with a reason). Hotato never invents a verdict from a state it could not observe.
A model’s opinion never wears Authority 1 or 2.
The evaluators read the trace and the state adapter by construction. A model verdict lives only in the advisory rubric lane, marked deterministic:false: it describes how a call sounded, and leaves certifying that the refund posted to the deterministic lane.
When the state adapter is missing, the Outcome dimension stays INCONCLUSIVE and visible on the page.
version: 1 assertions: - id: disclosure-and-language kind: policy # banned language + required disclosures dimension: policy # a named, versioned, offline pack - id: workflow-order kind: tool_call # from the ingested trace require_order: [verify_identity, lookup_account, issue_refund] never_before: {tool: issue_refund, until: verify_identity}
- policy pack · versioned
- tool_call · required order
- value never echoed
Did it follow the rules? Same evidence, same lane.
Policy rides with Outcome. A named, versioned, offline policy pack checks banned language and required disclosures; a tool_call assertion holds the workflow to its required order and its “never before” constraints.
It is checked by the same deterministic machinery that grounds every outcome. No model runs; the result is byte-stable on a fixed version.
Outcome answers whether the task happened. Conversation checks whether it listened; Speech checks what was said.