Did the task happen, or just get claimed?
Say-do: the claim, graded from the trace and the system of record, never from words.
| kind | what it reads | example verdict |
|---|---|---|
| outcome | all_of or any_of the sub-predicates below, as a met / of fraction | met 3 / of 3 → pass |
| tool_call | the ingested trace spans, never the transcript | issue_refund span present → pass |
| tool_result | a tool’s returned value, from the trace | refund result ok → pass |
| tool_error | a tool’s raised error, from the trace | charge raised timeout → fail |
| state | a state adapter snapshot: the system of record | balance shows the credit → pass |
| state_change | a before / after delta (mock adapter) | balance moved by the refund → pass |
| sequence | an ordered set of steps across tools | lookup before refund → pass |
| count | a bounded number of occurrences | at most one refund → pass |
Three authorities: the trace, your data, a human.
Every verdict is grounded in one of three sources, and can read no fourth.
A state query resolves three ways: match (PASS), no such row (a grounded FAIL), or unreachable (INCONCLUSIVE, with a reason).
What the agent did.
tool_result and tool_error read the authenticated voice_trace. The span is the evidence a tool ran.
What your data shows.
state and state_change query a post-call state adapter: your DB or API. Absent adapter → INCONCLUSIVE.
What a person confirmed.
An Ed25519-signed human label record: the final word when a machine can’t settle it.
A model’s opinion never wears Authority 1 or 2.
A model verdict runs on its own lane: advisory, and out of the deterministic exit code.
version: 1 assertions: - id: disclosure-and-language kind: policy # banned language + required disclosures dimension: policy # a named, versioned, offline pack - id: workflow-order kind: tool_call # from the ingested trace require_order: [verify_identity, lookup_account, issue_refund] never_before: {tool: issue_refund, until: verify_identity}
never_before constraint on issuing the refund, plus its required order.- policy pack · versioned
- tool_call · required order
- value never echoed
Did it follow the rules? Same evidence, no model.
A versioned, offline policy pack checks banned language and disclosures; a tool_call assertion holds the workflow to its required order.
The same deterministic machinery grounds both: no model runs, byte-stable on a fixed version.
Outcome answers whether the task happened. Turn-taking answers whether it listened, and checks the words when you bring a transcript.
How do I test voice agent tool calls?
Assert them against the trace, never the transcript. tool_call, tool_result, and tool_error read the authenticated voice_trace spans; state and state_change query your own system of record after the call.
An agent’s spoken claim can never satisfy one of these checks. The span is the evidence a tool ran.
The tool call fired twice
A count assertion bounds it: count the tool’s spans in the ingested trace and require at most one. Two spans against a bound of one is a grounded fail, even when the transcript sounds fine.
A sequence assertion holds the surrounding steps to their required order.
The agent charged or booked twice
The same count assertion bounds the charge or booking tool to one occurrence per call, and state_change confirms the system of record moved once.
Authority 1 is the trace; Authority 2 is your DB or API behind a state adapter. With no adapter supplied, the state check reports INCONCLUSIVE instead of guessing.
The agent hangs silent after a tool call
A latency assertion times the span: name the tool, set max_ms, and the slowest matching span’s latency_ms must come in under it. A count of tool_timeout spans catches the calls where the tool never came back.
A sequence of tool_call, tool_timeout, tool_retry pins the recovery shape. All three read the ingested trace.
Prove it happened. Not just claimed.
autopsy reads one recording and lists its incidents, offline, in one command.
$ hotato autopsy ./call.wav
One recording is enough to start. Get started → · Turn-taking →