hotato
GitHub
Home / Transcript

What was said, without scoring how it was said.

The Speech dimension checks the words deterministically (phrase, PII, entity accuracy, latency), and treats the transcript as opt-in context, never a graded artifact.

  • 4speech checks
  • 4PII detectors
  • 0egress default
  • MITlicense
The Speech checks

Four checks on the words. Every one deterministic.

Each is regex, a checksum, or a lookup against the timing envelope: no model, byte-stable on a fixed version.

phrase

A pattern is present, or absent.

A regex is found (or, in absent mode, never found), with an optional role filter and a first / last / any position.

  • deterministic
pii

A value that must never leak.

Deterministic detectors sweep the transcript in must_not_leak mode. A hit fails the check, and is reported without ever echoing the value.

  • deterministic
entity_accuracy

The right name, number, ID.

Whether the entities that had to be captured match, exactly, against your reference. A lookup and a comparison, not a judgment call.

  • deterministic
latency

The reply landed in time.

A tool or turn responded within its millisecond budget, read straight from the timing envelope. No transcript needed, no model in the path.

  • deterministic
pii-hit.json · [REDACTED] transcript
# transcript artifact: the matched value is never stored
turn 6 caller: my card is [redacted]
turn 7 agent : thanks, I have [redacted] on file

pii_hits:
  - detector: card_luhn
    turn: 6
    role: caller
    value: [redacted]   # never echoed
The result carries the detector, the turn index, and the role, and a [REDACTED] transcript. The matched value appears nowhere.
PII

The check catches the value. It never repeats it.

Four deterministic detectors: ssn, card_luhn (with a Luhn checksum), email, phone, sweep the transcript.

A hit is a FAIL, and the output is a [REDACTED] transcript plus hit metadata: which detector, which turn, which role. By construction, the matched digits never enter a result, a log, or a report.

The privacy posture in full →

Opt-in context

The transcript is context you turn on.

Point Hotato at a transcript you already have, or produce one with the --transcribe flag, faster-whisper, local, behind the [transcribe] extra, on run, assert run, or test run.

Transcription is a flag, so there is no standalone transcribe step. Whichever way the transcript arrives, the deterministic checks read it; the transcript itself is never graded.

transcribe locally, then check
$ hotato assert run --stereo call.wav --assertions assertions.yaml --transcribe

Speech checks what was said. Outcome checks whether the task happened; Conversation checks whether it listened.

assertions.yamldeterministic
version: 1
assertions:
  - id: disclosure       # transcript-derived
    kind: phrase
    regex: "recorded for quality"
    role: agent
    position: first
  - id: no-value-leak
    kind: pii
    detectors: [ssn, card_luhn, email, phone]
    mode: must_not_leak
  • reads the transcript
  • grades none of it
  • value never echoed