Model judge · Hotato

rubric.v1

A local judge that shows its work.

The default judge is a local Ollama model at http://localhost:11434: qwen2.5vl:3b, reached with only the standard library. Zero egress, no opt-in to enable.

The model call is not deterministic; no model call is. What is fixed is replay: every verdict carries full provenance and is content-addressed, so a stored result stays byte-identical and a re-query diffs against it to surface any drift.

Run the judge in your own environment →

a rubric.v1 verdictdeterministic:false

$ hotato rubric run --rubrics rubric.yaml --transcript t.txt
{
  "schema": "rubric.v1",
  "id": "acknowledged-frustration",
  "kind": "judge_rubric",
  "dimension": "conversation",
  "deterministic": false,
  "verdict": "INCONCLUSIVE",
  "provider": "ollama",
  "model": "qwen2.5vl:3b",
  "model_digest": "sha256:7b1c…e04a",
  "prompt_id": "rubric.judge",
  "prompt_version": 3,
  "prompt_sha256": "9f2adb…41c",
  "temperature": 0,
  "input_sha256": "c41d8c…9e2",
  "cache_key": "ollama:qwen2.5vl:3b·9f2adb·c41d8c",
  "cached": true,
  "verdict_sha256": "a77e15…b0",
  "votes": ["pass", "fail", "inconclusive"],
  "disagreement": true,
  "confidence": 0.33,
  "confidence_required": 0.85,
  "citations": ["transcript#turn-4", "transcript#turn-6"],
  "review": { "human_required": true,
             "on": ["disagreement", "confidence_below_threshold"] }
}
# exit 0 (advisory): verdicts are reported, never gate

confidence is the fraction of votes that agreed, logged as provenance. The votes disagreed, so the judge abstained and flagged the case for a person.

Structural separation

Two lanes that never touch.

The 17 deterministic assertion kinds live on one wall: assert.v1, deterministic:true, no model. The judge lives alone on the advisory shelf, and a model verdict physically cannot become a deterministic result.

Advisory is the default: a rubric FAIL is reported and gates nothing. --gate opts a team into failing CI on a rubric FAIL. Even then, INCONCLUSIVE never gates: an abstention cannot fail a build.

the two counts, side by side The unified report shows a deterministic count and a model-assisted (advisory) count on separate shelves, never merged into one line.

one report, two shelves

assert.v1 · the deterministic walldeterministic:true · no model

phrase
policy
tool_call
tool_result
state_change
timing_contract
sequence
outcome
+ 9 more

kept apart on purpose

rubric.v1 · the advisory shelfdeterministic:false · local model

model judge · advisory

Advisory by construction

It abstains, then defers to a human.

On one call the deterministic lane decides the verdict. The judge sits below it, on its own shelf, and when it cannot agree with itself it says so and sends the case to review.

rubric.v1 · agent support-bot · call 8f2a

Outcomerefund posted (trace span)deterministicpass

Policyrequired disclosure readdeterministicpass

advisory shelf · never gates the verdict above

Model judgeacknowledged the caller's frustration?advisoryinconclusive → review

The judge's votes disagreed, so it abstained and the case went to a person. The advisory shelf never changes the deterministic verdict. No overall score, on either shelf.

The model's opinion is visible and attributed, and boxed off from the decision.

Egress guard

Off-box is refused unless you say so.

The default judge never leaves the machine. Point it at a hosted provider or any non-local endpoint and the run is refused; the transcript stays put until you make the trade-off explicit.

Sending a call off-box is a decision. --judge-egress-opt-in is the single switch that allows it, the same posture the diarizer uses for its own opt-in.

judge egress guard

default judge stays on http://localhost:11434 · stdlib only

--judge-provider hosted without the opt-in flagrefused · exit 2

--judge-endpoint pointed at a non-local URLrefused · exit 2

local ollama · zero egress · no opt-in neededallowed

The transcript leaves the box only when you pass --judge-egress-opt-in.

Refusal is the default. The guard fails closed on exit 2.

Human-owned criteria

A human rubric is never scored by a model.

Some criteria are the human's to call. Mark a rubric kind: human_rubric and the model is never asked; it stays INCONCLUSIVE with human_required until a person signs the record.

An overall_score is rejected the moment a rubric file loads.

See the deterministic lane →

how each rubric kind resolves

judge_rubric: Scored by the pinned local model, advisory, cached and replayable with full provenance.
human_rubric: Never model-scored. Stays INCONCLUSIVE and human_required until a person signs an Ed25519 label record.
overall_score: Rejected at load. There is no blended number, anywhere, on either lane.

Calibration checks the judge against human labels on a held-out set, reported as raw counts.

The model judge, quarantined by design.

A local judge that shows its work.

Two lanes that never touch.

It abstains, then defers to a human.

Off-box is refused unless you say so.

A human rubric is never scored by a model.