LLM-as-a-Judge

A judge is only as good as its criteria. Grade one dimension at a time with binary or 3-point scales — 'rate quality 1–10' produces noise; specific checkable criteria produce signal:

You are grading a support answer. Score each criterion 0 or 1:

1. grounded: every factual claim is supported by the provided context
2. complete: the user's actual question is answered
3. safe_commitments: makes no promises (refunds, timelines) beyond policy
4. tone: professional and direct

Return JSON: {"grounded": _, "complete": _, "safe_commitments": _, "tone": _,
"worst_failure": "<one sentence or 'none'>"}

Direct scoring grades one output against the rubric — use for absolute gates in CI ('grounded must be 1'). Pairwise comparison shows the judge two outputs and asks which is better — far more reliable for A/B decisions (prompt v1 vs v2, model A vs B) because relative judgment is easier than absolute. For pairwise: randomize which side is A, and run both orderings on ties.

Known failure modes, with standard mitigations: position bias (favors the first/last option → randomize order), verbosity bias (favors longer answers → instruct explicitly that length is not quality; cap lengths), self-preference (models favor their own family's style → use a different model as judge than the one being judged), and rubric drift (judge improvises criteria → require the JSON schema, reject nonconforming grades).

Hand-grade 30–50 outputs yourself, run the judge on the same set, and measure agreement. High agreement → automate at scale, spot-audit weekly. Low agreement → the rubric is ambiguous (usually) or the task is beyond the judge (sometimes); fix and re-calibrate. An uncalibrated judge is a random number generator with confidence — calibration is what converts it into measurement.