Why Traditional Testing Fails for AI
Nondeterminism, no single right answer, silent regressions — and the golden dataset + failure analysis foundation that replaces assertEquals.
Unit tests assume determinism and a known right answer; LLM systems have neither. Outputs vary across runs and model versions; most tasks have many acceptable answers and fuzzy quality gradients; and the failure surface is distributional — the system is 94% good, and which 6% fails shifts when anything changes. A provider model update, a prompt tweak, a retriever change: each can silently reshape behavior while every traditional test stays green. Teams without evals discover regressions from users; teams with evals discover them in CI.
The foundation is a golden dataset: curated inputs with expected outcomes or grading criteria — not just happy paths but the hard cases, edge cases, and known past failures. Start scrappy (the 15-question table from aa-08 was one) and grow it from production: every real failure becomes a row, which is how your eval set converges on the distribution that matters. Where exact answers exist, score exactly; where they don't, write per-case criteria a grader can check ('cites the refund policy; does not promise a timeline').
Failure analysis is the discipline that directs effort: read the failures and label WHY each failed — retrieval miss, wrong tool, format break, hallucinated fact, refusal. The taxonomy tells you where to work; without it, teams 'improve the prompt' when 70% of failures were retrieval. This loop — dataset, run, labeled failures, targeted fix, re-run — is the engineering core of AI quality. Everything else in this module is machinery for running it at scale.