Production Eval Loops
Wiring evals into CI, canaries, and production feedback — the loop that catches regressions and hallucinations before users do.
Pre-merge: eval suites run in CI on every change to prompts, retrieval, tools, or model config — exactly like tests, because that's what they are. Structure them in tiers: a fast smoke suite (a few dozen golden cases) on every PR, the full suite (hundreds of cases, judge-graded) nightly and before releases. Gate on both pass-rate thresholds AND no-regression-on-previously-passing-cases — aggregate scores hide it when your fix for one case breaks three others. Every prompt change ships with its eval diff, the way code ships with test results.
Deploy-time: roll AI changes out like risky infrastructure changes, because they are. Canary the new prompt/model on a few percent of traffic and compare quality proxies and cost against baseline before ramping; keep the old configuration one toggle away. For high-stakes surfaces, shadow-mode first — run the new version alongside production without showing users, and judge-compare outputs offline. Provider model migrations deserve the full ceremony: run the entire golden suite on the new model, read the failure diff, then canary.
In production, sample continuously: judge-grade a slice of live traffic for groundedness and quality, monitor the drift metrics from aa-20, and detect hallucinations where they matter most (RAG groundedness checks — does the answer's claims appear in the retrieved context?). Feed every confirmed production failure back into the golden dataset — this is the flywheel: production failures become eval cases, eval cases prevent recurrence, and quality ratchets forward instead of oscillating. Teams that skip the loop re-fix the same failure every quarter and call it firefighting.