Course contents
Project 5: Evals, Tracing and a CI Gate
Build the quality system for the support desk: a 60-case golden dataset, code checks plus a calibrated judge, traces for every call, a five-metric dashboard, and a CI gate that blocks regressions.
Right now, “the desk works” means you tried it. This project replaces that with numbers you can defend: a versioned dataset, checks that run in minutes, a judge you've calibrated against your own labels, traces you can open when something breaks, and a gate that stops a regression from merging. Start from the failures you wrote down at the end of the mid-course project.
The dataset
| Area | Cases | Include |
|---|---|---|
| Triage (Project 1) | 15 | Each label, two-problem emails, non-requests, one injection attempt |
| Help-center answers (Project 2) | 15 | Identifiers, follow-ups, 4 questions the docs don't answer |
| Order scenarios (Project 3) | 15 | The five scenarios, plus amount limits and unknown order ids |
| Routing and escalation (mid-course) | 10 | Two-problem tickets, follow-ups, cases that must escalate |
| Adversarial | 5 | Injected instructions in emails and in tool results |
Give each case an id, the input, the area, and either an expected value (for code checks) or criteria (for the judge). Synthetic customer details only. Keep the file in the repo, and review changes to it like code.
Write the dataset
Create
evals/golden.jsonlwith the 60 cases. Draft inputs with a model if you like, then review every one by hand and write the expected values and criteria yourself.VerifyEvery case has an id, an area and either an expected value or criteria, and no line contains real customer data.Turn code checks into tests
Write parametrized pytest tests over the dataset: schema validity and labels for triage, the refusal line and real citations for answers, tool names and arguments for order scenarios (read them from your Project 3 logs or traces). Save each run's per-case results to a JSON file.
Verifypytest evals/runs the code checks in a few minutes and writes a per-case results file.Add and calibrate the judge
Use the aa-19 judge for groundedness and safe commitments. Hand-label 40 cases first, run the judge on them, and record agreement plus precision and recall on fails. Tighten the rubric until you're satisfied, then fix the pass mark.
VerifyAJUDGE.mdfile records the judge model, the rubric version, agreement, precision, recall and the chosen pass mark.Trace every call
Instrument the desk with OpenTelemetry using the GenAI attribute names (by hand, as in aa-20, or with an auto-instrumentation library) and export to Langfuse or Phoenix, self-hosted or cloud. Put the trace id in your eval results so a failing case links straight to its trace.
VerifyFrom a failing case in your results file, you can open its trace and see the assembled prompt, the tool calls and the cost.Gate merges in CI
Commit a baseline results file. In GitHub Actions, run the smoke suite (code checks plus a dozen judged cases) on every pull request, compare per-case results with the baseline, and fail the job if any previously passing case fails. Run the full suite nightly.
VerifyA pull request that deliberately breaks the refusal line fails CI and names the regressed cases.Build the dashboard
Chart five metrics per day: cost split by input, cached input and output; p95 time to first token; error and rate-limit rate; refusal rate; and judge pass rate on a 2% sample of traffic. Add an alert on a three-day rise in refusal rate.
VerifyYour dashboard shows a week of the five metrics, and the alert fires when you replay a day of forced refusals.
Plain-English definitions from the AI Coding Dictionary.