Observability & Monitoring
Traces for multi-step AI systems, the metrics dashboard that matters, and an honest survey of the platform landscape.
When a five-step agent gives a bad answer, the bug is in one step — and without traces you're guessing which. LLM observability captures the full tree: each model call (exact assembled prompt, response, model, tokens, cost, latency), each tool call (arguments, results), each retrieval (query, returned chunks). The non-negotiable is logging what the model actually SAW — assembled context, not your template — because most quality bugs are context-assembly bugs (aa-06), invisible unless you can replay the exact input.
The platform landscape, honestly: LangSmith (deepest LangChain/LangGraph integration), Langfuse (open-source, self-hostable — the default when data can't leave), Braintrust (evals-first with strong CI workflows), Arize Phoenix (open-source, retrieval-analysis strength), and W&B Weave (natural if you already live in Weights & Biases). Most use OpenTelemetry-compatible tracing underneath — instrument once via OTel and the platform choice stays swappable. Pick by constraint: self-hosting need → Langfuse/Phoenix; framework depth → LangSmith; eval-centric CI → Braintrust. Switching later is annoying but survivable if you avoided proprietary SDK calls at every call site.
Monitoring closes the loop production-side. Dashboards: cost per feature per day, p50/p95 latency (per model and per route), error and rate-limit counts, token distributions, and quality proxies — refusal rate, retry rate, thumbs-down rate, 'I don't find that in the docs' rate for RAG. Alert on trend breaks, not single events: a 3-day drift in refusal rate is signal where one bad Tuesday hour is noise. And wire feedback capture (thumbs, edits, escalations) into the trace — those become tomorrow's golden-dataset rows, which is the whole flywheel of Module 5.