Agentic AI Engineering/AI Security & Production Readiness
Lesson 6 / 6 · 480 min
Course contents
Project

Capstone: Ship a Production-Ready Agentic System

Bring the course together in a system you can show: grounded answers, tools behind approval gates, evals in CI, traces and dashboards, a justified model choice, current red-team results, and a one-page system card.

Finish the Acme support desk, or apply the same architecture to a domain you care about: an internal IT help desk, a research assistant over your company's documents, an operations agent for a small business. Either way, the capstone is judged by the readiness checklist below and by a short demo. Breadth matters less than evidence: every claim about the system should have a number, a trace or a test behind it.

Requirements, module by module

  • Building blocks: typed outputs and validated tool calls; model ids, prices and budgets in config.
  • Grounding: answers from your sources with citations, and honest refusals when the sources don't cover a question.
  • Agents: at least one agent acting through tools served over MCP, with capped loops and approval gates on consequential actions.
  • Evals and observability: 60 or more eval cases with a calibrated judge, a CI gate on per-case regressions, traces for every call, and a dashboard with alerts.
  • Local and fine-tuned models: a component that runs locally or on a fine-tuned model where the evals justify it, or a written, evidence-based reason why not.
  • Security and governance: a threat model, current red-team results, rails that fail closed, PII handling, a kill switch per feature, and a governance note.

Deliverables

  • The system: a repository, or exported n8n workflows with setup notes.
  • A one-page system card: purpose and users; an architecture sketch; models and versions, with each one's data-retention terms; data flows; eval results by area; known limitations; a risk register mapped to the OWASP LLM and agentic lists; an EU AI Act risk-category assessment; and an owner and incident contact.
  • A five-minute demo: one happy path, one approval gate, one blocked attack, and one walk through a trace from question to answer.
Definition of done: production readiness0 / 9
Pick your track · finish either one
Steps · 0 / 6 done

  1. Scope it and draft the system card

    Decide the domain and the three to five jobs the system does. Draft the system card first; its empty sections show you what's missing.

    VerifyA draft card exists, with every section either filled in or marked as work to do.

  2. Consolidate the codebase

    One repository, one model interface, config for model ids, prices, budgets and kill switches, and the MCP server, agents and retrieval as separate modules with tests.

    VerifyA new teammate can run the system locally from the README in under 30 minutes.

  3. Deploy it

    Deploy to any host you're comfortable with: a container platform, a VM, or serverless functions plus a hosted vector store. Keep secrets in the environment, add health checks and rate limits, and keep a staging environment.

    VerifyStaging and production URLs both work, and no secret appears anywhere in the repository.

  4. Gate it

    Run the Project 5 smoke suite on every pull request, the full suite nightly, and the Project 7 attack set nightly against staging. Block merges on per-case regressions.

    VerifyA pull request that breaks one previously passing case, or re-enables one attack, fails CI.

  5. Watch it

    Turn on traces in production, publish the five-metric dashboard, and set alerts for cost, error rate and refusal rate. Replay a day of synthetic traffic at small scale to check budgets.

    VerifyThe dashboard shows the replay, the costs match your budget within a margin you can explain, and a forced error spike fires an alert.

  6. Demo it and publish the card

    Record the five-minute demo, finish the system card with real numbers, and publish both with the repository.

    VerifySomeone who reads only the card and watches the demo can say what the system does, how well, and what it does on its worst day.
· Tick off the 6 step(s) above.
Vocabulary in this lesson
ModelThe trained network itself: billions of parameters that turn a context into the next token, and nothing more.ToolA named capability the harness lets the model invoke, such as reading a file, running a shell command or fetching a page.MCPModel Context Protocol: an open standard for plugging external tool servers into any agent harness.AgentA model wired to tools and a loop: it reads, acts, checks the result, and goes again until the task is done or you stop it.Tool callThe model's request to run a tool: a structured message naming the tool and its arguments, which the harness then executes.Model provider requestA single API call: the harness ships the full context to the provider and gets one reply back. Most turns need several of them.

Plain-English definitions from the AI Coding Dictionary.