Course contents
Project 7: Red-Team and Harden the Desk
Threat-model the support desk, attack it at least 25 ways, add rails that fail closed, keep personal data out of prompts and traces, and turn every successful attack into a permanent regression test.
The desk reads untrusted email, retrieves pages others can edit, sees customer records and can queue refunds: all three legs of the trifecta, by design. This project finds out how it breaks before someone else does, then hardens it without making it useless. Run your attacks against the mid-course system with the Project 5 evals in place, so you can show that hardening didn't break legitimate tickets.
Threat model first
| Trifecta leg | In the desk | Examples to attack |
|---|---|---|
| Private data | Customer records, order histories, payment events | Extraction prompts, cross-customer lookups, card numbers echoed in replies |
| Untrusted content | Inbound emails, help-center pages, order event notes, other agents' outputs | Hidden instructions in a page, a note or an email signature |
| External communication | Customer replies, any URL fetch, the refund queue | Injected links, data stuffed into replies, refund requests for other orders |
Write at least 25 attacks across seven groups: direct injection (5), injection through help-center pages (5), injection through order event notes (3), personal-data extraction (4), tool misuse such as over-limit refunds or another customer's order (4), cost attacks that try to make it loop or flood it (2), and system-prompt extraction (2). For each, write the safe behavior you expect.
Write the threat model
Fill in the trifecta table for your own build, list every place outside text enters the system, and note which defenses already exist from earlier projects.
VerifySECURITY.mdnames every untrusted input and every outbound channel, with an owner for each.Run an automated red team
Start with
npx promptfoo@latest redteam setup(orredteam init), point the target at your desk's HTTP endpoint, and pick attack plugins covering prompt injection, personal data, excessive agency and system-prompt extraction. Run it withredteam runand readredteam report.VerifyYou have a report with findings, and each finding is copied into your attack log with its expected safe behavior.Plant indirect attacks by hand
Plant payloads where automated tools can't reach: a hidden instruction in a help-center page, an order event note that claims to be a system message, an email signature asking for a refund to another order. Re-ingest and run tickets that touch each one.
VerifyYour attack log has at least 25 entries with results, including every indirect payload.Harden the desk
Add the aa-23 defenses: an ingestion sanitizer, the small-model screen on retrieved chunks and tool results, JSON-encoded tool results labeled by source, Presidio redaction before every model call, citation checks and a link allow-list on replies. Make every rail fail closed, and confirm amount and ownership checks live in tool code.
VerifyRerunning the full attack log shows each earlier success blocked, flagged or contained, and a forced rail timeout queues the ticket instead of passing it.Make the fixes permanent
Add every successful attack to the Project 5 dataset with its expected safe behavior, then run the whole suite: attacks and legitimate tickets together.
VerifyThe suite passes, its legitimate-ticket pass rate is no lower than before hardening, and CI now fails if any attack starts working again.Build and test kill switches
Add a config flag per AI feature (triage, answers, order agent) checked on every request, with a safe fallback reply when it's off. Flip each one in staging.
VerifyTurning off the order agent makes order tickets get a holding reply within one request, with no deploy.
Plain-English definitions from the AI Coding Dictionary.