Agentic AI Engineering/Grounding: RAG & Context Engineering
Lesson 5 / 5 · 180 min
Course contents
Project

Project 2: Grounded Help-Center Assistant

Give the support desk a memory of the truth: a RAG assistant over Acme's help center with hybrid retrieval, citations, honest refusals and a 20-question eval. Project 1's triage routes how-to and policy tickets here.

Project 1 sorted tickets. Now the desk starts answering them. Policy and how-to questions (returns, refunds, shipping times, warranty) should be answered from Acme's help center and nowhere else, with a citation for every claim and a plain refusal when the help center doesn't say. Measured retrieval quality is part of the deliverable, not an afterthought.

The help center

Write six to ten short pages in Markdown, or have a model draft them and then edit them into a consistent policy: returns, refunds, shipping, lost parcels, warranty, accounts, payments, gift cards. Be specific (numbers, steps, exceptions), because specific docs make testable answers. Include at least one page with identifiers, such as a table of payment error codes like PAY-402, so hybrid search has something to prove. Here's a sample page to start from:

docs/returns.md
# Returns and refunds

Returns are free within 30 days of delivery for unused items in their original packaging. Used gear can be returned within 30 days only if it is defective. To start a return, open Orders, choose the item, and select Start a return. Print the prepaid label and drop the parcel at any carrier location. Refunds go back to the original payment method and usually arrive within five business days after the return is approved. Store credit is issued instantly instead if you choose it when you start the return. Items marked final sale cannot be returned. If a parcel never arrived, do not start a return; contact support and we will open a lost-parcel claim with the carrier. Gift recipients can exchange items, but refunds go to the original buyer.

The eval set

Write 20 questions before you build: 12 answerable (note the page and section that answers each), 3 that use exact identifiers, 2 conversational follow-ups (“and for gift orders?”), and 3 the help center doesn't answer at all. For each answerable question, record the source passage so you can score retrieval separately from the answer.

Definition of done0 / 6
Pick your track · finish either one
Steps · 0 / 6 done

  1. Ingest the help center

    Reuse the chunker and the re-runnable upsert from lesson aa-08. Store source and section headings as metadata, and keep a parallel list of chunk texts and ids for the keyword index in the next step.

    VerifyRunning ingestion twice leaves col.count() unchanged, and one printed chunk shows its source and section.

  2. Add keyword search and fuse the rankings

    Build a BM25 index over the same chunks (the rank-bm25 package is enough at this size), run both retrievers for each question, and fuse their ranked id lists with reciprocal rank fusion.

    python
    from rank_bm25 import BM25Okapi

bm25 = BM25Okapi([t.lower().split() for t in chunk_texts])

def keyword_ids(question: str, k: int = 20) -> list[str]:
    scores = bm25.get_scores(question.lower().split())
    best = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:k]
    return [chunk_ids[i] for i in best]

def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)
    VerifyAn identifier question such as “What does PAY-402 mean?” finds the error-code page in the fused top 5, and you can show it missing from the vector-only top 5.

  3. Answer with citations and refusals

    Send the top five fused chunks, each tagged with its source, to Claude Haiku 5.5 (or gemma4 through Ollama's OpenAI-compatible endpoint) with the grounded system prompt from aa-08.

    VerifyEach of the three uncovered questions returns the exact refusal line, and every other answer carries at least one citation that points to a real file.

  4. Score retrieval and answers

    For each question, record whether the labeled source passage is in the top 5 (recall@5) and whether the answer passes: correct, cited, or an honest refusal where one is expected. Run the eval for vector-only and for hybrid retrieval, and keep both results.

    VerifyYou have two rows of numbers (vector-only and hybrid) for recall@5 and pass rate, and hybrid is no worse on any question that passed before.

  5. Try a reranker and keep it only if it earns its place

    Retrieve 30 fused candidates, rerank them with an open cross-encoder such as BAAI/bge-reranker-v2-m3 through sentence-transformers' CrossEncoder, keep the top 5, and rerun the eval. Note the added latency per question.

    VerifyYou can state the reranker's effect in one sentence with numbers, for example: “recall@5 from 80% to 90%, plus 300 ms a question”.

  6. Connect it to triage

    When Project 1's triage labels a ticket as a how-to or policy question, call answer() with the ticket text and attach the cited answer to the triage record as a suggested reply.

    VerifyTicket 5 from Project 1 (“Do you ship to Iceland?”) gets a cited answer or an honest refusal, depending on what your shipping page says.
· Tick off the 6 step(s) above.
Vocabulary in this lesson
TicketA bounded unit of work with a clear done state, sized so an agent can finish it in one session.ModelThe trained network itself: billions of parameters that turn a context into the next token, and nothing more.ToolA named capability the harness lets the model invoke, such as reading a file, running a shell command or fetching a page.System promptThe standing instructions the harness puts in front of every request: who the agent is, what tools it has, how it should behave.

Plain-English definitions from the AI Coding Dictionary.