Build a RAG Pipeline
Ship a minimal, honest RAG system: ingest, retrieve, answer with citations, and eyeball-test it before you trust it.
Ingest: chunk and embed your documents
Start with one folder of markdown/text. Chunk on structure with overlap, embed, and store. Chroma keeps this to a few lines locally (swap for pgvector/Qdrant in production — the shape is identical):
import chromadb from chunkers import split_markdown # headings-aware, ~500 tokens, 60 overlap db = chromadb.PersistentClient("./index") col = db.get_or_create_collection("docs") for path, text in load_docs("./docs"): for i, chunk in enumerate(split_markdown(text)): col.add(ids=[f"{path}#{i}"], documents=[chunk.text], metadatas=[{"source": path, "heading": chunk.heading}])Verifycol.count() roughly matches docs × chunks-per-doc; spot-check one stored chunk reads as a self-contained passage.Retrieve: top-k with metadata
Embed the query, pull the nearest chunks, and keep the metadata — you'll need it for citations and for debugging what the model saw:
hits = col.query(query_texts=[question], n_results=5) context = "\n\n".join( f"[{m['source']} — {m['heading']}]\n{d}" for d, m in zip(hits["documents"][0], hits["metadatas"][0]) )VerifyPrint the five chunks for a test question — a human judges them relevant BEFORE any model sees them.Answer: grounded, cited, allowed to say no
The generation prompt does three jobs: answer only from context, cite sources inline, and refuse honestly when the context doesn't contain the answer:
prompt = f"""Answer using ONLY the context below. Cite the [source] you used. If the context does not contain the answer, say exactly: "I don't find that in the docs." <context> {context} </context> Question: {question}"""VerifyAsk something you KNOW isn't in the docs — the correct behavior is the refusal line, not a plausible improvisation.Eyeball-test with a starter question set
Before trusting it, write 15 questions with known answers: 10 answerable (note which doc), 5 unanswerable. Run all 15 and score by hand: right answer? real citation? honest refusal? This scrappy table is your first eval — Module 5 grows it into a golden dataset with automated judges.
VerifyYou have a 15-row table of question / expected / actual / pass. Anything under ~12 passing means fix retrieval before touching prompts.