1.6 · Retrieval and RAG

1.6 · 30 min

Retrieval and RAG

A founder asks you to build "a chatbot that knows our company docs", and before the sentence is finished, someone has already said the word "RAG" and someone else is pricing vector databases. Slow down. Retrieval-augmented generation is one of four ways to get knowledge into a model, it is the most complex of the four, and in 2026 it is chosen by default far more often than it is chosen correctly. This lesson teaches you how RAG actually works, mechanically, and then the decision framework that tells you when not to build it, which is most of the time.

The problem, stated precisely

Lesson 1.3 established that a model knows only what is in its weights (frozen at cutoff) or in its context window (finite, attention-taxed). The problem RAG addresses: knowledge that is too big for the window and too specific or too fresh for the weights. A quarter million support tickets. A decade of case law. Every deliverable your agency ever shipped. You cannot stuff it in, and the model was never trained on it. Something has to select the relevant slice at question time. That selection machinery is retrieval; RAG is retrieval feeding generation.

The pipeline, stage by stage

INGEST TIME (on write)DOCSknowledgeCHUNKsplit smallEMBEDvectorsINDEXvectors + keywordsstore bothQUERY TIME (per question)QUERYa questionRETRIEVEhybrid searchtop candidatesRERANKkeep best kscore, filterGENERATEcite sourcestop chunks onlyeval loopgolden queries gate every changefailuresfix chunkingactiondata

Chunking. Documents get split into pieces, per Anthropic's contextual-retrieval guidance usually no more than a few hundred tokens each, because retrieval selects chunks, not documents. This unglamorous step makes or breaks the system. Chunk boundaries that sever meaning produce fragments like "the company's revenue grew by 3% over the previous quarter", which company? which quarter?, that no search can rescue, because the information needed to match the question was in the sentence above the cut. Anthropic's fix, contextual retrieval, prepends a short generated context to each chunk before indexing ("This chunk is from ACME Corp's Q2 2023 SEC filing..."), which alone cut top-20 retrieval failures by 35% in their published benchmarks.

Embedding. Each chunk goes through an embedding model that maps text to a vector, coordinates in a space where distance tracks meaning. "Refund policy" and "returning a purchase" land close together despite sharing no words. Note a fact that surprises people: Anthropic does not offer an embedding model. Its docs point to third-party providers, prominently Voyage AI (the voyage-4 series, 32k-token input, 1024-dimension default, as of July 2026). Your RAG stack is multi-vendor from day one.

Storage and search. Vectors go in an index that answers "nearest neighbors to this query vector" fast. But semantic similarity alone misses things exact matching catches: error codes, SKUs, function names, "TS-999" as Anthropic's example goes. The older lexical algorithm BM25 (term-frequency based, exact-match biased) catches those. Production systems run hybrid search: embeddings and BM25 in parallel, results merged. Anthropic's numbers: contextual embeddings plus contextual BM25 cut top-20 retrieval failure by 49% (5.7% to 2.9%) versus standard embedding-only RAG.

Reranking. First-pass retrieval is fast and sloppy: it over-fetches (Anthropic's setup pulls the top 150 candidates), and a reranking model then scores each against the query properly, keeping the best 20 for the prompt. Reranking plus everything above reached a 67% reduction in retrieval failures in the same benchmarks. Cost: an extra model call of latency per query. Worth it almost always, because irrelevant chunks in the prompt are not neutral filler; they are attention-thieves (1.3) and hallucination seeds.

Generation. The winning chunks go into the prompt with the question, ideally with the model instructed to cite which chunks support which claims, so retrieval failures surface as visible non-answers instead of confident fabrications.

Every stage is a quality dial, which is the honest reason RAG earns its reputation for disappointment: it is a pipeline where five mediocre stages multiply into one bad product, and where the eval loop drawn around the figure, golden queries with known-correct sources, run after every change, is the only way to tune any dial without breaking another. That eval harness arrives in 3.7, and the full build is elective A8.

The decision framework

Four ways to give a model knowledge. In order of increasing complexity:

1. Long context: just put it in the window. Anthropic's contextual-retrieval post says this plainly: if your knowledge base is under roughly 200,000 tokens, about 500 pages, skip RAG and include the whole thing, with prompt caching making repeat calls fast and cheap. An employee handbook, one product's docs, a policy manual: they fit. The simplest thing that works has zero retrieval failures, no index to go stale, and no pipeline to tune.

2. Agentic search: let the agent go read. When knowledge lives in a navigable structure, a codebase, a wiki, a file system, an agent with grep and read tools retrieves just in time (1.3), following the structure the way you would. Anthropic's context-engineering guidance describes Claude Code itself working this way, bypassing "the issues of stale indexing" that plague embedding pipelines: nothing to re-embed when a file changes, and metadata like paths and naming carries signal an embedding flattens. The honest tradeoff: it spends loop iterations and tokens per question, so it is slower per query than a tuned index. For coding agents, agentic search beating naive RAG stopped being a hot take somewhere around 2025; treat it as the default for structured, navigable corpora.

3. RAG: build the pipeline. Earns its complexity when all three hold: the corpus is genuinely large (multiples of the window, not 1.2x), it has no navigable structure an agent could walk (a hundred thousand flat documents), and you need runtime queries with sub-second-ish latency at scale. Legal discovery, support-ticket history, large product catalogs. That is a real and valuable niche. It is a niche.

4. Fine-tuning: change the weights. Almost never the answer for knowledge, and this course will say so bluntly. Fine-tuning teaches patterns and style, per Anthropic's glossary it makes the model mimic the tuning dataset, not facts you can update; your data changes tomorrow and your weights do not. Anthropic's API does not offer fine-tuning as of July 2026 (the glossary states this directly). If someone proposes fine-tuning so the model "knows the docs", the proposal is confused about what fine-tuning does.

The position, explicitly: RAG is for large, queryable knowledge at runtime. It is not a default. Reach for it fourth, after the window, after agentic search, and after asking whether the product needs runtime retrieval at all. When you do reach for it, build it with the evals on from day one, or you will be tuning five dials blindfolded. Elective A8 does exactly that build, production-grade, on the Track A capstone.

Lablab-1-6Five scenarios, four options, defend your call

Goal: Choose long context, agentic search, RAG, or fine-tuning for five realistic scenarios and defend each choice in writing.

For each scenario, write: your choice, the two decision factors that drove it, and what change in the scenario would flip your answer. Commit the writeup to the practice repo as retrieval-decisions.md, then check the reference answers.

  1. A 12-person agency wants an internal assistant answering questions from their operations handbook: about 140 pages, updated quarterly.
  2. A coding agent needs deep knowledge of your 300,000-file monorepo to answer "where is X handled and what breaks if I change it" questions.
  3. A legal-tech product must answer questions across 250,000 past case documents, with citations, for hundreds of concurrent users at interactive latency.
  4. A B2B SaaS founder wants the assistant to "write exactly in our brand voice" and suggests fine-tuning on 40 marketing emails.
  5. A support bot must answer from 3,000 help-center articles and 200,000 historical tickets; articles change weekly, and answers must cite current article text.

Verify

  • retrieval-decisions.md committed, with a choice, two factors, and a flip condition for all five scenarios.
  • Your calls match the reference answers below on scenarios 1 through 4; scenario 5 is legitimately arguable, and what matters is that your defense addresses freshness and corpus split.
>Troubleshooting
  • Reference answer 1: long context. 140 pages sits under the roughly 200k-token line from Anthropic's guidance; prompt caching handles cost. Flip: the handbook grows past the window, or per-query cost at high volume argues for retrieval.
  • Reference answer 2: agentic search. Highly structured, constantly changing corpus; an embedding index of a monorepo is stale by lunchtime, and paths/names carry meaning. Flip: needing sub-second answers at high query volume without an agent loop, which starts to argue for a code-aware index alongside.
  • Reference answer 3: RAG, hybrid plus reranking. Huge flat corpus, no navigable structure, hard latency and citation requirements: the exact niche. Flip: corpus shrinks to fit the window, or queries become rare enough that an agent reading per-question is acceptable.
  • Reference answer 4: none of the four as proposed; this is prompting. Voice is a style pattern: a style guide plus 3 to 5 example emails in the prompt (1.4) gets there. Fine-tuning on 40 examples is both unavailable on Anthropic's API and the wrong tool sized absurdly. Flip: tens of thousands of examples, a provider that tunes, and a measured gap prompting cannot close.
  • Reference answer 5: the defensible split is RAG over the tickets (large, flat, historical) and either long context or re-indexed-on-publish retrieval over the 3,000 articles, with freshness handled by re-embedding on every publish. Pure RAG over everything is acceptable if the writeup addresses how stale embeddings of edited articles are avoided; ignoring freshness is the failing answer.

Where it breaks

The classic RAG failure is silent: retrieval returns plausible-but-wrong chunks, generation weaves them into a fluent answer, and no error appears anywhere. Users just get confidently wrong answers at some percentage. This is why citations back to sources are not a nice-to-have (a wrong citation is checkable; an uncited claim is not) and why the golden-query eval is load-bearing: without it you cannot even measure the failure rate you are shipping.

The equal and opposite failure is the one this lesson exists to prevent: three weeks building a vector pipeline for knowledge that fit in the prompt. That mistake does not fail loudly either. It just costs a month and a maintenance surface, and works slightly worse than pasting the docs would have.

Knowledge check

Knowledge check

Q1A client's knowledge base is 120 pages of policy documents. The team is evaluating vector databases. What do you tell them, and on what basis?
Q2Your RAG system retrieves well in demos but misses queries containing exact error codes like 'TS-999'. Which pipeline change targets this directly?
Q3Why does the pipeline over-fetch (say, 150 candidates) and then rerank down to 20 instead of just retrieving the best 20 directly?
Q4For a coding agent working in a large, actively developed monorepo, why does this course position agentic file search above embedding-based RAG as the default?

Checkpoint 1: the autopsy

CheckpointUnder-specify, autopsy, correct

This checkpoint integrates everything Part 1 built: the loop, the window, the prompt, and the failure taxonomy, on a task you sabotage on purpose.

In your practice repo, dispatch something real but under-specified, in the "add rate limiting" or "improve the error handling" genre: a goal with no scope, no constraints, no verification. Let it run to completion without steering. Then perform the autopsy: work through the transcript and diff with the 1.5 taxonomy open, and name at least three failure signatures with quoted evidence (an unrequested dependency, a weakened assertion, an invented API, a plan built on a stale comment). Trace each to its cause using Part 1's vocabulary. Then write the corrected dispatch, 1.4-grade: scope, constraints, verification defined up front. Run it clean and collect the evidence.

Passing means you can reliably produce the failure, name it, explain it, and prevent it. If you cannot check every box honestly, do not check it: Part 2 hands you sharper tools, and unverified ground under a taller stack is how the cold open of Lesson 0.1 happens to you.

Sources