Context and retrieval

If the model is fixed and context is your leverage, the central engineering problem is: of everything my system knows, which few thousand tokens does this request need? Answering that at runtime is retrieval; RAG is simply the pattern of doing it before every call — fetch what is relevant, place it in the context, ask the model to answer from it. The pipeline is short, but every joint is a real decision.

The pipeline, joint by joint

• Chunking. Documents are split into retrievable pieces — commonly a few hundred tokens with some overlap — and this is where quality is won or lost before any AI is involved: split mid-thought and no retriever can save you. Structure-aware splitting (headings, functions, paragraphs) beats fixed sizes; attaching a little metadata (title, section) to each chunk pays at prompt-assembly time.
• Embedding + index. An embedding model maps each chunk to a vector so that “nearest vectors” approximates “related meaning”; an ANN index (HNSW is the name you will see) makes nearest-neighbour search fast at millions of chunks. Distances are only comparable within one model — re-embed everything when you change it.
• Search — hybrid, in practice. Embeddings fail predictably on exactly the things keyword search handles trivially: identifiers, part numbers, names, rare jargon. Production systems therefore run lexical (BM25) and vector search side by side and merge results (reciprocal rank fusion is the standard trick). Twenty years of information retrieval did not become obsolete; it became a component.
• Rerank, then assemble. A cheap search over-fetches (say 50 candidates); a cross-encoder reranker — which reads query and chunk together, more accurately and more expensively — picks the best handful. Those survivors are assembled into the prompt with their sources, under an explicit token budget.

Measure the R before blaming the G

The discipline separating working RAG from demo RAG is measuring the halves separately. Retrieval has boring, honest metrics: recall@k (was the gold passage in the top k?) and MRR (how high did it rank?). Build a small labelled set — even 50 real questions with known source passages — and you can tune chunking, k, and hybrid weights without touching the model at all. In practice, when a RAG system is bad, it is usually the R: the right passage never reached the context, and the best model in the world is answering a different question than you think you asked.

Assembling context that gets used

Placement matters: models attend most reliably to the beginning and end of the context, so instructions go first, the most relevant material near the question, and mid-context is where marginal chunks go to be ignored (“lost in the middle” is the literature’s name for it). Ask for answers with citations to the provided sources and instruct honestly for the miss case (“if the sources do not contain the answer, say so”) — grounding plus a licensed “I don’t know” removes a large share of hallucinations by construction. Long context windows change the economics — sometimes you can ship the whole manual — but not the principle: a million tokens of prefill is paid in latency and money (Stage 1), and deliberate placement outperforms bulk inclusion. Retrieval is how you stay deliberate.