Learn RAG From Scratch: Lance Martin's Guide to Retrieval-Augmented Generation

Most of the world's data is private, but LLMs are trained on public data. Retrieval-augmented generation (RAG) bridges that gap, and in this freeCodeCamp course, LangChain software engineer Lance Martin builds it up piece by piece, from a basic pipeline to self-correcting flows. Here's a condensed tour of the techniques he covers, and why he thinks long-context models will change RAG rather than replace it.

This post condenses the course. The explanations, demos and results are Martin's.

The Basic Pipeline: Index, Retrieve, Generate

Martin frames RAG as three steps. Indexing prepares documents so they can be found later. Because embedding models have limited context windows (he cites roughly 512 to 8,000 tokens), documents are split into chunks, and each chunk is converted into a fixed-length vector that captures its meaning. In his demo, OpenAI embeddings produce vectors of length 1,536. He contrasts these learned embeddings with older sparse vectors built from word counts.

Retrieval embeds the user's question the same way and runs a nearest-neighbor search: chunks that sit close to the question in embedding space are likely to be about the same thing. A parameter k sets how many chunks come back.

Generation puts those chunks into the LLM's context window through a prompt template with slots for context and question. Martin builds this with LangChain Expression Language (LCEL), chaining a prompt, a model and an output parser, then adds the retriever so retrieval runs automatically. Throughout, he uses LangSmith traces to show exactly what went into and came out of each step.

Query Translation: Fixing the Question First

The first family of advanced techniques rewrites the user's question, because a poorly worded question may not land near the right documents in embedding space.

• Multi-query asks an LLM to rephrase the question from several perspectives, retrieves for each version, and keeps the unique union of results.
• RAG-Fusion does the same fan-out, then uses reciprocal rank fusion to merge the separate ranked lists into one ranking.
• Decomposition, inspired by Google's least-to-most prompting, breaks a question into sub-questions. They can be answered in sequence, with each answer feeding the next, or answered independently and combined when they don't depend on each other.
• Step-back prompting, also from Google, goes the other way and generates a more abstract version of the question, then retrieves for both. Martin suggests it helps with conceptual material like textbooks or technical documentation.
• HyDE has an LLM write a hypothetical passage that answers the question, then searches with that passage instead of the question. The reasoning: questions are short and documents are long, so a fake document may sit closer to the real one.

Routing and Query Construction

Once the question is in good shape, it has to reach the right place. Martin shows two kinds of routing. Logical routing lets the LLM choose a destination and return a structured answer, using function calling and a Pydantic schema limited to fixed options; his example picks between Python, JavaScript and Go documentation. Semantic routing embeds a set of prompts and picks the one most similar to the question, such as a physics prompt versus a math prompt.

Query construction converts natural language into whatever query language the source uses: text-to-SQL for relational databases, text-to-Cypher for graph databases, or metadata filters for vector stores. In his demo, a question about LangChain videos published after 2024 becomes a structured query with a content search plus a date filter, using a schema that describes the index's available fields.

Better Indexing: Summaries, Trees and Token-Level Embeddings

Martin then covers three ways to index more cleverly.

Multi-representation indexing separates what you search on from what you hand the model. An LLM writes a summary of each document; the summaries go into a vector store and the full documents into a document store, linked by an ID. A search matches the summary but returns the whole document, which he says suits long-context models.

RAPTOR builds a hierarchy. It clusters documents, summarizes each cluster, and repeats until it reaches one top-level summary, then indexes every level together. Detailed questions match the original chunks, and broad questions match the summaries. He applied it to about 30 LCEL documents without splitting them.

ColBERT avoids squeezing a document into a single vector. It embeds every token in the document and the question. For each question token it finds the best-matching document token, and the document's score is the sum of those best matches. Martin says it reports strong performance but that latency and production readiness need checking, and demos it with the RAGatouille library.

Active RAG With LangGraph

The most hands-on section moves from linear chains to flows where the LLM checks its own work. Using LangGraph, Martin builds RAG as a state machine: a dictionary holds the question, documents and answer; each node is a function that updates that state; and conditional edges decide where to go next.

His first example implements corrective RAG (CRAG). Retrieved documents are graded by a yes/no grader backed by a Pydantic schema. If any are irrelevant, the flow rewrites the query and runs a web search with Tavily before generating. He calls this "flow engineering" and says ten minutes spent designing the flow up front is very valuable.

His second example, adaptive RAG, adds routing to a vector store, web search, or a plain LLM fallback, then runs three checks: are the documents relevant, is the answer grounded in them, and does it actually answer the question? Failures send the flow back to web search or regeneration. Using Cohere's Command R, the whole run took about six to seven seconds. He argues LangGraph suits well-defined, reliable flows that can use smaller, faster models, while agents suit open-ended problems but are less reliable and depend on larger models.

Is RAG Dead?

The course closes with a talk Martin gave asking whether million-token context windows make retrieval obsolete. With Greg Kamradt, he ran a "multi-needle" test, hiding three pizza ingredients in a long context. With GPT-4 at 120,000 tokens, results got worse as the number of needles grew (about 60% retrieved for 10 needles), got worse again when the model also had to reason about them, and needles near the start of the context were missed most. He adds practical costs: roughly $1 per 100,000-token GPT-4 call, higher latency, no way to audit retrieval, and harder access control.

His verdict borrows Frank Zappa's line that jazz isn't dead, it just smells funny. RAG will shift away from finely tuned chunking toward retrieving full documents and toward cyclic flows that grade and correct themselves. Routing, query analysis and query construction, he says, remain just as relevant.

Conclusion

Martin's course is a map of the RAG design space: fix the question, send it to the right place, index in a way that fits your documents, and build in checks so the system can catch its own mistakes. Each technique comes with a notebook demo, so the full video is worth following along with.

---

Originally published on freeCodeCamp.org. Watch the full episode: https://www.youtube.com/watch?v=sVcwVQRHIc8