Learn RAG From Scratch – Python AI Tutorial from a LangChain Engineer

Key Insights

1. 1

   Why RAG exists

   Lance Martin, a software engineer at LangChain, starts from a simple observation: most of the world's data is private, while LLMs are trained on public data. At the same time, context windows have grown from about 4,000–8,000 tokens a year earlier to as many as a million tokens, so feeding private data into a model is increasingly practical. He frames retrieval-augmented generation (RAG) as three steps: index external data, retrieve documents relevant to a question, and have the LLM generate an answer grounded in what was retrieved.

2. 2

   Indexing and retrieval: embeddings and nearest neighbors

   Documents are split into chunks because embedding models have limited context windows (he cites roughly 512 up to 8,000 tokens). Each chunk is turned into a fixed-length vector that captures its meaning; in the demo, OpenAI embeddings produce vectors of length 1,536. He contrasts older sparse vectors built from word frequencies with learned embeddings. A question is embedded the same way, and retrieval becomes a nearest-neighbor search in that space, controlled by a parameter *k*, the number of chunks returned. The notebooks use Chroma as a local vector store and LangSmith to trace each step.

3. 3

   Generation: prompts and LangChain Expression Language

   Generation means "stuffing" retrieved chunks into the LLM's context window. A prompt template has placeholders such as `context` and `question`, filled from a dictionary built from the retrieved documents and the user's question. Using LangChain Expression Language (LCEL), he composes the prompt, model and output parser into a chain that supports methods like `invoke`, `batch` and `stream`. A full RAG chain wires in the retriever so retrieval happens automatically when the chain is invoked.

4. 4

   Multi-query and RAG-Fusion: rewrite the question several ways

   User questions can be ambiguous or poorly worded, and a badly phrased question may not land close to the right documents in embedding space. Multi-query asks an LLM to rephrase the question from several perspectives, retrieves for each version, and takes the unique union of the results. RAG-Fusion does the same fan-out but then applies reciprocal rank fusion to merge the separate ranked lists into one consolidated ranking. He notes this is useful when retrieving across several vector stores or when you only want the top few documents overall.

5. 5

   Decomposition: break a question into sub-questions

   Drawing on Google's least-to-most prompting, decomposition splits a question into smaller sub-problems. In one approach, each sub-question is answered in turn, with earlier question-and-answer pairs plus fresh retrieval fed into the next step. He relates this to work on interleaving retrieval with chain-of-thought reasoning. In the other approach, sub-questions are answered independently and the answers concatenated, which he suggests suits questions whose parts don't depend on each other.

6. 6

   Step-back prompting: ask a more abstract question

   Step-back prompting, also from Google, goes in the opposite direction from decomposition: it uses few-shot examples to generate a more general "step-back" question. In his demo, a question about task decomposition for LLM agents becomes a broader question about the process of task decomposition in general. The system retrieves for both the original and the step-back question and uses both sets of context. Martin suggests it can help in domains with a lot of underlying conceptual knowledge, such as textbooks or technical documentation.

7. 7

   HyDE: search with a hypothetical document

   HyDE starts from the point that questions and documents are very different kinds of text: questions are short and possibly ill-worded, documents are long and dense. So an LLM first writes a hypothetical passage that answers the question, and that passage, rather than the raw question, is embedded and used for retrieval. The idea is that a hypothetical document may sit closer to the real target document in embedding space. He notes the generation prompt can be tuned for a particular domain.

8. 8

   Routing: logical vs. semantic

   Routing sends a question to the right destination, such as a particular vector store, a relational database, a graph database, or a specific prompt. Logical routing lets the LLM reason about the available sources and return a structured output, using function calling and a Pydantic schema constrained to a fixed set of options; his example routes between Python, JavaScript and Go documentation. Semantic routing instead embeds a set of prompts and picks the one most similar to the question; his example chooses between a physics prompt and a math prompt.

9. 9

   Query construction: natural language to a query language

   Query construction converts a natural-language question into the domain-specific language of a data source. Examples include text-to-SQL for relational databases, text-to-Cypher for graph databases, and metadata filters for vector stores. In the demo, an index of YouTube tutorial transcripts has fields like publish date, view count and length. The available filters are described in a schema object that is bound to the LLM through structured output, so a question like "videos on chat LangChain published after 2024" becomes a structured query with content search plus a date filter.

10. 10

    Multi-representation indexing: retrieve with a summary, return the full document

    Building on the idea of proposition indexing, Martin decouples what you retrieve on from what you give the LLM. An LLM writes a summary of each document; the summaries are embedded in a vector store, while the full documents are kept in a separate document store, linked by a document ID through LangChain's multi-vector retriever. A search matches against the summary but returns the whole document, which he says suits long-context models that can handle full documents during generation.

11. 11

    RAPTOR: a hierarchical index of summaries

    Some questions need a detail from a single chunk; others need information spread across many chunks, possibly more than *k* would return. RAPTOR addresses this by clustering documents, summarizing each cluster, and repeating the process recursively up to a single high-level summary. All the leaves and summaries are indexed together, so high-level questions match the summaries and low-level questions match the detailed chunks. He applied it to about 30 LangChain Expression Language docs (most under 4,000 tokens) without splitting them, using three levels of recursion.

12. 12

    ColBERT: token-level embeddings

    Compressing a whole document into one vector can be overly restrictive, so ColBERT produces an embedding for every token in the document and in the question. For each question token, it finds the maximum similarity against all document tokens, and the document's score is the sum of those maximums. Martin says it reports strong performance but that latency and production readiness need checking. He demos it with the RAGatouille library on a Wikipedia page about Miyazaki, and wraps it as a LangChain retriever.

13. 13

    Corrective RAG as a LangGraph state machine

    For "active RAG," where the LLM decides when and where to retrieve, Martin uses LangGraph to build a state machine. The state is a dictionary holding the question, documents and generation; each node is a function that modifies that state, and conditional edges decide what happens next. Following the corrective RAG (CRAG) paper, he retrieves documents, grades each one with a binary yes/no grader backed by a Pydantic schema, and, if any are irrelevant, rewrites the query and runs a web search with Tavily before generating. He calls this "flow engineering" and says ten minutes spent designing the flow up front is very valuable.

14. 14

    Adaptive RAG: routing plus inline "unit tests"

    Combining query analysis with checks during inference, his adaptive RAG flow routes each question to a vector store, web search, or a plain LLM fallback. It then runs three graders: document relevance, whether the answer is grounded in the documents (hallucination check), and whether it actually answers the question. Failing a check sends the flow back to web search or regeneration. He cites Hamel Husain's writing on using assertions for automatic retries during model inference. Using Cohere's Command R, the whole run took about six to seven seconds. He argues LangGraph suits well-defined, reliable flows that can use smaller, faster models, while agents suit open-ended problems but are less reliable and lean on larger models.

15. 15

    Is RAG dead? Long context changes it rather than replacing it

    In a talk he gave at San Francisco meetups, Martin tests whether long-context models can replace retrieval. With Greg Kamradt, he ran a "multi-needle" test, hiding three pizza ingredients (figs, prosciutto and goat cheese) in a long context. With GPT-4 at 120,000 tokens, retrieval got worse as the number of needles grew (about 60% for 10 needles), got worse again when the model also had to reason about them, and needles near the start of the context were missed more often, possibly due to recency bias. He also notes costs: roughly $1 per 100,000-token GPT-4 generation, plus latency, no ability to audit retrieval, and harder access control. His conclusion, borrowing Frank Zappa's "Jazz isn't dead, it just smells funny": RAG will change, shifting toward full-document retrieval and cyclic flows that grade and correct, while query analysis, routing and query construction stay relevant.

Chapter Breakdown