## Introduction This is Part 2 of my series on building RAG systems. Part 1 covered the foundations — what RAG is, how the pipeline works, and touched on some of these techniques briefly. Here I want to go deeper. The goal is to give you intuition for how these methods work. Some of them I've used myself and seen work well in practice. Some I haven't used yet — I'll be honest about that. And honestly, not all of them are equally useful. A lot depends on your data, your query patterns, and where your pipeline is actually failing. If you haven't read Part 1, it's worth a quick skim — this part assumes you're familiar with the basics. RAG diagram

## Ingest ### Extract First step is extracting text from your documents. A few tools, roughly in order of complexity: _PyMuPDF_ is fast and works well on clean text-based PDFs. It struggles with messier layouts — tables, multi-column pages, presentations. You can patch some of that with regex, but how far you get depends heavily on your documents. _Unstructured_ goes further. It cuts documents into structured elements (titles, tables, narrative text, headers) which makes the next steps — cleaning, chunking — much easier to reason about. _Docling_ is the heavy option. It runs ML models on every page, which makes it slow, but the table extraction is remarkably good. If your documents are table-heavy, it's worth the cost. As always: look at your actual extraction output before optimizing anything else. Bad extraction compounds through the entire pipeline. ### Chunk #### Semantic Chunking Then we chunk. Fixed-size chunking cuts every N characters — it can cut mid-sentence, mid-idea. Semantic chunking tries to fix that: you embed sentence by sentence, compare adjacent sentences, and split where similarity drops. The idea is that a drop in similarity signals an idea shift. The intuition is clean. In practice, it's more complicated. First, you still need a maximum and minimum chunk size. No matter how well you detect topic boundaries, a single "topic" can run for thousands of tokens — or be a single short sentence. So you end up capping by character count anyway, which partially defeats the purpose. Second, the quality of the boundaries depends heavily on your embedding model and your data. On well-structured documents — articles, documentation — it can find decent splits. On messier text, the similarity signal is noisy and the boundaries aren't obviously better than a simpler approach. Lastly, and this is important: if you extract properly, you can guide simple chunking using document structure — cut on paragraphs and sections, prepend titles, keep full tables in a single chunk. A well-extracted, structure-aware chunking strategy can beat noisy embedding-based cuts. In the end, the author decided where ideas shift and made it clear in the text — we just need to extract that signal cleanly. It's worth knowing and worth trying on your data. But in my experience, semantic chunking didn't produce a wow effect — and you still need to tune hyperparameters: min, max size, the similarity threshold. #### Hierarchical Chunking The core tension in chunking: small chunks retrieve precisely but give the LLM thin context. Large chunks give rich context but embed poorly and . Hierarchical chunking tries to get both — retrieve small, read big. *Parent/child chunks*: You store two levels: small child chunks for retrieval, and larger parent chunks for context. When a child chunk is retrieved, you pass its parent to the LLM instead. Precise search, rich answer. *A lighter approach*: I didn't implement the full parent/child store. Instead: chunks are stored with metadata (document ID, chunk number), and at retrieval time I fetch the surrounding chunks — the ones before and after the match. Same intuition, less infrastructure. Works well in practice and is much simpler to The tradeoff: surrounding context is positional, not semantic. You get what's adjacent in the document, not necessarily what's most relevant. For well-structured documents it's usually fine — LLMs are powerful enough to extract the right information from a wider window. The real problem is context window size: fetching too many surrounding chunks bloats your prompt fast. But you can use (Contextual Compression) see furher. #### LLM-Augmented Chunks The three techniques below share the same idea: run an LLM at ingest time to enrich what gets stored in your index. On Retrieval You are no longer comparing your query against the original text, but against the LLM's interpretation of it. The cost is paid once when you build the index, not on every query. They differ in what they augment and how much they touch your original text: - — rewrites original chunks into atomic facts - — generates questions per chunk, stored alongside the original - — generates summaries for routing. The main tradeoff: making an LLM work well on your documents is hard. For some it will generate good facts, questions, summaries — for others it can generalize too much, lose precision. Prompt design matters, and evaluating the output at scale is not trivial, more then that, the output you try to find out depens on how users query your system. On top of that, you can blow up your index size fast. In practice I used something between HyPE and Hierarchical Summary: for each document I generated a summary, extracted keywords, and listed possible document names — stored as a simple chunks alongside the originals. No two-level routing, no complex index. At retrieval time I gave these chunks higher weight than regular chunks. You can use **Contextual Retrieval** technique you use chunk level summary, append id directly to each chunk, and embed summary + chunk. It can smooth, and reduce semantic noise. ### Store There are basically two ways to store chunks: a vector store and a keyword store. __Vector store__: you embed the chunk semantics into vector [0.23, -0.11, 0.87, ...], store it alongside the original text and metadata. The vector captures semantic meaning and enables fast similarity search at scale. __Keyword store__: an inverted index — think . No embeddings. Stores which words appear in which chunks. Fast exact match. Works great when the user query contains specific terms, product names, codes, or IDs. In my experience, both should work together, **hybrid**. People search by exact product name, error code, contract number — keyword wins there (Give me RFC 1918 details please). People search by concept, intent, or question — vector wins (need a specification for private network ranges). Depending on your data and query patterns, one will outperform the other. ## Retrieve ### Query Enhancement The problem with embedding a raw query is that queries and documents live in different spaces. A query is short, often vague, sometimes just a bunch of keywords — "transformer attention mechanism". A document chunk is dense, structured, full of context. Even with a good embedding model, the vectors don't always land close enough to retrieve the right chunks. #### HyDE HyDE flips the approach: instead of embedding the query directly, you ask the LLM to generate a hypothetical answer first — use that instead. The hypothetical answer is longer, richer, and lands much closer in vector space to the real chunks you're looking for. In practice it helps quite a bit — especially for vague or conceptual queries. The hypothetical answer expands the signal: it introduces relevant keywords, concepts, and plausible phrasing that the query alone was missing. The tradeoff: one extra LLM call, and a prompt that's hard to get right. If the LLM generates something off — wrong domain terms, wrong framing — your retrieval suffers. One bad hypothetical sinks the whole search. #### RAG Fusion RAG Fusion addresses that directly. Instead of generating one hypothetical, you generate N variants — HyDEs, query reformulations, extracted keywords, or a mix. Run a search for each, then merge the results with . RRF surfaces chunks that appear consistently across multiple searches — not just ones that scored well in a single pass. If one variant was off, the others compensate. The signal becomes more robust than trusting any single hypothetical. #### Hybrid search Hybrid search applies the same RRF logic, but varies the search method rather than the query — vector and keyword search run in parallel, results merged with RRF. Vector wins on conceptual queries; keyword wins on exact terms — product names, contract numbers, error codes. You can combine all three dimensions simultaneously: multiple query variants across multiple search methods. Whether that complexity is worth it depends entirely on your data and query patterns. In my experience: for most client projects, queries were keyword-heavy. Elasticsearch alone performed as well as hybrid retrieval — vector search helped at the margins but wasn't transformative. If your users search by concept or intent, the balance shifts the other way. The lesson: don't assume hybrid or vector is always better. Profile your actual queries first. ### Reranking This is the single biggest quality improvement I've seen in practice. After retrieval you have a pool of candidate chunks — say 50 — ranked RRF score. That ranking is decent but noisy. A reranker takes each retrieved chunk, pairs it with the original query, and scores how likely that chunk is to actually answer the question. You reorder by that score and pass the top 10 to the LLM. The difference from embedding similarity: embeddings compare query and chunk independently — each gets its own vector, and you measure distance. A reranker looks at the query and chunk *together*, which gives it much more signal to work with. It's slower, but you're only running it on 50 candidates, not your entire index. Retrieve wide, rerank, pass the best. Simple pattern, real results. ### Contextual Control #### Contextual Augmentation Once you have your retrieved chunks, you can extend the context you pass to the LLM before or after reranking. Two approaches from earlier apply here: fetching surrounding chunks by index, or using a parent/child structure to retrieve small but read large. For some documents — a pricing page, a short spec — the entire document fits in context anyway, making this trivial. Timing matters: if you expand context before reranking, the reranker scores richer inputs but runs slower proportionally to how much you've added. If you expand after, reranking stays fast but operates on thinner context. In practice I'd expand after — rerank on the raw chunks, then widen the window only for the top results that get passed to the LLM. #### Compression On the other hand After reranking you have your top chunks — but each chunk might be 300 tokens with only 2 relevant sentences. Contextual compression runs a lightweight LLM pass over each chunk to strip the irrelevant parts before passing to the final LLM. Less noise, smaller context window usage, potentially better answers. The risk: the compression model drops something important. It's making a relevance judgment, and it can be wrong. I haven't used this in practice. The pattern I'd try: retrieve 15 → rerank to top 5 → compress → pass to LLM. But I'd only add compression if context window size is actually a bottleneck. # Conclusion You now have everything you need to feed good context into your LLM. The post got long, so I won't go deep here — Part 1 already covered the prompt side a little: LLMs pay more attention to the beginning and end of context, be precise in your instructions, and use the metadata you stored. One thing I didn't cover is Agentic RAG — honestly because I haven't built one. The short version: you give an LLM a description of your RAG system as a tool, and it decides how to use it. It can judge whether a retrieved answer is good enough and query again if not, decide which metadata filters to apply based on the question, or skip RAG entirely and go to web search instead. Worth knowing about, but a topic for another post. --- The embedding model you use for retrieval is a **bi-encoder**: it encodes query and chunk independently into vectors, then compares them. Fast — you can precompute chunk vectors at index time. But the query and chunk never "see" each other during encoding, which limits relevance precision. A **cross-encoder** (reranker) takes the query and chunk concatenated as a single input and outputs a relevance score directly. The model attends to both simultaneously — it can catch things like "this chunk uses the right words but answers a different question." ```bash # Bi-encoder (retrieval) embed("What is multi-head attention?") → [0.21, -0.10, ...] embed("Multi-head attention allows...") → [0.22, -0.11, ...] cosine_similarity → 0.91 # Cross-encoder (reranking) score("What is multi-head attention?" + "Multi-head attention allows...") → 0.94 score("What is multi-head attention?" + "Training was done on WMT 2014...") → 0.12 ``` Popular reranker models: Cohere Rerank (API), BGE-reranker, cross-encoder/ms-marco (open source). The cost: one forward pass per candidate chunk. With 15 candidates that's fine. With 1000 it's not — which is why reranking always comes after an initial retrieval step, never instead of it. RRF merges ranked lists by scoring each chunk based on its rank across all searches: ```bash score(chunk) = Σ 1 / (k + rank) ``` `k` is a constant (typically 60) that dampens the influence of very top ranks. A chunk appearing consistently across searches scores higher than one that ranked 1st in a single search. ```bash # Results across 3 query reformulations Query 1: [chunk_A rank 1, chunk_B rank 2, chunk_C rank 3] Query 2: [chunk_A rank 1, chunk_C rank 2, chunk_D rank 3] Query 3: [chunk_B rank 1, chunk_A rank 2, chunk_C rank 3] # RRF scores (k=60) chunk_A: 1/61 + 1/61 + 1/62 = 0.0492 ← wins chunk_C: 1/63 + 1/62 + 1/63 = 0.0477 chunk_B: 1/62 + 1/61 + 0 = 0.0325 ``` chunk_A consistently appears near the top across all three searches — RRF surfaces it as the most relevant result. Instead of splitting text by boundaries, you use an LLM to decompose it into atomic factual statements. Each chunk becomes a single self-contained fact: ```bash # Original text "The Transformer uses multi-head attention in three different ways. In encoder-decoder attention layers, queries come from the previous decoder layer, and keys and values come from the encoder output." # Proposition chunks "The Transformer uses multi-head attention in three different ways." "In encoder-decoder attention, queries come from the previous decoder layer." "In encoder-decoder attention, keys and values come from the encoder output." ``` Each chunk maps to exactly one idea, no noise, no dilution. But proposition chunking rewrites your document. If you need to display the original text, you have no clean way back. I prefer to stay close to the original. HyPE keeps the original text untouched and generates hypothetical questions for each chunk — questions that this chunk could answer. Those questions are embedded and stored as the retrieval index. ```bash # Original chunk — stored as-is "Multi-head attention allows the model to jointly attend to information from different representation subspaces..." # Hypothetical questions — stored as retrieval index "How does multi-head attention work?" "What is the purpose of multiple attention heads?" "How does the Transformer attend to different subspaces?" ``` At retrieval time, the user's query matches against stored questions — question-to-question matching instead of question-to-document. Queries naturally land closer to other queries than to dense document text. The questions are stored as separate vectors, each with a `chunk_id` pointing back to the original chunk. ```bash # Vector store { embedding: embed("How does multi-head attention work?"), chunk_id: "chunk_42" } { embedding: embed("What is the purpose of multiple attention heads?"), chunk_id: "chunk_42" } # Original chunk fetched on match { chunk_id: "chunk_42", text: "Multi-head attention allows the model..." } ``` Here, you generate a summary — at document level, section level, or chunk level — and store it as the retrieval vector. **Document or section level** — useful when your index is large and queries are broad. Search summaries first to find the right document, then search detailed chunks within it. ```bash # Summary index { doc_id: "doc_42", embedding: embed(summary), summary: "This paper introduces the Transformer..." } # On match → search detailed chunks within doc_42 only { doc_id: "doc_42", chunk_id: "chunk_7", text: "Multi-head attention allows..." } ``` **Chunk level** — the summary becomes the retrieval vector, but the original chunk text is returned on match. Cleaner signal for retrieval, full fidelity for the LLM. ```bash # What gets embedded (for retrieval) summary: "The paper describes three uses of multi-head attention in the Transformer." # What gets passed to the LLM (original) "The Transformer uses multi-head attention in three different ways. In encoder-decoder attention layers, queries come from the previous decoder layer, and keys and values come from the encoder output..." ``` At chunk level it's close in spirit to HyPE — both keep original text intact and use LLM-generated content only for matching. The difference: HyPE generates questions (query-shaped), summaries are document-shaped. Questions tend to match user queries better; summaries tend to be more faithful to the chunk content. **Step 1 — Generate a hypothetical answer** ```bash # User query "How does multi-head attention work?" # Hypothetical answer generated by LLM "Multi-head attention allows the model to jointly attend to information from different representation subspaces. For each head, queries, keys, and values are projected into a lower-dimensional space, attention is computed independently, and the results are concatenated and projected back. This allows the model to capture different types of relationships in parallel." ``` **Step 2 — Embed the hypothetical answer** You embed the generated answer, not the original query. The resulting vector is richer and lands much closer to real document chunks in vector space. **Step 3 — Retrieve as normal** From here, retrieval works exactly as before — cosine similarity against your chunk index. The key insight: the hypothetical answer doesn't need to be correct. It just needs to be semantically close to what a real answer would look like. Even a partially wrong hypothetical retrieves better than a three-word query. **Question: "What are the current pricing offers?"** ---- Chunk 1: _Pricing offers: Basic, Business._ → Easy to find, but not much information. ---- Chunk 2: _Pricing offers: Basic, Business. Basic: $10/month, 20 requests/day. Business: $20/month, 100 requests/day, CRM integration, customer support, fully customizable interface._ → Harder to find (semantic meaning is diluted by the feature list), but the LLM response will be much more detailed. Each chunk is stored with two metadata fields: `doc_id` and `chunk_index`. When a chunk is retrieved, instead of passing just that chunk to the LLM, you query for `chunk_index - N` to `chunk_index + N` from the same `doc_id`. ```bash # Retrieved chunk { doc_id: "doc_42", chunk_id: 7, text: "...the model converges..." } # Fetch surrounding context chunks where doc_id = "doc_42" and chunk_index in [5, 6, 7, 8, 9] ``` You pass the full window to the LLM. The retrieved chunk is what matched — the window is what gives it meaning. **Step 1 — Split into individual sentences** Nothing fancy — a sentence tokenizer or even splitting on punctuation. **Step 2 — Embed each sentence** Each sentence gets embedded independently into a vector, using the same embedding model. ```bash "The Transformer uses multi-head attention." → [0.21, -0.10, 0.83, ...] "It allows every position to attend over all." → [0.22, -0.11, 0.81, ...] ← similar "We trained on the WMT 2014 dataset." → [0.74, 0.31, -0.22, ...] ← different ``` **Step 3 — Compare adjacent sentences** For each pair of consecutive sentences compute cosine similarity. Or N sentences - smooths out noisy signals. ```bash sentence 1 ↔ sentence 2 → similarity: 0.91 # same topic, no split sentence 2 ↔ sentence 3 → similarity: 0.43 # topic shift → split here ``` **Step 4 — Split at drops** You define a similarity threshold — say 0.7. Anywhere similarity drops below it, you cut a new chunk. Too high and you over-split, too low and you under-split. The threshold can be set manually, or derived statistically from your documents — using pecentile similarity across all sentence pairs as a baseline. Extraction for sames document with defaults. _Source: "Attention Is All You Need" (Vaswani et al.)_ ```bash ####################### #PyMuPDF 3.2.3 Applications of Attention in our Model The Transformer uses multi-head attention in three different ways: • In "encoder-decoder attention" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence. This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models such as [38, 2, 9]. #Table Model BLEU Training Cost (FLOPs) EN-DE EN-FR EN-DE EN-FR ByteNet [18] 23.75 ####################### #Unstructured [Title] 3.2.3 Applications of Attention in our Model [NarrativeText] The Transformer uses multi-head attention in three different ways: [ListItem] In "encoder-decoder attention" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence. This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models such as [38, 2, 9]. #Table [Title] Model [NarrativeText] ByteNet [18] Deep-Att + PosUnk [39] GNMT + RL [38] ConvS2S [9] MoE [32] Deep-Att + PosUnk Ensemble [39] GNMT + RL Ensemble [38] ConvS2S Ensemble [9] Transformer (base model) Transformer (big) [Title] BLEU [Title] EN-DE EN-FR 23.75 [UncategorizedText] 24.6 25.16 26.03 ####################### #Docling 3.2.3 Applications of Attention in our Model The Transformer uses multi-head attention in three different ways: - In "encoder-decoder attention" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence. This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models such as [38, 2, 9]. #Table | Model | BLEU | BLEU | Training Cost (FLOPs) | Training Cost (FLOPs) | |---------------------------------|--------|--------|-------------------------|-------------------------| | | EN-DE | EN-FR | EN-DE | EN-FR | | ByteNet [18] | 23.75 | | | | | Deep-Att + PosUnk [39] | | 39.2 | | 1 . 0 · 10 20 | | GNMT + RL [38] | 24.6 | 39.92 | 2 . 3 · 10 19 | 1 . 4 · 10 20 | ```

Rag posts