Introduction

This is Part 2 of my series on building RAG systems. Part 1 covered the foundations — what RAG is, how the pipeline works, and touched on some of these techniques briefly. Here I want to go deeper.

The goal is to give you intuition for how these methods work. Some of them I've used myself and seen work well in practice. Some I haven't used yet — I'll be honest about that. And honestly, not all of them are equally useful. A lot depends on your data, your query patterns, and where your pipeline is actually failing.

If you haven't read Part 1, it's worth a quick skim — this part assumes you're familiar with the basics.

Ingest

Extract

First step is extracting text from your documents. A few tools, roughly in order of complexity: PyMuPDF is fast and works well on clean text-based PDFs. It struggles with messier layouts — tables, multi-column pages, presentations. You can patch some of that with regex, but how far you get depends heavily on your documents. Unstructured goes further. It cuts documents into structured elements (titles, tables, narrative text, headers) which makes the next steps — cleaning, chunking — much easier to reason about. Docling is the heavy option. It runs ML models on every page, which makes it slow, but the table extraction is remarkably good. If your documents are table-heavy, it's worth the cost. Check an example

As always: look at your actual extraction output before optimizing anything else. Bad extraction compounds through the entire pipeline.

Chunk

Semantic Chunking

Then we chunk. Fixed-size chunking cuts every N characters — it can cut mid-sentence, mid-idea. Semantic chunking tries to fix that: you embed sentence by sentence, compare adjacent sentences, and split where similarity drops. The idea is that a drop in similarity signals an idea shift. How it works under the hood

The intuition is clean. In practice, it's more complicated.

First, you still need a maximum and minimum chunk size. No matter how well you detect topic boundaries, a single "topic" can run for thousands of tokens — or be a single short sentence. So you end up capping by character count anyway, which partially defeats the purpose.

Second, the quality of the boundaries depends heavily on your embedding model and your data. On well-structured documents — articles, documentation — it can find decent splits. On messier text, the similarity signal is noisy and the boundaries aren't obviously better than a simpler approach.

Lastly, and this is important: if you extract properly, you can guide simple chunking using document structure — cut on paragraphs and sections, prepend titles, keep full tables in a single chunk. A well-extracted, structure-aware chunking strategy can beat noisy embedding-based cuts. In the end, the author decided where ideas shift and made it clear in the text — we just need to extract that signal cleanly.

It's worth knowing and worth trying on your data. But in my experience, semantic chunking didn't produce a wow effect — and you still need to tune hyperparameters: min, max size, the similarity threshold.

Hierarchical Chunking

The core tension in chunking: small chunks retrieve precisely but give the LLM thin context. Large chunks give rich context but embed poorly and retrieve noisily. Hierarchical chunking tries to get both — retrieve small, read big.

Parent/child chunks: You store two levels: small child chunks for retrieval, and larger parent chunks for context. When a child chunk is retrieved, you pass its parent to the LLM instead. Precise search, rich answer.

A lighter approach: I didn't implement the full parent/child store. Instead: chunks are stored with metadata (document ID, chunk number), and at retrieval time I fetch the surrounding chunks — the ones before and after the match. Same intuition, less infrastructure. Works well in practice and is much simpler to build.

The tradeoff: surrounding context is positional, not semantic. You get what's adjacent in the document, not necessarily what's most relevant. For well-structured documents it's usually fine — LLMs are powerful enough to extract the right information from a wider window. The real problem is context window size: fetching too many surrounding chunks bloats your prompt fast. But you can use (Contextual Compression) see furher.

LLM-Augmented Chunks

The three techniques below share the same idea: run an LLM at ingest time to enrich what gets stored in your index. On Retrieval You are no longer comparing your query against the original text, but against the LLM's interpretation of it. The cost is paid once when you build the index, not on every query.

They differ in what they augment and how much they touch your original text:

• Proposition chunking — rewrites original chunks into atomic facts
• HyPE — generates questions per chunk, stored alongside the original
• Hierarchical summary — generates summaries for routing.

The main tradeoff: making an LLM work well on your documents is hard. For some it will generate good facts, questions, summaries — for others it can generalize too much, lose precision. Prompt design matters, and evaluating the output at scale is not trivial, more then that, the output you try to find out depens on how users query your system. On top of that, you can blow up your index size fast.

In practice I used something between HyPE and Hierarchical Summary: for each document I generated a summary, extracted keywords, and listed possible document names — stored as a simple chunks alongside the originals. No two-level routing, no complex index. At retrieval time I gave these chunks higher weight than regular chunks. You can use Contextual Retrieval technique you use chunk level summary, append id directly to each chunk, and embed summary + chunk. It can smooth, and reduce semantic noise.

Store

There are basically two ways to store chunks: a vector store and a keyword store.

Vector store: you embed embed the chunk semantics into vector [0.23, -0.11, 0.87, ...], store it alongside the original text and metadata.

The vector captures semantic meaning and enables fast similarity search at scale.

Keyword store: an inverted index — think Elasticsearch or BM25. No embeddings. Stores which words appear in which chunks. Fast exact match. Works great when the user query contains specific terms, product names, codes, or IDs.

In my experience, both should work together, hybrid. People search by exact product name, error code, contract number — keyword wins there (Give me RFC 1918 details please). People search by concept, intent, or question — vector wins (need a specification for private network ranges). Depending on your data and query patterns, one will outperform the other.

Retrieve

Query Enhancement

The problem with embedding a raw query is that queries and documents live in different spaces. A query is short, often vague, sometimes just a bunch of keywords — "transformer attention mechanism". A document chunk is dense, structured, full of context. Even with a good embedding model, the vectors don't always land close enough to retrieve the right chunks.

HyDE

HyDE flips the approach: instead of embedding the query directly, you ask the LLM to generate a hypothetical answer first — use that instead. The hypothetical answer is longer, richer, and lands much closer in vector space to the real chunks you're looking for. How it works

In practice it helps quite a bit — especially for vague or conceptual queries. The hypothetical answer expands the signal: it introduces relevant keywords, concepts, and plausible phrasing that the query alone was missing.

The tradeoff: one extra LLM call, and a prompt that's hard to get right. If the LLM generates something off — wrong domain terms, wrong framing — your retrieval suffers. One bad hypothetical sinks the whole search.

RAG Fusion

RAG Fusion addresses that directly. Instead of generating one hypothetical, you generate N variants — HyDEs, query reformulations, extracted keywords, or a mix. Run a search for each, then merge the results with Reciprocal Rank Fusion (RRF).

RRF surfaces chunks that appear consistently across multiple searches — not just ones that scored well in a single pass. If one variant was off, the others compensate. The signal becomes more robust than trusting any single hypothetical.

Hybrid search

Hybrid search applies the same RRF logic, but varies the search method rather than the query — vector and keyword search run in parallel, results merged with RRF. Vector wins on conceptual queries; keyword wins on exact terms — product names, contract numbers, error codes.

You can combine all three dimensions simultaneously: multiple query variants across multiple search methods. Whether that complexity is worth it depends entirely on your data and query patterns.

In my experience: for most client projects, queries were keyword-heavy. Elasticsearch alone performed as well as hybrid retrieval — vector search helped at the margins but wasn't transformative. If your users search by concept or intent, the balance shifts the other way.

The lesson: don't assume hybrid or vector is always better. Profile your actual queries first.

Reranking

This is the single biggest quality improvement I've seen in practice.

After retrieval you have a pool of candidate chunks — say 50 — ranked RRF score. That ranking is decent but noisy. A reranker takes each retrieved chunk, pairs it with the original query, and scores how likely that chunk is to actually answer the question. You reorder by that score and pass the top 10 to the LLM. How it works

The difference from embedding similarity: embeddings compare query and chunk independently — each gets its own vector, and you measure distance. A reranker looks at the query and chunk together, which gives it much more signal to work with. It's slower, but you're only running it on 50 candidates, not your entire index.

Retrieve wide, rerank, pass the best. Simple pattern, real results.

Contextual Control

Contextual Augmentation

Once you have your retrieved chunks, you can extend the context you pass to the LLM before or after reranking. Two approaches from earlier apply here: fetching surrounding chunks by index, or using a parent/child structure to retrieve small but read large. For some documents — a pricing page, a short spec — the entire document fits in context anyway, making this trivial.

Timing matters: if you expand context before reranking, the reranker scores richer inputs but runs slower proportionally to how much you've added. If you expand after, reranking stays fast but operates on thinner context. In practice I'd expand after — rerank on the raw chunks, then widen the window only for the top results that get passed to the LLM.

Compression

On the other hand After reranking you have your top chunks — but each chunk might be 300 tokens with only 2 relevant sentences. Contextual compression runs a lightweight LLM pass over each chunk to strip the irrelevant parts before passing to the final LLM.

Less noise, smaller context window usage, potentially better answers. The risk: the compression model drops something important. It's making a relevance judgment, and it can be wrong.

I haven't used this in practice. The pattern I'd try: retrieve 15 → rerank to top 5 → compress → pass to LLM. But I'd only add compression if context window size is actually a bottleneck.

Conclusion

You now have everything you need to feed good context into your LLM. The post got long, so I won't go deep here — Part 1 already covered the prompt side a little: LLMs pay more attention to the beginning and end of context, be precise in your instructions, and use the metadata you stored. One thing I didn't cover is Agentic RAG — honestly because I haven't built one. The short version: you give an LLM a description of your RAG system as a tool, and it decides how to use it. It can judge whether a retrieved answer is good enough and query again if not, decide which metadata filters to apply based on the question, or skip RAG entirely and go to web search instead. Worth knowing about, but a topic for another post.

---