RAG Cheatsheet: Chunking, Hybrid Search, Reranking

Most RAG systems that "hallucinate" are not suffering from a generation problem. They are suffering from a retrieval problem. The model answered from the chunks it was given, and the chunks were wrong, incomplete, or cut in half at exactly the sentence that mattered.

Retrieval is plain information retrieval engineering with a language model at the end. The ideas that work are mostly old (BM25 is from the 1990s, reciprocal rank fusion from 2009, cross-encoder reranking from 2019) and they combine well with newer ones like contextual chunk embeddings. The frameworks change every few months; the pipeline does not.

This cheatsheet covers the retrieval side end to end: how to cut documents, what to store next to each chunk, how to pick embeddings, how to combine lexical and dense search, how to rerank, and how to tell whether any of it helped. It closes with the question everyone is asking in May 2025 now that 1M-token context windows are common: do you still need RAG?

1. The pipeline at a glance

Every stage can lose the answer. The later stages cannot recover what an earlier stage dropped: a reranker cannot promote a chunk that was never retrieved, and retrieval cannot find a sentence that the parser mangled.

2. Chunking strategies

Practical rules:

• Start with structure-aware splitting, fall back to recursive. Headings, functions and table rows are the boundaries the author already chose.
• Size: 200-800 tokens covers most cases. Smaller chunks give sharper matches but less context per hit. Chroma's chunking study (July 2024) found a recursive splitter at about 200 tokens with no overlap performed well, and OpenAI's Assistants default (800 tokens, 400 overlap) did poorly on their token-level metrics. Test on your data; do not copy a number.
• Overlap: 0-20%. Overlap rescues sentences cut at a boundary but duplicates content in results. With structure-aware boundaries you often need none.
• Never split tables and code blocks in the middle. Keep the header row with each table chunk; keep a function signature with its body.
• Prepend the breadcrumb. "Billing Guide > Refunds > EU customers" at the top of a chunk costs a few tokens and helps both BM25 and embeddings.

3. Metadata: the cheapest precision gain

Store metadata next to every chunk and use it as filters before (or during) vector search:

A filter that removes 90% of the corpus before ranking beats any embedding upgrade for questions like "what changed in v3 of the API?".

4. Choosing embeddings

Dense embeddings are good at paraphrase and meaning. They are bad at exact tokens: error codes, SKUs, function names, rare acronyms and people's names. That is why the next section exists.

5. Hybrid search: BM25 plus dense

BM25 scores documents by term overlap, weighted by how rare each term is and normalized for document length. It finds ERR_CONN_RESET and kubectl rollout undo exactly. Dense search finds "my connection keeps dropping" when the doc says "socket reset". Run both and fuse.

Reciprocal rank fusion (RRF)

Scores from BM25 and cosine similarity live on different scales, so adding them is fragile. RRF ignores scores and uses only ranks (Cormack, Clarke and Buettcher, SIGIR 2009):

RRF(d) = sum over retrievers r of 1 / (k + rank_r(d))

where rank starts at 1 and k is a constant, usually 60. A document ranked high by both retrievers beats one ranked first by only one. The constant k damps the influence of the very top ranks so no single retriever dominates.

from collections import defaultdict


def rrf(rankings: list[list[str]], k: int = 60, top_n: int = 50) -> list[tuple[str, float]]:
    """Fuse several ranked lists of doc ids with reciprocal rank fusion."""
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)[:top_n]


bm25_hits = ["c7", "c2", "c9", "c4"]
dense_hits = ["c2", "c5", "c7", "c1"]
for doc_id, score in rrf([bm25_hits, dense_hits]):
    print(doc_id, round(score, 4))
# c2 0.0325  (rank 2 + rank 1)
# c7 0.0323  (rank 1 + rank 3)
# c5 0.0161, c9 0.0159, c1 0.0156, c4 0.0156

Alternatives: weighted score fusion after min-max normalization (tunable, but sensitive to score distributions) or weighted RRF (multiply each term by a retriever weight). Start with plain RRF; it has no parameters worth tuning on day one.

Practical defaults: retrieve 50-150 candidates from each retriever, fuse, keep the top 50-100 for reranking.

6. Reranking with cross-encoders

Embedding search is a bi-encoder: query and chunk are encoded separately and compared with a dot product, which is fast but coarse. A cross-encoder reads the query and the chunk together and outputs a relevance score. It sees word-level interactions ("refund" near "EU" near "after 14 days"), which makes it much more accurate and much slower.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")


def rerank(query: str, candidates: list[dict], top_k: int = 8) -> list[dict]:
    pairs = [(query, c["text"]) for c in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [c for c, _ in ranked[:top_k]]

Rules of thumb: rerank 20-100 candidates, pass 5-20 to the model, and measure latency at p95, not on average. Hosted rerank APIs and LLM-as-reranker (ask a model to score or order candidates) are alternatives with the same shape. The idea traces back to Nogueira and Cho's BERT passage reranker (2019).

7. Contextual retrieval

A chunk like "Revenue grew 3% over the previous quarter" is useless out of context: which company, which quarter? Anthropic's Contextual Retrieval (September 2024) fixes this at index time. For each chunk, an LLM reads the whole document and writes a short context (usually 50-100 tokens), which is prepended to the chunk before both embedding and BM25 indexing.

Reported results, measured as retrieval failure rate (1 minus recall@20) averaged across their datasets:

With prompt caching (the document is cached once, then each chunk is a small incremental call), they estimated the one-time cost at about $1.02 per million document tokens. The same post notes that for a knowledge base under about 200,000 tokens, you can skip retrieval and put everything in the prompt.

Late chunking (section 2) attacks the same problem from the embedding side, without an LLM call per chunk, but only helps the dense retriever.

8. Query rewriting and HyDE

User queries are short, vague and conversational. Fix the query before you search:

HyDE comes from Gao et al. (2022), "Precise Zero-Shot Dense Retrieval without Relevance Labels". The hypothetical answer may be factually wrong; that is fine, because it is used only as a search vector, never shown to the user. Each rewrite step adds an LLM call of latency, so use a small fast model.

9. Measuring retrieval (briefly)

You need a small labeled set: queries plus the chunk ids that answer them. 50-200 queries from real logs is enough to compare chunking or fusion choices. Three metrics cover most decisions:

from math import log2


def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    return len(set(retrieved[:k]) & relevant) / len(relevant)


def mrr(retrieved: list[str], relevant: set[str]) -> float:
    for i, doc_id in enumerate(retrieved, start=1):
        if doc_id in relevant:
            return 1.0 / i
    return 0.0


def ndcg_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    dcg = sum(1 / log2(i + 1) for i, d in enumerate(retrieved[:k], start=1) if d in relevant)
    ideal = sum(1 / log2(i + 1) for i in range(1, min(len(relevant), k) + 1))
    return dcg / ideal if ideal else 0.0

Measure recall at the k you actually pass to the model, and recall at the candidate pool size before reranking. If recall@100 is low, fix chunking, metadata or hybrid search. If recall@100 is high but recall@5 is low, fix reranking.

10. Failure modes

11. Long context vs RAG (May 2025)

Context windows grew fast this spring. Gemini 2.5 Pro (March 2025) ships with 1M tokens, OpenAI's GPT-4.1 family (April 14, 2025) supports 1M tokens, and Meta's Llama 4 Scout (April 5, 2025) advertises 10M. It is fair to ask whether retrieval is still needed.

Two cautions about "just put everything in". First, a window is not the same as usable attention: Liu et al. ("Lost in the Middle", 2023) showed accuracy drops when the relevant passage sits in the middle of a long context, and NoLiMa (February 2025) found that when the question and the answer share few literal words, most models degrade sharply as context grows. GPT-4o fell from 99.3% at short lengths to 69.7% at 32K tokens. Second, cost and latency scale with every token you send, on every request.

The pragmatic answer in 2025 is both: retrieve well, then use the bigger window to send more and larger chunks (parent sections, whole short documents) instead of five tiny fragments.

Why this matters

• Retrieval sets the ceiling. The model cannot cite what you did not retrieve. Most answer quality work is retrieval work.
• Hybrid plus rerank is the default, not an optimization. BM25 catches the exact tokens embeddings miss; a cross-encoder fixes the ordering both get wrong.
• Chunks need context. Breadcrumbs, metadata and contextual retrieval turn orphan paragraphs into findable units.
• Measure recall at two depths. It tells you which stage to fix.
• Long context changes the budget, not the need. Bigger windows let you pass more context, but selection still decides what the model sees.

Final Thoughts

A good retrieval pipeline looks boring: structure-aware chunks with breadcrumbs, metadata filters, BM25 and dense search fused with RRF, a cross-encoder on top, and a standalone query rewrite for chat. Each piece is well understood and cheap to test.

Build that baseline first, label a hundred real queries, and only then reach for semantic chunkers, agents that search iteratively, or a bigger embedding model. Most of the time, the boring pipeline is the one that ships.

Sources

• Anthropic, Introducing Contextual Retrieval (September 19, 2024): method, failure-rate reductions (35%, 49%, 67%), 50-100 token context, $1.02 per million document tokens, 200K-token guidance.
• Cormack, Clarke and Buettcher, Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (SIGIR 2009): RRF formula and k = 60.
• Smith and Troynikov (Chroma), Evaluating Chunking Strategies for Retrieval (July 3, 2024): chunk size and overlap results, recursive splitter at 200 tokens, OpenAI 800/400 default underperforming.
• Günther et al. (Jina AI), Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models (2024).
• Nogueira and Cho, Passage Re-ranking with BERT (2019): cross-encoder reranking.
• Gao et al., Precise Zero-Shot Dense Retrieval without Relevance Labels (2022): HyDE.
• Muennighoff et al., MTEB: Massive Text Embedding Benchmark (2022): embedding model comparison.
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023): position effects in long context.
• Modarressi et al., NoLiMa: Long-Context Evaluation Beyond Literal Matching (February 2025): degradation with low lexical overlap, GPT-4o 99.3% to 69.7% at 32K.
• OpenAI, Introducing GPT-4.1 in the API (April 14, 2025): 1M-token context.
• Google, Gemini 2.5: Our most intelligent AI model (March 25, 2025): 1M-token context.
• Meta, The Llama 4 herd (April 5, 2025): Llama 4 Scout 10M-token context.