Chunking Strategy Guide

How you split documents into chunks is the single most impactful decision in a RAG pipeline. Get it wrong and retrieval quality tanks — no amount of embedding model tuning or reranking will save you. This guide explains when to use each strategy and how to tune them.

Strategy Overview

Decision Tree

Is the text homogeneous (single topic, uniform structure)?
├── YES → Fixed-size (chunk_size=500, overlap=100)
└── NO
    ├── Do you have an embedding model available at chunking time?
    │   ├── YES → Semantic (threshold=0.5, min_sentences=2)
    │   └── NO
    │       ├── Is precision more important than recall?
    │       │   ├── YES → Sentence-window (window_size=3)
    │       │   └── NO → Recursive (chunk_size=1000, overlap=200)
    │       └──
    └──

1. Fixed-Size Chunking

How it works: Slides a window of chunk_size characters across the text with chunk_overlap characters of overlap between consecutive windows.

Pros:

• Deterministic and fast
• Easy to reason about chunk sizes for embedding models
• Works on any text format

Cons:

• Cuts sentences in half at window boundaries
• Splits semantic units across chunks
• Overlap wastes embedding budget on redundant content

When to use:

• Homogeneous text: server logs, chat transcripts, code files
• When you need consistent chunk sizes for batch embedding
• As a baseline to compare smarter strategies against

Tuning parameters:

Rule of thumb: Set chunk_size to roughly the embedding model's optimal input length. For text-embedding-3-small, that's about 500–1000 characters. Set chunk_overlap to 15–25% of chunk_size.

2. Recursive Text Splitting

How it works: Tries to split on paragraph boundaries (\n\n) first. If a piece is still too large, falls back to sentence boundaries (. ), then word boundaries ( ), then character-level splits. Merges small pieces back up to approach the target size.

Pros:

• Respects natural text structure (paragraphs, sentences)
• Good default for most text types
• No external dependencies

Cons:

• Still heuristic — paragraph boundaries are not always semantic boundaries
• Regex-based sentence detection mishandles abbreviations (Dr., U.S.)

When to use:

• Documentation, articles, books, README files
• When you want reasonable quality without embedding-time overhead
• As the default strategy when you're not sure what to pick

Tuning parameters:

Pro tip: For Markdown docs, add "\n## " and "\n### " to the front of the separator list so heading-level sections are preferred split points.

3. Semantic Chunking

How it works: Splits text into sentences, embeds each sentence, computes cosine similarity between consecutive sentence pairs, and places split points where similarity drops below a threshold (indicating a topic shift).

Pros:

• Chunks align with actual topic boundaries
• Variable-length chunks that match semantic units
• Best retrieval quality for heterogeneous documents

Cons:

• Requires an embedding model at chunking time (expensive for large corpora)
• Slower than text-based strategies
• Threshold tuning can be fiddly

When to use:

• Meeting transcripts with multiple agenda items
• Long-form documents with diverse sections
• When retrieval quality justifies the embedding cost

Tuning parameters:

Calibration: Run the chunker on a sample document, print the per-sentence similarity scores, and pick a threshold that places splits where you'd naturally see topic changes.

4. Sentence-Window Chunking

How it works: Embeds individual sentences (for precise retrieval) but when a sentence matches a query, returns a window of N surrounding sentences for richer generation context.

Pros:

• Highest retrieval precision (each sentence has its own embedding)
• Rich context for generation (surrounding sentences are included)
• No sentence-cutting artifacts

Cons:

• More chunks = more embeddings = higher cost
• Surrounding window may include irrelevant sentences
• Only effective with dense retrieval (not BM25)

When to use:

• Technical QA where individual sentences carry distinct facts
• Legal/medical documents where precise citation matters
• When you have budget for many embeddings

Tuning parameters:

Rule of thumb: Start with window_size=3. If generation answers are missing context, increase to 5. If answers include too much noise, decrease to 1.

Chunk Size vs. Embedding Model

Different embedding models have different optimal input lengths. Sending too-long text wastes the tail; too-short text under-utilises the model.

Evaluation: How to Compare Strategies

Run the same query set against different chunking strategies and compare with the evaluation module:

from src.chunking.strategies import FixedSizeChunker, RecursiveChunker
from src.evaluation import RAGEvaluator, EvalSample

# Chunk the same document with two strategies
fixed_chunks = FixedSizeChunker(chunk_size=500).chunk(doc)
recursive_chunks = RecursiveChunker(chunk_size=500).chunk(doc)

# Run evaluation on each
evaluator = RAGEvaluator()
fixed_report = evaluator.evaluate_batch(fixed_samples)
recursive_report = evaluator.evaluate_batch(recursive_samples)

print(f"Fixed: {fixed_report.avg_overall:.3f}")
print(f"Recursive: {recursive_report.avg_overall:.3f}")

Retrieval Tuning Guide

Getting good retrieval is the difference between a RAG system that feels like magic and one that hallucinates. This guide covers the knobs you can turn and the order in which to tune them.

The Retrieval Quality Stack

Retrieval quality depends on these layers, from most to least impactful:

1. Chunking strategy          ← Biggest impact (see chunking guide)
2. Embedding model choice     ← Second biggest
3. Search mode (dense/hybrid) ← Third
4. Reranking                  ← Fourth (but cheap wins)
5. Top-k and context window   ← Fine-tuning

Fix them in this order. Tuning reranking won't help if your chunks are terrible.

Dense vs. Keyword vs. Hybrid Search

Dense Search (Embedding Similarity)

Uses cosine similarity between the query embedding and chunk embeddings.

Strengths:

• Understands semantic meaning ("automobile" matches "car")
• Works across languages (with multilingual embeddings)
• Handles paraphrased queries

Weaknesses:

• Misses exact keyword matches that BM25 catches
• Requires embedding model and vector index
• Can return semantically similar but factually irrelevant results

Keyword Search (BM25)

Classic term-frequency scoring with inverse document frequency weighting.

Strengths:

• Catches exact matches that dense search misses ("error code E-4521")
• No embedding model needed
• Very fast on large corpora
• Handles acronyms and product codes well

Weaknesses:

• No semantic understanding ("car" won't match "automobile")
• Sensitive to vocabulary mismatch
• Requires text tokenization

Hybrid Search (Recommended)

Combines dense and keyword results using Reciprocal Rank Fusion (RRF).

RRF_score(d) = Σ  weight / (k + rank(d))

Where k = 60 is a smoothing constant and weight is either dense_weight or keyword_weight = 1 - dense_weight.

When to use hybrid:

• Almost always. Hybrid consistently outperforms either mode alone.
• Start with dense_weight=0.7 (70% dense, 30% keyword).

When to use dense-only:

• Cross-language retrieval
• Queries that are heavily paraphrased

When to use keyword-only:

• Error code lookups
• Exact identifier matching
• When you don't have an embedding model

Tuning dense_weight

Reranking

First-stage retrieval (bi-encoder) optimises for recall — it casts a wide net. Second-stage reranking optimises for precision — it promotes the most relevant results to the top.

Cross-Encoder Reranking

The gold standard. A cross-encoder model jointly encodes the query and each passage, capturing fine-grained interactions. Typically improves NDCG@10 by 5–15%.

Cost: One API call per (query, passage) pair. For top_k=5 retrieved × 3x over-retrieval = 15 API calls per query.

Keyword Boost Reranking

A free alternative. Boosts results that contain exact query terms. The formula blends the original retrieval score with a keyword overlap score:

final = (1 - alpha) * normalised_score + alpha * keyword_overlap

When to use:

• When API reranking is too slow or expensive
• For keyword-sensitive domains (legal, medical, technical)
• alpha=0.3 is a good starting point

Top-k Selection

Rule of thumb: Set top_k based on your LLM's context window. Each chunk is roughly 250 tokens, so top_k=5 uses about 1250 context tokens.

Context Window Management

The max_context_chars setting prevents prompt overflow. Calculate it as:

max_context_chars = (model_context_window - prompt_tokens - max_generation_tokens) * 4

For GPT-4 (128k context), you have plenty of room. For GPT-3.5 (16k context), be conservative:

Evaluation-Driven Tuning

The most reliable way to tune retrieval is to build a small evaluation set (20–50 question/answer pairs) and measure metrics after each change:

from src.evaluation import RAGEvaluator, EvalSample

# Build eval set
samples = [
    EvalSample(
        query="What is the refund policy?",
        ground_truth="Full refund within 30 days of purchase.",
        answer=rag.query("What is the refund policy?").answer,
        contexts=[r.chunk.content for r in rag.query("What is the refund policy?").sources],
    ),
    # ... more samples
]

evaluator = RAGEvaluator()
report = evaluator.evaluate_batch(samples)
print(report.to_markdown())

Key metrics to watch:

• Context Recall < 0.7: Your retriever is missing relevant chunks. Try hybrid search, increase top_k, or improve chunking.
• Context Precision < 0.5: Too much noise in retrieved context. Add reranking, reduce top_k, or tighten chunk_size.
• Faithfulness < 0.7: The LLM is hallucinating. Tighten the prompt template, reduce context noise, or use a more instruction-following model.
• Answer Relevancy < 0.6: The answer doesn't address the query. Check if the retriever is returning relevant chunks (context recall) and if the prompt template is clear.

Common Failure Modes