How you split documents into chunks is the single most impactful decision in a RAG pipeline. Get it wrong and retrieval quality tanks — no amount of embedding model tuning or reranking will save you. This guide explains when to use each strategy and how to tune them.
| Strategy | Best For | Chunk Quality | Speed | Complexity |
|---|---|---|---|---|
| Fixed-size | Logs, transcripts, code | Low | Very fast | Trivial |
| Recursive | General text, docs, articles | Medium | Fast | Low |
| Semantic | Multi-topic docs, meetings | High | Slow (needs embeddings) | Medium |
| Sentence-window | Technical QA, factual recall | Highest | Fast | Low |
Is the text homogeneous (single topic, uniform structure)?
├── YES → Fixed-size (chunk_size=500, overlap=100)
└── NO
├── Do you have an embedding model available at chunking time?
│ ├── YES → Semantic (threshold=0.5, min_sentences=2)
│ └── NO
│ ├── Is precision more important than recall?
│ │ ├── YES → Sentence-window (window_size=3)
│ │ └── NO → Recursive (chunk_size=1000, overlap=200)
│ └──
└──
How it works: Slides a window of chunk_size characters across the text with chunk_overlap characters of overlap between consecutive windows.
Pros: - Deterministic and fast - Easy to reason about chunk sizes for embedding models - Works on any text format
Cons: - Cuts sentences in half at window boundaries - Splits semantic units across chunks - Overlap wastes embedding budget on redundant content
When to use: - Homogeneous text: server logs, chat transcripts, code files - When you need consistent chunk sizes for batch embedding - As a baseline to compare smarter strategies against
Tuning parameters:
| Parameter | Default | Range | Effect |
|---|---|---|---|
chunk_size |
1000 | 200–2000 | Larger = more context per chunk, fewer chunks |
chunk_overlap |
200 | 0–500 | Larger = fewer boundary artifacts, more redundancy |
Rule of thumb: Set chunk_size to roughly the embedding model's optimal input length. For text-embedding-3-small, that's about 500–1000 characters. Set chunk_overlap to 15–25% of chunk_size.
How it works: Tries to split on paragraph boundaries (\n\n) first. If a piece is still too large, falls back to sentence boundaries (.), then word boundaries (), then character-level splits. Merges small pieces back up to approach the target size.
Pros: - Respects natural text structure (paragraphs, sentences) - Good default for most text types - No external dependencies
Cons: - Still heuristic — paragraph boundaries are not always semantic boundaries - Regex-based sentence detection mishandles abbreviations (Dr., U.S.)
When to use: - Documentation, articles, books, README files - When you want reasonable quality without embedding-time overhead - As the default strategy when you're not sure what to pick
Tuning parameters:
| Parameter | Default | Range | Effect |
|---|---|---|---|
chunk_size |
1000 | 200–2000 | Target chunk length |
chunk_overlap |
200 | 0–500 | Boundary overlap |
separators |
["\n\n", "\n", ". ", " ", ""] |
Custom list | Priority order of split points |
Pro tip: For Markdown docs, add "\n## " and "\n### " to the front of the separator list so heading-level sections are preferred split points.
How it works: Splits text into sentences, embeds each sentence, computes cosine similarity between consecutive sentence pairs, and places split points where similarity drops below a threshold (indicating a topic shift).
Pros: - Chunks align with actual topic boundaries - Variable-length chunks that match semantic units - Best retrieval quality for heterogeneous documents
Cons: - Requires an embedding model at chunking time (expensive for large corpora) - Slower than text-based strategies - Threshold tuning can be fiddly
When to use: - Meeting transcripts with multiple agenda items - Long-form documents with diverse sections - When retrieval quality justifies the embedding cost
Tuning parameters:
| Parameter | Default | Range | Effect |
|---|---|---|---|
similarity_threshold |
0.5 | 0.2–0.8 | Lower = fewer splits, larger chunks |
min_chunk_sentences |
2 | 1–5 | Prevents tiny single-sentence chunks |
Calibration: Run the chunker on a sample document, print the per-sentence similarity scores, and pick a threshold that places splits where you'd naturally see topic changes.
How it works: Embeds individual sentences (for precise retrieval) but when a sentence matches a query, returns a window of N surrounding sentences for richer generation context.
Pros: - Highest retrieval precision (each sentence has its own embedding) - Rich context for generation (surrounding sentences are included) - No sentence-cutting artifacts
Cons: - More chunks = more embeddings = higher cost - Surrounding window may include irrelevant sentences - Only effective with dense retrieval (not BM25)
When to use: - Technical QA where individual sentences carry distinct facts - Legal/medical documents where precise citation matters - When you have budget for many embeddings
Tuning parameters:
| Parameter | Default | Range | Effect |
|---|---|---|---|
window_size |
3 | 1–10 | Sentences before/after the match to include |
Rule of thumb: Start with window_size=3. If generation answers are missing context, increase to 5. If answers include too much noise, decrease to 1.
Different embedding models have different optimal input lengths. Sending too-long text wastes the tail; too-short text under-utilises the model.
| Embedding Model | Max Tokens | Optimal Chunk Size (chars) |
|---|---|---|
text-embedding-3-small |
8191 | 500–1500 |
text-embedding-3-large |
8191 | 500–1500 |
text-embedding-ada-002 |
8191 | 500–1000 |
Cohere embed-english-v3.0 |
512 | 300–600 |
| BGE-large | 512 | 300–600 |
Run the same query set against different chunking strategies and compare with the evaluation module:
from src.chunking.strategies import FixedSizeChunker, RecursiveChunker
from src.evaluation import RAGEvaluator, EvalSample
# Chunk the same document with two strategies
fixed_chunks = FixedSizeChunker(chunk_size=500).chunk(doc)
recursive_chunks = RecursiveChunker(chunk_size=500).chunk(doc)
# Run evaluation on each
evaluator = RAGEvaluator()
fixed_report = evaluator.evaluate_batch(fixed_samples)
recursive_report = evaluator.evaluate_batch(recursive_samples)
print(f"Fixed: {fixed_report.avg_overall:.3f}")
print(f"Recursive: {recursive_report.avg_overall:.3f}")
Getting good retrieval is the difference between a RAG system that feels like magic and one that hallucinates. This guide covers the knobs you can turn and the order in which to tune them.
Retrieval quality depends on these layers, from most to least impactful:
1. Chunking strategy ← Biggest impact (see chunking guide)
2. Embedding model choice ← Second biggest
3. Search mode (dense/hybrid) ← Third
4. Reranking ← Fourth (but cheap wins)
5. Top-k and context window ← Fine-tuning
Fix them in this order. Tuning reranking won't help if your chunks are terrible.
Uses cosine similarity between the query embedding and chunk embeddings.
Strengths: - Understands semantic meaning ("automobile" matches "car") - Works across languages (with multilingual embeddings) - Handles paraphrased queries
Weaknesses: - Misses exact keyword matches that BM25 catches - Requires embedding model and vector index - Can return semantically similar but factually irrelevant results
Classic term-frequency scoring with inverse document frequency weighting.
Strengths: - Catches exact matches that dense search misses ("error code E-4521") - No embedding model needed - Very fast on large corpora - Handles acronyms and product codes well
Weaknesses: - No semantic understanding ("car" won't match "automobile") - Sensitive to vocabulary mismatch - Requires text tokenization
Combines dense and keyword results using Reciprocal Rank Fusion (RRF).
RRF_score(d) = Σ weight / (k + rank(d))
Where k = 60 is a smoothing constant and weight is either dense_weight or keyword_weight = 1 - dense_weight.
When to use hybrid:
- Almost always. Hybrid consistently outperforms either mode alone.
- Start with dense_weight=0.7 (70% dense, 30% keyword).
When to use dense-only: - Cross-language retrieval - Queries that are heavily paraphrased
When to use keyword-only: - Error code lookups - Exact identifier matching - When you don't have an embedding model
dense_weight| dense_weight | Behaviour | Best for |
|---|---|---|
| 0.9 | Almost pure dense | Semantic/conceptual queries |
| 0.7 | Default hybrid | General QA (recommended start) |
| 0.5 | Equal weight | Mixed technical + conceptual |
| 0.3 | Keyword-heavy | Code search, error lookups |
First-stage retrieval (bi-encoder) optimises for recall — it casts a wide net. Second-stage reranking optimises for precision — it promotes the most relevant results to the top.
The gold standard. A cross-encoder model jointly encodes the query and each passage, capturing fine-grained interactions. Typically improves NDCG@10 by 5–15%.
Cost: One API call per (query, passage) pair. For top_k=5 retrieved × 3x over-retrieval = 15 API calls per query.
A free alternative. Boosts results that contain exact query terms. The formula blends the original retrieval score with a keyword overlap score:
final = (1 - alpha) * normalised_score + alpha * keyword_overlap
When to use:
- When API reranking is too slow or expensive
- For keyword-sensitive domains (legal, medical, technical)
- alpha=0.3 is a good starting point
| top_k | Tradeoff |
|---|---|
| 3 | Precise but may miss relevant context |
| 5 | Good balance (recommended default) |
| 10 | High recall but noisy context |
| 20 | Only useful with strong reranking |
Rule of thumb: Set top_k based on your LLM's context window. Each chunk is roughly 250 tokens, so top_k=5 uses about 1250 context tokens.
The max_context_chars setting prevents prompt overflow. Calculate it as:
max_context_chars = (model_context_window - prompt_tokens - max_generation_tokens) * 4
For GPT-4 (128k context), you have plenty of room. For GPT-3.5 (16k context), be conservative:
| Model | Context Window | Recommended max_context_chars |
|---|---|---|
| GPT-4o | 128k tokens | 32000 |
| GPT-4 | 8k tokens | 6000 |
| GPT-3.5 Turbo | 16k tokens | 10000 |
| Claude 3.5 | 200k tokens | 50000 |
The most reliable way to tune retrieval is to build a small evaluation set (20–50 question/answer pairs) and measure metrics after each change:
from src.evaluation import RAGEvaluator, EvalSample
# Build eval set
samples = [
EvalSample(
query="What is the refund policy?",
ground_truth="Full refund within 30 days of purchase.",
answer=rag.query("What is the refund policy?").answer,
contexts=[r.chunk.content for r in rag.query("What is the refund policy?").sources],
),
# ... more samples
]
evaluator = RAGEvaluator()
report = evaluator.evaluate_batch(samples)
print(report.to_markdown())
Key metrics to watch: - Context Recall < 0.7: Your retriever is missing relevant chunks. Try hybrid search, increase top_k, or improve chunking. - Context Precision < 0.5: Too much noise in retrieved context. Add reranking, reduce top_k, or tighten chunk_size. - Faithfulness < 0.7: The LLM is hallucinating. Tighten the prompt template, reduce context noise, or use a more instruction-following model. - Answer Relevancy < 0.6: The answer doesn't address the query. Check if the retriever is returning relevant chunks (context recall) and if the prompt template is clear.
| Symptom | Likely Cause | Fix |
|---|---|---|
| Answer is correct but vague | Context chunks too large | Reduce chunk_size, try sentence-window |
| Answer contradicts the context | LLM hallucination | Strengthen prompt, use faithfulness metric |
| Retriever misses obvious matches | Vocabulary mismatch | Enable hybrid search, add BM25 |
| Answer includes irrelevant details | Too much context noise | Add reranking, reduce top_k |
| Slow query latency | Too many chunks embedded | Reduce over-retrieval factor, use approximate search |
Get the full RAG Pipeline Framework and unlock everything.
Get the complete guide with every chapter unlocked, including code samples, diagrams, and best practices.
Access all interactive tools with complete data, all workload profiles, and the full scenario library.
Downloadable source code, configuration files, and working examples from every chapter.
Free updates for life. Every new chapter, tool, and improvement included.