Chunking Strategies for RAG: What Actually Works

Why Chunking Decisions Matter More Than Model Choice

The retrieval step in RAG can only return what you have indexed. If your chunking creates units that are too small, each chunk lacks context. Too large, and retrieval precision suffers - the relevant passage is buried in noise. The right chunking strategy determines your retrieval recall and precision before any model runs.

The Five Main Strategies

1. Fixed-Size with Overlap

pythonCopy
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,      # tokens per chunk
    chunk_overlap=64,    # tokens of overlap between adjacent chunks
    separators=["\n\n", "\n", ". ", " "],  # preferred split points
)
chunks = splitter.split_text(document_text)

Pros: Simple, reproducible, works well for uniform text. Cons: Splits semantic units arbitrarily. A paragraph about one concept may be split across two chunks. Best for: Web articles, documentation, general prose.

2. Sentence-Based

pythonCopy
import spacy
nlp = spacy.load("en_core_web_sm")

def sentence_chunks(text: str, max_sentences_per_chunk: int = 5) -> list[str]:
    doc = nlp(text)
    sentences = [sent.text.strip() for sent in doc.sents]
    chunks = []
    for i in range(0, len(sentences), max_sentences_per_chunk):
        chunks.append(" ".join(sentences[i:i + max_sentences_per_chunk]))
    return chunks

Pros: Respects sentence boundaries. Chunks make semantic sense. Cons: Chunk sizes vary widely. Very long or very short sentences create uneven chunks. Best for: Conversational text, emails, support tickets.

3. Semantic (Topic-Aware)

Split on semantic similarity discontinuities - when adjacent sentences discuss different topics.

pythonCopy
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

embed_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

def semantic_chunking(text: str, breakpoint_threshold: float = 0.5) -> list[str]:
    sentences = text.split(". ")
    embeddings = embed_model.encode(sentences)

    # Find breakpoints where cosine similarity drops
    breakpoints = [0]
    for i in range(1, len(embeddings)):
        sim = cosine_similarity([embeddings[i-1]], [embeddings[i]])[0][0]
        if sim < breakpoint_threshold:
            breakpoints.append(i)
    breakpoints.append(len(sentences))

    chunks = []
    for i in range(len(breakpoints) - 1):
        chunk = ". ".join(sentences[breakpoints[i]:breakpoints[i+1]])
        if chunk.strip():
            chunks.append(chunk)
    return chunks

Pros: Chunks correspond to coherent topics. Better precision for topic-specific queries. Cons: Slower (requires embedding sentences). Threshold requires tuning. Best for: Long technical documents, research papers, multi-topic content.

4. Document Structure-Based

pythonCopy
import re

def header_based_chunking(markdown_text: str) -> list[dict]:
    """Split on markdown headers, preserving hierarchy."""
    sections = re.split(r'\n(#{1,3} .+)\n', markdown_text)
    chunks = []
    current_header = "Introduction"

    for i, section in enumerate(sections):
        if re.match(r'^#{1,3} ', section):
            current_header = section.strip('# ').strip()
        elif section.strip():
            chunks.append({
                "text": section.strip(),
                "header": current_header,
                "level": section.count('#') if section.startswith('#') else 0
            })
    return chunks

Best for: Markdown documentation, HTML content with clear structure, structured reports.

5. Parent-Child (Small-to-Big)

Index small chunks for precise retrieval. Return the parent chunk (containing the small chunk) for context.

pythonCopy
def create_parent_child_chunks(text: str, parent_size: int = 1024, child_size: int = 256) -> list[dict]:
    parent_splitter = RecursiveCharacterTextSplitter(chunk_size=parent_size, chunk_overlap=0)
    child_splitter = RecursiveCharacterTextSplitter(chunk_size=child_size, chunk_overlap=32)

    chunks = []
    parents = parent_splitter.split_text(text)

    for parent_id, parent in enumerate(parents):
        children = child_splitter.split_text(parent)
        for child_id, child in enumerate(children):
            chunks.append({
                "child_text": child,      # What gets embedded and searched
                "parent_text": parent,    # What gets returned to the LLM
                "parent_id": parent_id,
                "child_id": child_id,
            })
    return chunks

Best for: Dense technical content where small passages are often retrieved but need surrounding context.

Evaluating Chunking Quality

pythonCopy
def evaluate_chunking(
    chunking_fn,
    eval_queries: list[dict],  # [{"query": "...", "expected_content": "..."}]
    embed_model,
    top_k: int = 5
) -> float:
    """Measure what fraction of queries retrieve the expected content."""
    hits = 0
    for case in eval_queries:
        chunks = chunking_fn(case["document"])
        chunk_embeddings = embed_model.encode([c["text"] for c in chunks])
        query_emb = embed_model.encode([case["query"]])[0]

        sims = cosine_similarity([query_emb], chunk_embeddings)[0]
        top_chunks = [chunks[i] for i in np.argsort(sims)[::-1][:top_k]]
        if any(case["expected_content"] in c["text"] for c in top_chunks):
            hits += 1
    return hits / len(eval_queries)

Practical Recommendations

• Start with: Fixed-size (512 tokens, 64 overlap) - works for 80% of use cases
• Switch to semantic when you see retrieval misses on topic-specific queries
• Use parent-child when retrieved chunks are too short to contain enough context for the LLM
• Always measure on your specific eval set before declaring a strategy "better"

The right chunk size is different for every corpus. Measure, do not guess.

Common Mistakes

Splitting mid-sentence at chunk boundaries. Fixed-size character splits do not respect sentence boundaries. A chunk that ends mid-sentence loses syntactic coherence, which degrades both embedding quality and the model's ability to reason over the retrieved text. Always use a recursive or sentence-aware splitter that prioritizes natural break points over exact size targets.

Using fixed character splits on markdown that breaks code blocks. Markdown code blocks are a single semantic unit; splitting inside a fenced block produces chunks that contain incomplete code snippets with mismatched delimiters. The LLM cannot use broken code effectively. Use a markdown-aware splitter that treats fenced blocks and headers as structural boundaries.

Never measuring retrieval recall after changing chunk size. Chunk size is the single most impactful hyperparameter in a RAG pipeline, yet many teams tune it by intuition. Without measuring recall@k before and after a change you cannot know whether you improved or degraded retrieval quality. Treat chunk size as a tunable hyperparameter with a defined metric.

What to Practice Next

• Implement the LangChain (or equivalent) recursive character text splitter and inspect five random chunks from a real document; verify that no chunk starts or ends mid-sentence.
• Measure recall@5 on your evaluation set with chunk sizes of 256, 512, and 1024 tokens and plot the results to find the inflection point for your corpus.
• Experiment with semantic chunking (split on embedding similarity drops) on the same document and compare recall@5 against fixed-size chunking.