Document Processing Guide

Supported Formats

Format	Parser	External Deps	Notes
`.txt`	Stdlib	None	Split by blank lines
`.md`	Stdlib regex	None	Headings, lists, code blocks, blockquotes
`.html`	Stdlib HTMLParser	None	Extracts body content with structure
`.pdf`	pdfplumber	`pip install pdfplumber`	Text + table extraction
`.docx`	python-docx	`pip install python-docx`	Paragraphs, headings, tables
Images	Tesseract OCR	`pip install pytesseract`	Via OCR interface

Chunking Strategy Selection

Choose the right chunking strategy based on your documents:

Is the document well-structured (headings, sections)?
├── Yes → Use SEMANTIC chunking
│         (each section becomes one or more chunks with heading context)
└── No
    ├── Is it prose (articles, reports)?
    │   └── Use SENTENCE chunking (preserves sentence boundaries)
    ├── Is it technical docs with clear paragraphs?
    │   └── Use PARAGRAPH chunking (atomic paragraph units)
    └── Is it raw/unstructured text?
        └── Use FIXED or RECURSIVE chunking

Chunk Size Guidelines for RAG

Embedding Model	Recommended Chunk Size	Max Tokens
OpenAI text-embedding-3-small	500-800 chars	8191
Cohere embed-v3	500-1000 chars	512
sentence-transformers	300-500 chars	512
E5-large	300-500 chars	512

Overlap: 10-20% of chunk size prevents information loss at boundaries.

Schema-Guided Extraction

Define a schema for structured data extraction:

python

from src.schema_extractor import ExtractionSchema, SchemaExtractor

schema = ExtractionSchema("contract_data")
schema.add_field("parties", "list", "Names of contracting parties", required=True)
schema.add_field("effective_date", "date", "Contract start date", required=True)
schema.add_field("termination_date", "date", "Contract end date")
schema.add_field("total_value", "currency", "Total contract value")
schema.add_field("governing_law", "string", "Jurisdiction / governing law")

extractor = SchemaExtractor(schema)
result = extractor.extract(document_text)

The regex fallback handles common types (dates, emails, currency, phones) without an LLM.

For complex extraction, provide an llm_fn callback.

Summarization Strategies

Strategy	Best For	How It Works
Extractive	Any length, no LLM needed	Picks top sentences by word frequency + position
Direct	Short docs (< 3000 chars)	Sends full text to LLM
Map-Reduce	Medium docs (3k-30k chars)	Summarize chunks, then combine
Refine	Long docs (30k+ chars)	Iteratively refine summary with each chunk