Data Preparation for Fine-Tuning

The most impactful thing you can do for fine-tuning quality is improve your data.

A 500-example dataset of expert-curated pairs will outperform 50,000 noisy scraped examples.

Dataset Formats

The pipeline supports five input formats (auto-detected from JSONL structure):

Alpaca Format (Recommended)

{"instruction": "Explain recursion.", "input": "", "output": "Recursion is..."}

OpenAI Chat Format

{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

ShareGPT Format

{"conversations": [{"from": "human", "value": "..."}, {"from": "gpt", "value": "..."}]}

Simple Q&A

{"question": "What is LoRA?", "answer": "LoRA is..."}

Data Quality Checklist

Before you start training, check every item:

• [ ] Duplicates removed: Run content-hash deduplication. Near-duplicates (same question, slightly different wording) should also be caught.
• [ ] Minimum length enforced: Instructions under 10 characters and responses under 20 characters are almost always garbage.
• [ ] No boilerplate responses: Filter out "As an AI language model, I..." and similar refusal-only answers (unless you want refusal behavior).
• [ ] Balanced categories: If 80% of your data is one task type, the model will be heavily biased. Aim for reasonable coverage.
• [ ] Correct format: Each record has a clear instruction and a complete response. No truncated answers.
• [ ] No PII leakage: Check that training data doesn't contain real email addresses, phone numbers, API keys, or other sensitive information.

Quality Score Heuristic

The DatasetPreparer computes a quality score (0.0–1.0) for each record using:

Records below min_quality_score (default: 0.3) are dropped.

Dataset Size Guidelines

Train/Validation Split

The default 90/10 split works for most cases. Adjust based on dataset size:

• < 500 examples: Use 85/15 so validation has enough signal
• 500-5000 examples: Default 90/10 is fine
• > 5000 examples: Can use 95/5 — validation set is large enough

Always use a fixed random seed for reproducible splits.

Running the Preparation Pipeline

# Basic usage
python scripts/prepare_dataset.py data/sample_instructions.jsonl

# Multiple files with custom settings
python scripts/prepare_dataset.py data/*.jsonl \
    --output-dir data/prepared \
    --val-ratio 0.15 \
    --min-quality 0.4 \
    --seed 42

# Verbose mode for debugging
python scripts/prepare_dataset.py data/raw.jsonl -v

Output structure:

data/prepared/
├── train.jsonl              # Training examples
├── val.jsonl                # Validation examples
└── preparation_stats.json   # Statistics from the preparation run

Hyperparameter Tuning Guide

The 80/20 Rule for Fine-Tuning

These four parameters determine 80% of your training outcome:

1. Learning Rate — Most impactful single parameter

2. Dataset Quality — Garbage in, garbage out

3. Number of Epochs — Too few = underfit, too many = overfit

4. LoRA Rank — Capacity vs. efficiency tradeoff

Everything else is refinement on top of these.

Learning Rate Selection

Learning Rate Schedule

Cosine (default): Smooth decay from peak to near-zero. Works well across different training durations. Use this unless you have a specific reason not to.

Linear: Constant decrease. Simpler but slightly worse in practice for most tasks.

Constant with warmup: Good for very short training runs where you don't want the LR to decay at all.

Warmup

Warmup prevents the first few steps (with random adapter weights) from causing large, damaging gradient updates.

• 3-5% for large datasets (10k+ examples)
• 5-10% for small datasets (< 1000 examples)
• Fixed 100 steps for quick test runs

Epoch Count Decision Tree

Is your dataset > 10,000 examples?
├── Yes → Start with 1-2 epochs
│         Monitor val loss; stop when it starts rising
└── No
    ├── 1000-10000 examples → 2-3 epochs
    ├── 500-1000 examples → 3-5 epochs  
    └── < 500 examples → 5-10 epochs
                          (but watch for overfitting!)

Batch Size Strategy

Effective batch size = per_device_batch × gradient_accumulation × num_gpus

If VRAM-limited, reduce per_device_batch_size and increase gradient_accumulation_steps. The training dynamics are nearly identical.

Grid Search for LoRA

If you have the compute budget, sweep these combinations:

Learning Rate: [1e-4, 2e-4, 3e-4]
LoRA Rank:     [8, 16, 32]
Epochs:        [1, 2, 3]

That's 27 experiments. Most can be shortened to 1/4 of the data and 1 epoch to identify the best LR and rank, then do a full run with the winner.

Monitoring and Early Stopping

What to Watch

Early Stopping

Set early_stopping_patience to 3-5 evaluation rounds. If eval loss hasn't improved for that many consecutive evaluations, training stops automatically.

This prevents wasting compute on overfitting epochs and is especially important for small datasets.

Quick Reference: Default vs. Recommended by Task