Features

This chapter covers the core features and capabilities of Fine-Tuning Pipeline.

Features

• Format detection — Auto-detects chat, completion, and instruction (Alpaca) formats
• Format conversion — Convert between OpenAI chat, legacy completion, and Alpaca formats
• Text cleaning — Normalize whitespace, smart quotes, control characters, and unicode
• Validation — Check for missing fields, empty content, token limits, and format errors
• Token counting — Approximate token counts for budget estimation
• Train/test split — Reproducible random splitting with configurable ratio
• Dataset statistics — Token distributions, format breakdown, and system message analysis
• CLI interface — Full pipeline from raw data to fine-tuning-ready output

Quick Start

# Run demo with sample data
python src/fine_tuning_pipeline.py --demo

# Run the full pipeline: clean → convert → validate → split
python src/fine_tuning_pipeline.py --input raw_data.jsonl --output prepared/

# Validate a dataset
python src/fine_tuning_pipeline.py --validate dataset.jsonl

# Show dataset statistics
python src/fine_tuning_pipeline.py --stats dataset.jsonl

# Split with custom ratio
python src/fine_tuning_pipeline.py --split dataset.jsonl --ratio 0.9 --output prepared/

Project Structure

Follow this guide to get Fine-Tuning Pipeline up and running in your environment.

Project Structure

fine-tuning-pipeline/
├── README.md
├── LICENSE
├── src/
│   └── fine_tuning_pipeline.py    # Core engine (~430 lines)
└── examples/
    ├── basic_usage.py              # Programmatic usage example
    └── sample_training_data.jsonl  # Sample data in mixed formats

CLI Reference