← Back to all products

Fine-Tuning Pipeline

$29

Python data pipeline for LLM fine-tuning with data cleaning, formatting, and quality checks.

📁 11 files
MarkdownPythonLLMOpenAI

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 11 files

fine-tuning-pipeline/ ├── LICENSE ├── README.md ├── examples/ │ ├── basic_usage.py │ └── sample_training_data.jsonl ├── free-sample.zip ├── guide/ │ ├── 01_features.md │ ├── 02_project-structure.md │ ├── 03_supported-formats.md │ └── 04_faq.md ├── index.html └── src/ └── fine_tuning_pipeline.py

📖 Documentation Preview README excerpt

Fine-Tuning Pipeline

Python data pipeline for LLM fine-tuning: data cleaning, JSONL formatting, validation, train/test splitting, quality checks, and dataset statistics. Zero dependencies.

Part of the AI Toolkit collection by [CodeVault](https://ai-toolkit.codevault.dev).

Features

  • Format detection — Auto-detects chat, completion, and instruction (Alpaca) formats
  • Format conversion — Convert between OpenAI chat, legacy completion, and Alpaca formats
  • Text cleaning — Normalize whitespace, smart quotes, control characters, and unicode
  • Validation — Check for missing fields, empty content, token limits, and format errors
  • Token counting — Approximate token counts for budget estimation
  • Train/test split — Reproducible random splitting with configurable ratio
  • Dataset statistics — Token distributions, format breakdown, and system message analysis
  • CLI interface — Full pipeline from raw data to fine-tuning-ready output

Quick Start


# Run demo with sample data
python src/fine_tuning_pipeline.py --demo

# Run the full pipeline: clean → convert → validate → split
python src/fine_tuning_pipeline.py --input raw_data.jsonl --output prepared/

# Validate a dataset
python src/fine_tuning_pipeline.py --validate dataset.jsonl

# Show dataset statistics
python src/fine_tuning_pipeline.py --stats dataset.jsonl

# Split with custom ratio
python src/fine_tuning_pipeline.py --split dataset.jsonl --ratio 0.9 --output prepared/

Project Structure


fine-tuning-pipeline/
├── README.md
├── LICENSE
├── src/
│   └── fine_tuning_pipeline.py    # Core engine (~430 lines)
└── examples/
    ├── basic_usage.py              # Programmatic usage example
    └── sample_training_data.jsonl  # Sample data in mixed formats

CLI Reference

FlagDescription
--demoRun demo with sample data
--input FILEInput data file (JSONL)
--output DIROutput directory (default: ./prepared)
--validate FILEValidate a dataset file
--stats FILEShow dataset statistics
--split FILESplit a dataset into train/test
--ratio FLOATTrain/test split ratio (default: 0.8)

... continues with setup instructions, usage examples, and more.

📄 Code Sample .py preview

examples/basic_usage.py#!/usr/bin/env python3 """ Basic usage example for the Fine-Tuning Pipeline. Demonstrates: - Detecting data formats (chat, completion, instruction) - Cleaning text for fine-tuning - Converting between formats - Validating training examples - Running the full pipeline (clean → convert → validate → split) - Computing dataset statistics """ import json import sys import tempfile from pathlib import Path # Allow running from the examples/ directory sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) from fine_tuning_pipeline import ( DataFormat, clean_text, compute_stats, detect_format, estimate_tokens, run_pipeline, to_chat_format, to_completion_format, validate_example, ) def demo_format_detection() -> None: """Detect the format of different training examples.""" print("=== Format Detection ===\n") examples = [ {"messages": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello!"}]}, {"prompt": "Q: What is 2+2?\nA:", "completion": " 4"}, {"instruction": "Translate to French", "input": "Hello", "output": "Bonjour"},
Buy Now — $29 Back to Products