← Back to all products
$19
Data Cleaning Playbook
Step-by-step data quality frameworks: deduplication, standardization, outlier detection, missing value strategies, and validation rules.
YAMLMarkdownPython
📄 Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample📁 File Structure 30 files
data-cleaning-playbook/
├── LICENSE
├── README.md
├── examples/
│ ├── __pycache__/
│ │ └── cleaning_pipeline.cpython-312.pyc
│ ├── cleaning_pipeline.py
│ └── messy_customers.csv
├── free-sample.zip
├── guide/
│ ├── deduplication_strategies.md
│ ├── missing_data_playbook.md
│ └── outlier_detection_guide.md
├── guides/
│ ├── deduplication_strategies.md
│ ├── missing_data_playbook.md
│ └── outlier_detection_guide.md
├── index.html
├── requirements.txt
├── rules/
│ ├── common_patterns.yaml
│ └── data_quality_rules.yaml
└── src/
└── cleaners/
├── __init__.py
├── __pycache__/
│ ├── __init__.cpython-312.pyc
│ ├── deduplication.cpython-312.pyc
│ ├── main.cpython-312.pyc
│ ├── missing_values.cpython-312.pyc
│ ├── outliers.cpython-312.pyc
│ ├── standardization.cpython-312.pyc
│ └── validation_engine.cpython-312.pyc
├── deduplication.py
├── main.py
├── missing_values.py
├── outliers.py
├── standardization.py
└── validation_engine.py
📖 Documentation Preview README excerpt
Data Cleaning Playbook
A comprehensive toolkit combining deep playbook guides with runnable Python cleaners. Covers deduplication, standardization, outlier detection, missing-value strategies, and a rule-based validation engine.
Features
- Deduplication Engine — Exact-match, fuzzy-match, and time-window dedup strategies
- Standardization — Date formats, addresses, names, phone numbers, categorical encoding
- Outlier Detection — IQR, z-score, Modified Z-Score (MAD), and isolation-based methods
- Missing Value Strategies — Pattern analysis, imputation hierarchy, and indicator generation
- Validation Rule Engine — Declarative YAML rules for automated data quality checks
Quick Start
pip install -r requirements.txt
python -m src.cleaners.main
Structure
src/cleaners/
├── __init__.py # Package exports
├── main.py # Demo runner
├── deduplication.py # Exact and fuzzy deduplication
├── standardization.py # Date/name/address standardizers
├── outliers.py # IQR, z-score, MAD outlier detection
├── missing_values.py # Missing value analysis and imputation
├── validation_engine.py # YAML-driven rule validation
rules/
├── data_quality_rules.yaml # Example validation ruleset
├── common_patterns.yaml # Regex patterns for data standardization
examples/
├── messy_customers.csv # Sample messy data for testing
├── cleaning_pipeline.py # End-to-end cleaning example
guides/
├── deduplication_strategies.md # When to use which dedup approach
├── outlier_detection_guide.md # Choosing the right detection method
├── missing_data_playbook.md # Decision framework for missing values
Usage
Detect and Remove Duplicates
from src.cleaners.deduplication import find_duplicates, deduplicate
dupes = find_duplicates(df, subset=["email"], method="exact")
clean_df = deduplicate(df, subset=["email"], keep="latest", date_col="created_at")
Standardize Dates and Names
from src.cleaners.standardization import standardize_dates, standardize_names
df["date"] = standardize_dates(df["date_raw"], output_format="%Y-%m-%d")
df["name"] = standardize_names(df["name_raw"])
Detect Outliers
*... continues with setup instructions, usage examples, and more.*
📄 Code Sample .py preview
examples/cleaning_pipeline.py"""
End-to-end data cleaning pipeline example.
Demonstrates loading messy CSV data, applying cleaning transformations in
sequence, validating the results, and exporting the clean dataset.
This is a realistic workflow showing how to combine the playbook modules
to clean a customer dataset with common real-world issues:
- Duplicate records
- Inconsistent formatting (case, whitespace, abbreviations)
- Invalid values (negative ages, malformed emails)
- Missing data with different imputation strategies
- Outliers that need flagging or capping
Usage:
python cleaning_pipeline.py
Reads: ../examples/messy_customers.csv
Writes: ./cleaned_customers.csv, ./cleaning_report.json
"""
from __future__ import annotations
import csv
import json
import logging
import os
import re
import sys
from dataclasses import dataclass, field
from datetime import datetime
from pathlib import Path
from typing import Any
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
logging.basicConfig(
level=logging.INFO,