← Back to all products

Schema Evolution Toolkit

$29

Detect, validate, migrate, and analyze schema changes across Delta Lake tables safely and automatically.

📁 27 files🏷 v2.0.0 (updated 2026-07-01)

Production-ready

✓ Instant download✓ Lifetime updates✓ MIT licensed✓ MIT license✓ Secure checkout (Stripe)

📋 What's Inside 27 files

✓ README.md—
✓ LICENSE—
✓ manifest.json—
✓ src/schema_detector.py—
✓ src/schema_diff.py—
✓ src/schema_migrator.py—
✓ src/migration_generator.py—
✓ src/impact_analyzer.py—
✓ src/compatibility_checker.py—
✓ src/schema_registry.py—
✓ src/schema_validator.py—
✓ configs/schema_policy.yaml—
✓ configs/schemas/v1_customer.json—
✓ configs/schemas/v2_customer.json—
✓ configs/schemas/v3_customer.json—

📁 File Structure 27 files

schema-evolution-toolkit/
├── README.md
├── LICENSE
├── manifest.json
├── src/
│ ├── schema_detector.py
│ ├── schema_diff.py
│ ├── schema_migrator.py
│ ├── migration_generator.py
│ ├── impact_analyzer.py
│ ├── compatibility_checker.py
│ ├── schema_registry.py
│ ├── schema_validator.py
├── configs/
│ ├── schema_policy.yaml
│ ├── schemas/
│ │ ├── v1_customer.json
│ │ ├── v2_customer.json
│ │ ├── v3_customer.json
│ │ ├── v1_order.json
│ │ ├── v2_order.json
├── notebooks/
│ ├── detect_drift.py
│ ├── evolve_schema.py
│ ├── generate_migration.py
├── tests/
│ ├── conftest.py
│ ├── test_schema_detector.py
│ ├── test_compatibility.py
│ ├── test_schema_diff.py
│ ├── test_migration_generator.py
│ ├── test_impact_analyzer.py
├── guides/
│ ├── schema-evolution-strategy.md

📖 Documentation Preview README excerpt

Schema Evolution Toolkit

Detect, validate, migrate, and analyze schema changes across Delta Lake tables — safely and automatically.

By [Datanest Digital](https://datanest.dev) | Version 2.0.0 | $39

---

What You Get

A complete toolkit for managing schema evolution in Databricks / Delta Lake pipelines:

**Schema Detector** — compare live table schemas against expected definitions, detect drift

**Schema Diff** — detailed field-level diff between any two schema versions (no Spark required)

**Schema Migrator** — apply safe migrations (add columns, widen types, rename) with rollback

**Migration Generator** — auto-generate SQL forward, rollback, and Python migration scripts

**Impact Analyzer** — check which pipelines, views, and dashboards break before deploying

**Compatibility Checker** — verify backward/forward compatibility before deploying changes

**Schema Registry** — Delta-table-backed registry for versioned schema definitions

**Schema Validator** — validate DataFrames against registered schemas at runtime

**Ready-to-use Notebooks** — detect drift, evolve schemas, and generate migrations interactively

**Schema Versions** — example customer (v1/v2/v3) and order (v1/v2) JSON schema files

**Evolution Strategy Guide** — comprehensive guide to schema versioning, migration, and impact analysis

File Tree


schema-evolution-toolkit/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│   ├── schema_detector.py       # Detect schema drift between expected and actual
│   ├── schema_diff.py           # Field-level diff with severity classification
│   ├── schema_migrator.py       # Apply safe migrations with rollback support
│   ├── migration_generator.py   # Generate SQL/Python migration scripts from diffs
│   ├── impact_analyzer.py       # Analyze downstream impact of schema changes
│   ├── compatibility_checker.py # Backward/forward compatibility validation
│   ├── schema_registry.py       # Delta-table-backed schema version registry
│   └── schema_validator.py      # Runtime DataFrame schema validation
├── configs/
│   ├── schema_policy.yaml       # Evolution rules and policies per layer
│   └── schemas/
│       ├── v1_customer.json     # Customer schema v1 (4 fields)
│       ├── v2_customer.json     # Customer schema v2 (rename, add nested struct)
│       ├── v3_customer.json     # Customer schema v3 (arrays, nested structs, new fields)
│       ├── v1_order.json        # Order schema v1 (7 fields)
│       └── v2_order.json        # Order schema v2 (type widening, shipping, discounts)
├── notebooks/
│   ├── detect_drift.py          # Interactive drift detection notebook
│   ├── evolve_schema.py         # Interactive schema migration notebook
│   └── generate_migration.py    # Migration script generation notebook
├── tests/
│   ├── conftest.py              # Shared pytest fixtures (SparkSession, sample schemas)
│   ├── test_schema_detector.py  # Detector unit tests
│   ├── test_compatibility.py    # Compatibility checker tests
│   ├── test_schema_diff.py      # Schema diff tests
│   ├── test_migration_generator.py  # Migration generator tests
│   └── test_impact_analyzer.py  # Impact analyzer tests
└── guides/

... preview truncated, see full README in product download.

`📄 Code Sample .py preview`

src/schema_detector.py"""
Schema Detector — Detect schema changes between two Spark DataFrames or StructTypes.

Identifies new columns, dropped columns, type changes, nullable changes,
and nested struct changes. Produces a structured SchemaReport for downstream
consumption by the compatibility checker or schema migrator.

Part of the Schema Evolution Toolkit by Datanest Digital (https://datanest.dev).
"""

from __future__ import annotations

import json
from dataclasses import dataclass, field
from enum import Enum
from pathlib import Path
from typing import Dict, List, Optional, Tuple, Union

from pyspark.sql import DataFrame, SparkSession
from pyspark.sql.types import (
    ArrayType,
    DataType,
    MapType,
    StructField,
    StructType,
)


class ChangeKind(Enum):
    """Categories of schema change."""

    ADDED = "added"
    REMOVED = "removed"
    TYPE_CHANGED = "type_changed"
    NULLABLE_CHANGED = "nullable_changed"
    NESTED_CHANGED = "nested_changed"


@dataclass
class SchemaChange:

`📅 Changelog`

v2.0.0 — Initial release. Last updated 2026-07-01.

Purchases include lifetime updates. Check the product page for the latest version.

`📄 Product Preview`

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview)⚙ Try Demo Tools📦 Download Free Sample

`❓ Frequently Asked Questions`

What license is this under?

MIT License. You can use it in commercial, client, and internal projects. See our license terms page for details.

How do I download after purchase?

Immediately after purchase, you'll be redirected to a download page. A download link is also sent to your email.

Do I get updates?

Yes. All products include lifetime updates at no extra cost. Re-download anytime for the latest version.

What if it doesn't work for me?

DM @DatanestDigital on X and we'll make it right.

Can I get a refund?

DM @DatanestDigital on X with your order number and we'll help.

Is there support?

DM @DatanestDigital on X. We respond within 1-2 business days.



      
        Buy Now — $29
        
        Back to Products