← Back to all products

Schema Evolution Toolkit

$29

Detect, validate, migrate, and analyze schema changes across Delta Lake tables safely and automatically.

📁 27 files🏷 v2.0.0
JSONMarkdownPythonYAMLDatabricksPySparkSparkDelta LakeCI/CD

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 27 files

schema-evolution-toolkit/ ├── LICENSE ├── README.md ├── configs/ │ ├── schema_policy.yaml │ └── schemas/ │ ├── v1_customer.json │ ├── v1_order.json │ ├── v2_customer.json │ ├── v2_order.json │ └── v3_customer.json ├── guides/ │ └── schema-evolution-strategy.md ├── notebooks/ │ ├── detect_drift.py │ ├── evolve_schema.py │ └── generate_migration.py ├── src/ │ ├── compatibility_checker.py │ ├── impact_analyzer.py │ ├── migration_generator.py │ ├── schema_detector.py │ ├── schema_diff.py │ ├── schema_migrator.py │ ├── schema_registry.py │ └── schema_validator.py └── tests/ ├── conftest.py ├── test_compatibility.py ├── test_impact_analyzer.py ├── test_migration_generator.py ├── test_schema_detector.py └── test_schema_diff.py

📖 Documentation Preview README excerpt

Schema Evolution Toolkit

Detect, validate, migrate, and analyze schema changes across Delta Lake tables — safely and automatically.

By [Datanest Digital](https://datanest.dev) | Version 2.0.0 | $39

What You Get

A complete toolkit for managing schema evolution in Databricks / Delta Lake pipelines:

  • Schema Detector — compare live table schemas against expected definitions, detect drift
  • Schema Diff — detailed field-level diff between any two schema versions (no Spark required)
  • Schema Migrator — apply safe migrations (add columns, widen types, rename) with rollback
  • Migration Generator — auto-generate SQL forward, rollback, and Python migration scripts
  • Impact Analyzer — check which pipelines, views, and dashboards break before deploying
  • Compatibility Checker — verify backward/forward compatibility before deploying changes
  • Schema Registry — Delta-table-backed registry for versioned schema definitions
  • Schema Validator — validate DataFrames against registered schemas at runtime
  • Ready-to-use Notebooks — detect drift, evolve schemas, and generate migrations interactively
  • Schema Versions — example customer (v1/v2/v3) and order (v1/v2) JSON schema files
  • Evolution Strategy Guide — comprehensive guide to schema versioning, migration, and impact analysis

File Tree


schema-evolution-toolkit/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│   ├── schema_detector.py       # Detect schema drift between expected and actual
│   ├── schema_diff.py           # Field-level diff with severity classification
│   ├── schema_migrator.py       # Apply safe migrations with rollback support
│   ├── migration_generator.py   # Generate SQL/Python migration scripts from diffs
│   ├── impact_analyzer.py       # Analyze downstream impact of schema changes
│   ├── compatibility_checker.py # Backward/forward compatibility validation
│   ├── schema_registry.py       # Delta-table-backed schema version registry
│   └── schema_validator.py      # Runtime DataFrame schema validation
├── configs/
│   ├── schema_policy.yaml       # Evolution rules and policies per layer
│   └── schemas/
│       ├── v1_customer.json     # Customer schema v1 (4 fields)
│       ├── v2_customer.json     # Customer schema v2 (rename, add nested struct)
│       ├── v3_customer.json     # Customer schema v3 (arrays, nested structs, new fields)
│       ├── v1_order.json        # Order schema v1 (7 fields)
│       └── v2_order.json        # Order schema v2 (type widening, shipping, discounts)
├── notebooks/
│   ├── detect_drift.py          # Interactive drift detection notebook
│   ├── evolve_schema.py         # Interactive schema migration notebook
│   └── generate_migration.py    # Migration script generation notebook
├── tests/
│   ├── conftest.py              # Shared pytest fixtures (SparkSession, sample schemas)
│   ├── test_schema_detector.py  # Detector unit tests
│   ├── test_compatibility.py    # Compatibility checker tests
│   ├── test_schema_diff.py      # Schema diff tests
│   ├── test_migration_generator.py  # Migration generator tests
│   └── test_impact_analyzer.py  # Impact analyzer tests
└── guides/

*... continues with setup instructions, usage examples, and more.*

📄 Code Sample .py preview

src/compatibility_checker.py """ Compatibility Checker — Check forward, backward, and full compatibility between schema versions using Avro-style rules. Rules: - **Backward compatible**: consumers using the *old* schema can read data written with the *new* schema. Adding nullable columns and widening types is allowed; removing columns and narrowing types is not. - **Forward compatible**: consumers using the *new* schema can read data written with the *old* schema. Removing columns is allowed; adding required columns is not. - **Full compatible**: both backward and forward compatible simultaneously. Part of the Schema Evolution Toolkit by Datanest Digital (https://datanest.dev). """ from __future__ import annotations from dataclasses import dataclass, field from enum import Enum from typing import Dict, List, Optional, Set from pyspark.sql.types import StructField, StructType from schema_detector import ChangeKind, SchemaDetector, SchemaReport class CompatibilityMode(Enum): """Supported compatibility modes.""" BACKWARD = "backward" FORWARD = "forward" FULL = "full" NONE = "none" # Safe type widening paths (same as migrator) _SAFE_WIDENINGS: Dict[str, Set[str]] = { "byte": {"short", "int", "bigint", "double"}, "short": {"int", "bigint", "double"}, "int": {"bigint", "double"}, "bigint": {"double"}, "float": {"double"}, "date": {"timestamp"}, } @dataclass class CompatibilityResult: """Result of a compatibility check.""" # ... 139 more lines ...
Buy Now — $29 Back to Products