← Back to all products

Delta Lake Patterns

$29

Production-ready Delta Lake merge, optimization, and maintenance patterns for Databricks.

📁 17 files🏷 v1.0.0

Production-ready

✓ Instant download✓ Lifetime updates✓ MIT licensed✓ MIT license✓ Secure checkout (Stripe)

📋 What's Inside 17 files

✓ README.md—
✓ manifest.json—
✓ LICENSE—
✓ src/merge_patterns.py—
✓ src/optimization.py—
✓ src/time_travel.py—
✓ src/change_data_feed.py—
✓ src/table_utilities.py—
✓ src/liquid_clustering.py—
✓ configs/table_maintenance.yaml—
✓ configs/table_properties.yaml—
✓ notebooks/setup_tables.py—
✓ notebooks/maintenance_runner.py—
✓ notebooks/cdf_processor.py—
✓ tests/test_merge_patterns.py—

📁 File Structure 17 files

delta-lake-patterns/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│ ├── merge_patterns.py
│ ├── optimization.py
│ ├── time_travel.py
│ ├── change_data_feed.py
│ ├── table_utilities.py
│ ├── liquid_clustering.py
├── configs/
│ ├── table_maintenance.yaml
│ ├── table_properties.yaml
├── notebooks/
│ ├── setup_tables.py
│ ├── maintenance_runner.py
│ ├── cdf_processor.py
├── tests/
│ ├── test_merge_patterns.py
│ ├── conftest.py
├── guides/
│ ├── delta-lake-best-practices.md

📖 Documentation Preview README excerpt

Delta Lake Patterns

Production-ready Delta Lake merge, optimization, and maintenance patterns for Databricks.

> Master the full spectrum of Delta Lake operations — from SCD Type 2 merges to Liquid Clustering migration, Change Data Feed processing, and automated table maintenance.

---

What You Get

**5 merge strategies** — SCD1, SCD2, upsert, delete+insert, and conditional merge with full PySpark implementations

**Table optimization toolkit** — OPTIMIZE, ZORDER, vacuum scheduling, and ANALYZE TABLE automation

**Time travel operations** — Version history queries, point-in-time restore, and audit trail generation

**Change Data Feed processing** — Incremental CDF readers with watermark tracking and replay support

**Liquid Clustering** — Setup, migration from ZORDER, and monitoring utilities

**Table utilities** — Clone, convert-to-delta, property management, and schema inspection

**Maintenance scheduler** — YAML-driven maintenance configs for bronze/silver/gold layers

**Runnable notebooks** — Setup, maintenance runner, and CDF processor ready for Databricks

**Tests included** — Merge pattern tests with sample data and pytest fixtures

File Tree


delta-lake-patterns/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│   ├── merge_patterns.py          # SCD1, SCD2, upsert, delete+insert, conditional
│   ├── optimization.py            # OPTIMIZE, ZORDER, vacuum, maintenance scheduler
│   ├── time_travel.py             # Version history, restore, audit trail
│   ├── change_data_feed.py        # CDF reader, incremental processing
│   ├── table_utilities.py         # Clone, convert-to-delta, describe history
│   └── liquid_clustering.py       # Liquid clustering setup and migration
├── configs/
│   ├── table_maintenance.yaml     # Maintenance schedule per layer
│   └── table_properties.yaml      # Standard table properties
├── notebooks/
│   ├── setup_tables.py            # Create Delta tables with configs
│   ├── maintenance_runner.py      # Run maintenance across schemas
│   └── cdf_processor.py           # Process Change Data Feed
├── tests/
│   ├── test_merge_patterns.py     # Test SCD1/SCD2 merge logic
│   └── conftest.py                # Pytest fixtures
└── guides/
    └── delta-lake-best-practices.md


Getting Started

1. Run a SCD Type 2 Merge


from src.merge_patterns import scd2_merge

scd2_merge(
    target_table="catalog.silver.dim_customer",
    source_df=incoming_customers,
    merge_keys=["customer_id"],
    tracked_columns=["email", "address", "phone"],

... preview truncated, see full README in product download.

`📄 Code Sample .py preview`

src/merge_patterns.py"""
Delta Lake Merge Patterns
=========================

Production-ready MERGE INTO patterns for Databricks Delta Lake tables.

Includes SCD Type 1 (overwrite), SCD Type 2 (history tracking), standard upsert,
delete+insert, and conditional merge strategies.

Datanest Digital | https://datanest.dev
"""

from __future__ import annotations

import logging
from datetime import datetime
from typing import Optional

from pyspark.sql import DataFrame
from pyspark.sql import functions as F
from pyspark.sql.types import BooleanType, StringType, TimestampType
from delta.tables import DeltaTable

# Databricks globals
spark = spark  # type: ignore[name-defined]

logger = logging.getLogger(__name__)


# ---------------------------------------------------------------------------
# SCD Type 1 — Overwrite existing records with latest values
# ---------------------------------------------------------------------------

def scd1_merge(
    target_table: str,
    source_df: DataFrame,
    merge_keys: list[str],
    update_columns: Optional[list[str]] = None,
    insert_only_columns: Optional[list[str]] = None,
) -> dict[str, int]:

`📅 Changelog`

v1.0.0 — Initial release.

Purchases include lifetime updates. Check the product page for the latest version.

`📄 Product Preview`

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview)⚙ Try Demo Tools📦 Download Free Sample

`❓ Frequently Asked Questions`

What license is this under?

MIT License. You can use it in commercial, client, and internal projects. See our license terms page for details.

How do I download after purchase?

Immediately after purchase, you'll be redirected to a download page. A download link is also sent to your email.

Do I get updates?

Yes. All products include lifetime updates at no extra cost. Re-download anytime for the latest version.

What if it doesn't work for me?

DM @DatanestDigital on X and we'll make it right.

Can I get a refund?

DM @DatanestDigital on X with your order number and we'll help.

Is there support?

DM @DatanestDigital on X. We respond within 1-2 business days.



      
        Buy Now — $29
        
        Back to Products