← Back to all products

CDC Replication Toolkit

$29

Change data capture pipeline for Databricks with Debezium parsing, merge into, and offset management.

📁 19 files🏷 v1.0.0

Production-ready

✓ Instant download✓ Lifetime updates✓ MIT licensed✓ MIT license✓ Secure checkout (Stripe)

📋 What's Inside 19 files

✓ README.md—
✓ manifest.json—
✓ LICENSE—
✓ src/cdc_processor.py—
✓ src/debezium_parser.py—
✓ src/merge_applier.py—
✓ src/offset_manager.py—
✓ src/schema_mapper.py—
✓ src/replication_monitor.py—
✓ configs/replication_config.yaml—
✓ configs/source_mappings/postgres_to_delta.yaml—
✓ configs/source_mappings/mysql_to_delta.yaml—
✓ configs/source_mappings/sqlserver_to_delta.yaml—
✓ notebooks/start_replication.py—
✓ notebooks/replication_status.py—

📁 File Structure 19 files

cdc-replication-toolkit/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│ ├── cdc_processor.py
│ ├── debezium_parser.py
│ ├── merge_applier.py
│ ├── offset_manager.py
│ ├── schema_mapper.py
│ ├── replication_monitor.py
├── configs/
│ ├── replication_config.yaml
│ ├── source_mappings/
│ │ ├── postgres_to_delta.yaml
│ │ ├── mysql_to_delta.yaml
│ │ ├── sqlserver_to_delta.yaml
├── notebooks/
│ ├── start_replication.py
│ ├── replication_status.py
├── tests/
│ ├── conftest.py
│ ├── test_cdc_processor.py
│ ├── test_debezium_parser.py
├── guides/
│ ├── cdc-replication-guide.md

📖 Documentation Preview README excerpt

CDC Replication Toolkit

Production-ready change data capture pipeline for Databricks — Debezium parsing, MERGE INTO application, offset management, and replication monitoring.

> By [Datanest Digital](https://datanest.dev) | Version 1.0.0 | $49

---

What You Get

**CDC Event Processor** — Process insert/update/delete events and apply to target Delta tables

**Debezium Parser** — Parse Debezium JSON envelope format with before/after images

**MERGE Applier** — Apply CDC changes via `MERGE INTO` with full operation type support

**Offset Manager** — Track Kafka offsets and LSN positions with checkpoint management

**Schema Mapper** — Map source database schemas to Delta targets with type conversion

**Replication Monitor** — Track replication lag, throughput, and error rates

File Tree


cdc-replication-toolkit/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│   ├── cdc_processor.py            # CDC event processing engine
│   ├── debezium_parser.py          # Debezium JSON format parser
│   ├── merge_applier.py            # MERGE INTO change application
│   ├── offset_manager.py           # Kafka offset & LSN tracking
│   ├── schema_mapper.py            # Source→target schema mapping
│   └── replication_monitor.py      # Lag, throughput, error monitoring
├── configs/
│   ├── replication_config.yaml     # Main replication configuration
│   └── source_mappings/
│       ├── postgres_to_delta.yaml  # PostgreSQL type mappings
│       ├── mysql_to_delta.yaml     # MySQL type mappings
│       └── sqlserver_to_delta.yaml # SQL Server type mappings
├── notebooks/
│   ├── start_replication.py        # Start CDC streaming pipeline
│   └── replication_status.py       # Replication status dashboard
├── tests/
│   ├── conftest.py                 # Shared fixtures
│   ├── test_cdc_processor.py       # CDC processing tests
│   └── test_debezium_parser.py     # Debezium parsing tests
└── guides/
    └── cdc-replication-guide.md    # CDC patterns & Debezium guide


Getting Started

1. Configure Source Mappings

Edit configs/replication_config.yaml with your source database and Kafka settings:


source:
  type: postgres
  kafka_bootstrap_servers: "broker1:9092,broker2:9092"
  topic_prefix: "dbserver1"
  tables:

... preview truncated, see full README in product download.

`📄 Code Sample .py preview`

src/cdc_processor.py"""
CDC Processor — Process CDC events from Kafka and apply to target Delta tables.

Orchestrates the end-to-end CDC pipeline: read from Kafka, parse Debezium
envelopes, map schemas, apply changes via MERGE INTO, and manage offsets.

Author: Datanest Digital
"""

from __future__ import annotations

import logging
from dataclasses import dataclass, field
from datetime import datetime
from enum import Enum
from typing import Any, Dict, List, Optional

import yaml
from pyspark.sql import DataFrame, SparkSession
from pyspark.sql import functions as F
from pyspark.sql.streaming import StreamingQuery
from pyspark.sql.types import StringType, StructField, StructType

logger = logging.getLogger(__name__)


# ---------------------------------------------------------------------------
# Data models
# ---------------------------------------------------------------------------

class OperationType(str, Enum):
    """CDC operation types aligned with Debezium conventions."""

    CREATE = "c"
    UPDATE = "u"
    DELETE = "d"
    READ = "r"  # Snapshot read


@dataclass

`📅 Changelog`

v1.0.0 — Initial release.

Purchases include lifetime updates. Check the product page for the latest version.

`📄 Product Preview`

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview)⚙ Try Demo Tools📦 Download Free Sample

`❓ Frequently Asked Questions`

What license is this under?

MIT License. You can use it in commercial, client, and internal projects. See our license terms page for details.

How do I download after purchase?

Immediately after purchase, you'll be redirected to a download page. A download link is also sent to your email.

Do I get updates?

Yes. All products include lifetime updates at no extra cost. Re-download anytime for the latest version.

What if it doesn't work for me?

DM @DatanestDigital on X and we'll make it right.

Can I get a refund?

DM @DatanestDigital on X with your order number and we'll help.

Is there support?

DM @DatanestDigital on X. We respond within 1-2 business days.



      
        Buy Now — $29
        
        Back to Products