CDC Replication Toolkit

CDC Replication Toolkit

Production-ready change data capture pipeline for Databricks — Debezium parsing, MERGE INTO application, offset management, and replication monitoring.

By Datanest Digital | Version 1.0.0 | $49

What You Get

• CDC Event Processor — Process insert/update/delete events and apply to target Delta tables
• Debezium Parser — Parse Debezium JSON envelope format with before/after images
• MERGE Applier — Apply CDC changes via MERGE INTO with full operation type support
• Offset Manager — Track Kafka offsets and LSN positions with checkpoint management
• Schema Mapper — Map source database schemas to Delta targets with type conversion
• Replication Monitor — Track replication lag, throughput, and error rates

File Tree

cdc-replication-toolkit/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│  ├── cdc_processor.py      # CDC event processing engine
│  ├── debezium_parser.py     # Debezium JSON format parser
│  ├── merge_applier.py      # MERGE INTO change application
│  ├── offset_manager.py      # Kafka offset & LSN tracking
│  ├── schema_mapper.py      # Source→target schema mapping
│  └── replication_monitor.py   # Lag, throughput, error monitoring
├── configs/
│  ├── replication_config.yaml   # Main replication configuration
│  └── source_mappings/
│    ├── postgres_to_delta.yaml # PostgreSQL type mappings
│    ├── mysql_to_delta.yaml   # MySQL type mappings
│    └── sqlserver_to_delta.yaml # SQL Server type mappings
├── notebooks/
│  ├── start_replication.py    # Start CDC streaming pipeline
│  └── replication_status.py    # Replication status dashboard
├── tests/
│  ├── conftest.py         # Shared fixtures
│  ├── test_cdc_processor.py    # CDC processing tests
│  └── test_debezium_parser.py   # Debezium parsing tests
└── guides/
  └── cdc-replication-guide.md  # CDC patterns & Debezium guide

Getting Started

Follow this guide to get cdc replication toolkit up and running in your environment.

Getting Started

1. Configure Source Mappings

Edit configs/replication_config.yaml with your source database and Kafka settings:

source:
 type: postgres
 kafka_bootstrap_servers: "broker1:9092,broker2:9092"
 topic_prefix: "dbserver1"
 tables:
  - "public.orders"
  - "public.customers"

2. Start the Replication Pipeline

from src.cdc_processor import CDCProcessor

processor = CDCProcessor(config_path="configs/replication_config.yaml")
processor.start_streaming() # Starts structured streaming with checkpoints

3. Parse and Apply Changes

from src.debezium_parser import DebeziumParser
from src.merge_applier import MergeApplier

parser = DebeziumParser()
events = parser.parse_stream(raw_kafka_df)

applier = MergeApplier(target_table="main.bronze.orders", key_columns=["id"])
applier.apply_changes(events)

4. Monitor Replication Health

from src.replication_monitor import ReplicationMonitor

monitor = ReplicationMonitor(config_path="configs/replication_config.yaml")
status = monitor.get_replication_status()
print(f"Lag: {status.lag_seconds}s | Throughput: {status.events_per_second}/s")

Architecture

┌──────────────┐  ┌──────────────┐  ┌──────────────────┐
│ Source DB  │───▶│ Debezium  │───▶│ Kafka Topics  │
│ (PG/MySQL/ │  │ Connector  │  │ (CDC events)  │
│  SQL Server)│  └──────────────┘  └────────┬─────────┘
└──────────────┘                 │
                         ▼
┌────────────────────────────────────────────────────────────┐
│ Databricks                        │
│ ┌─────────────┐ ┌──────────────┐ ┌───────────────────┐ │
│ │ Debezium  │─▶│ CDC     │─▶│ Merge Applier  │ │
│ │ Parser   │ │ Processor  │ │ (MERGE INTO)   │ │
│ └─────────────┘ └──────┬───────┘ └───────────────────┘ │
│             │                 │
│ ┌─────────────┐ ┌──────┴───────┐ ┌───────────────────┐ │
│ │ Schema   │ │ Offset   │ │ Replication   │ │
│ │ Mapper   │ │ Manager   │ │ Monitor     │ │
│ └─────────────┘ └──────────────┘ └───────────────────┘ │
└────────────────────────────────────────────────────────────┘