Streaming Pipeline Kit

Streaming Pipeline Kit

Real-time data pipelines that just work. Production-ready Spark Structured Streaming

templates with Kafka integration, exactly-once semantics, and Delta Lake sinks.

By Datanest Digital | Version 1.0.0 | $49

What You Get

• Kafka Consumer — Structured Streaming reader with schema registry integration,

watermarking, and configurable checkpointing

• Event Processor — Deduplication by event ID, late-arrival handling, and windowed

aggregations with customizable window sizes

• Delta Lake Writer — foreachBatch sink with merge/append modes, schema evolution,

and inline data quality checks

• Stream Monitor — Query progress listener, consumer lag tracking, dead letter queue

routing, and alerting hooks

• Schema Registry Client — Confluent-compatible client for fetching, registering,

and validating Avro schemas with compatibility checks

• Databricks Notebooks — Ready-to-run notebooks for starting streams and monitoring

active queries in real time

• Avro Schemas — Example schemas for user events and order events
• Streaming Patterns Guide — Best practices for watermarks, triggers, state management,

and failure recovery

File Tree

streaming-pipeline-kit/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│   ├── kafka_consumer.py          # Kafka source with schema registry
│   ├── event_processor.py         # Dedup, late arrivals, windowed aggs
│   ├── stream_to_delta.py         # foreachBatch Delta Lake writer
│   ├── stream_monitor.py          # Progress listener & lag monitoring
│   └── schema_registry.py         # Schema Registry client
├── configs/
│   ├── streaming_config.yaml      # Kafka, checkpoint, trigger settings
│   └── schemas/
│       ├── user_events.avsc       # User event Avro schema
│       └── order_events.avsc      # Order event Avro schema
├── notebooks/
│   ├── start_stream.py            # Launch streaming pipeline
│   └── monitor_streams.py         # Real-time monitoring dashboard
├── tests/
│   └── test_event_processor.py    # Unit tests for event processing
└── guides/
    └── streaming-patterns.md      # Patterns & best practices

Getting Started

Follow this guide to get streaming pipeline kit up and running in your environment.

Getting Started

1. Configure your environment

Edit configs/streaming_config.yaml with your Kafka bootstrap servers,

schema registry URL, and checkpoint locations:

kafka:
  bootstrap_servers: "broker1:9092,broker2:9092"
  topic: "user-events"
  consumer_group: "streaming-pipeline-v1"
  starting_offsets: "latest"

checkpoint:
  location: "/mnt/checkpoints/user-events"

2. Start a streaming pipeline

from src.kafka_consumer import KafkaStreamReader
from src.event_processor import EventProcessor
from src.stream_to_delta import DeltaStreamWriter

# Initialize components
reader = KafkaStreamReader.from_config("configs/streaming_config.yaml")
processor = EventProcessor(
    dedup_column="event_id",
    watermark_delay="10 minutes",
)
writer = DeltaStreamWriter(
    target_table="analytics.silver.user_events",
    merge_keys=["event_id"],
)

# Build and start the pipeline
stream_df = reader.read_stream()
processed_df = processor.process(stream_df)
query = writer.start(processed_df)

3. Monitor active streams

from src.stream_monitor import StreamMonitor

monitor = StreamMonitor(alert_on_lag_seconds=300)
monitor.attach_to_all_queries()
monitor.print_status()

Requirements

• Databricks Runtime 13.3 LTS or later
• Apache Spark 3.4+
• Apache Kafka 2.8+ (or Confluent Cloud)
• Delta Lake 2.4+
• Python 3.10+
• Confluent Schema Registry (optional, for Avro schema management)