Streaming Pipeline Kit

Streaming Pipeline Kit

Real-time data pipelines that just work. Production-ready Spark Structured Streaming

templates with Kafka integration, exactly-once semantics, and Delta Lake sinks.

By Datanest Digital | Version 1.0.0 | $49

What You Get

• Kafka Consumer — Structured Streaming reader with schema registry integration,

watermarking, and configurable checkpointing

• Event Processor — Deduplication by event ID, late-arrival handling, and windowed

aggregations with customizable window sizes

• Delta Lake Writer — foreachBatch sink with merge/append modes, schema evolution,

and inline data quality checks

• Stream Monitor — Query progress listener, consumer lag tracking, dead letter queue

routing, and alerting hooks

• Schema Registry Client — Confluent-compatible client for fetching, registering,

and validating Avro schemas with compatibility checks

• Databricks Notebooks — Ready-to-run notebooks for starting streams and monitoring

active queries in real time

• Avro Schemas — Example schemas for user events and order events
• Streaming Patterns Guide — Best practices for watermarks, triggers, state management,

and failure recovery

File Tree

streaming-pipeline-kit/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│  ├── kafka_consumer.py     # Kafka source with schema registry
│  ├── event_processor.py     # Dedup, late arrivals, windowed aggs
│  ├── stream_to_delta.py     # foreachBatch Delta Lake writer
│  ├── stream_monitor.py     # Progress listener & lag monitoring
│  └── schema_registry.py     # Schema Registry client
├── configs/
│  ├── streaming_config.yaml   # Kafka, checkpoint, trigger settings
│  └── schemas/
│    ├── user_events.avsc    # User event Avro schema
│    └── order_events.avsc   # Order event Avro schema
├── notebooks/
│  ├── start_stream.py      # Launch streaming pipeline
│  └── monitor_streams.py     # Real-time monitoring dashboard
├── tests/
│  └── test_event_processor.py  # Unit tests for event processing
└── guides/
  └── streaming-patterns.md   # Patterns & best practices

Getting Started

Follow this guide to get streaming pipeline kit up and running in your environment.

Getting Started

1. Configure your environment

Edit configs/streaming_config.yaml with your Kafka bootstrap servers,

schema registry URL, and checkpoint locations:

kafka:
 bootstrap_servers: "broker1:9092,broker2:9092"
 topic: "user-events"
 consumer_group: "streaming-pipeline-v1"
 starting_offsets: "latest"

checkpoint:
 location: "/mnt/checkpoints/user-events"

2. Start a streaming pipeline

from src.kafka_consumer import KafkaStreamReader
from src.event_processor import EventProcessor
from src.stream_to_delta import DeltaStreamWriter

# Initialize components
reader = KafkaStreamReader.from_config("configs/streaming_config.yaml")
processor = EventProcessor(
  dedup_column="event_id",
  watermark_delay="10 minutes",
)
writer = DeltaStreamWriter(
  target_table="analytics.silver.user_events",
  merge_keys=["event_id"],
)

# Build and start the pipeline
stream_df = reader.read_stream()
processed_df = processor.process(stream_df)
query = writer.start(processed_df)

3. Monitor active streams

from src.stream_monitor import StreamMonitor

monitor = StreamMonitor(alert_on_lag_seconds=300)
monitor.attach_to_all_queries()
monitor.print_status()

Requirements

• Databricks Runtime 13.3 LTS or later
• Apache Spark 3.4+
• Apache Kafka 2.8+ (or Confluent Cloud)
• Delta Lake 2.4+
• Python 3.10+
• Confluent Schema Registry (optional, for Avro schema management)