← Back to all products

Spark ETL Framework

$29

Production-ready medallion architecture ETL framework for Databricks and Apache Spark.

📁 16 files🏷 v1.0.0

Production-ready

✓ Instant download✓ Lifetime updates✓ MIT licensed✓ MIT license✓ Secure checkout (Stripe)

📋 What's Inside 16 files

✓ README.md—
✓ manifest.json—
✓ LICENSE—
✓ src/etl_base.py—
✓ src/bronze_loader.py—
✓ src/silver_transformer.py—
✓ src/gold_aggregator.py—
✓ src/quality_gate.py—
✓ src/config_manager.py—
✓ configs/pipeline_config.yaml—
✓ configs/quality_rules.yaml—
✓ notebooks/run_pipeline.py—
✓ notebooks/backfill.py—
✓ tests/conftest.py—
✓ tests/test_etl_base.py—

📁 File Structure 16 files

spark-etl-framework/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│ ├── etl_base.py
│ ├── bronze_loader.py
│ ├── silver_transformer.py
│ ├── gold_aggregator.py
│ ├── quality_gate.py
│ ├── config_manager.py
├── configs/
│ ├── pipeline_config.yaml
│ ├── quality_rules.yaml
├── notebooks/
│ ├── run_pipeline.py
│ ├── backfill.py
├── tests/
│ ├── conftest.py
│ ├── test_etl_base.py
├── guides/
│ ├── etl-patterns.md

📖 Documentation Preview README excerpt

Spark ETL Framework

Production-ready medallion architecture ETL framework for Databricks and Apache Spark.

> Build reliable, observable, and maintainable data pipelines with a battle-tested extract-transform-load pattern that scales from prototype to petabyte.

---

What You Get

**Abstract ETL base class** with built-in logging, metrics collection, and error handling

**Medallion architecture** (Bronze / Silver / Gold) with production patterns baked in

**Data quality gates** between every layer — catch issues before they propagate

**YAML-driven configuration** with environment overrides and secret scope integration

**Source extractors** for JDBC databases, file systems, and REST APIs with pagination and retry logic

**SCD handler** for Type 1 and Type 2 slowly changing dimension merges with soft deletes

**Deduplication engine** with exact, window, fuzzy, and hash-based strategies

**Metrics collector** for pipeline observability, SLA tracking, and batch comparison

**Lineage tracker** for data provenance, impact analysis, and Mermaid diagram export

**Databricks notebooks** for orchestration and date-range backfills

**Comprehensive test suite** with mock Spark sessions and fixture data

**Architecture guide** covering idempotency, partitioning, extraction, SCD, and dedup patterns

File Tree


spark-etl-framework/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│   ├── etl_base.py                    # Abstract base ETL class
│   ├── bronze_loader.py               # Bronze layer ingestion
│   ├── silver_transformer.py          # Silver layer transforms
│   ├── gold_aggregator.py             # Gold layer aggregations
│   ├── quality_gate.py                # Inter-layer quality checks
│   ├── config_manager.py              # YAML config + env overrides
│   ├── extractors/
│   │   ├── __init__.py
│   │   ├── jdbc_extractor.py          # JDBC database extraction (PostgreSQL, SQL Server, Oracle, MySQL)
│   │   ├── file_extractor.py          # File extraction (CSV, JSON, Parquet, Avro, ORC)
│   │   └── api_extractor.py           # REST API extraction with pagination & rate limiting
│   ├── transformers/
│   │   ├── __init__.py
│   │   ├── scd_handler.py             # SCD Type 1 & 2 merge operations
│   │   └── deduplication.py           # Dedup strategies (exact, window, fuzzy, hash)
│   └── utils/
│       ├── __init__.py
│       ├── metrics_collector.py       # Pipeline execution metrics & SLA tracking
│       └── lineage_tracker.py         # Data lineage DAG with Mermaid export
├── configs/
│   ├── pipeline_config.yaml           # Pipeline configuration
│   └── quality_rules.yaml             # Quality rule definitions
├── notebooks/
│   ├── run_pipeline.py                # Orchestration entry point
│   └── backfill.py                    # Date-range backfill utility
├── tests/
│   ├── conftest.py                    # Spark fixtures & sample data
│   ├── test_etl_base.py              # Core framework & utility tests
│   └── test_extractors.py            # Extractor unit tests

... preview truncated, see full README in product download.

`📄 Code Sample .py preview`

src/etl_base.py"""
Abstract Base ETL Class — Spark ETL Framework
================================================
Provides a reusable extract-transform-load pattern with built-in logging,
metrics collection, error handling, and layer-aware orchestration.

Usage:
    Subclass ``ETLBase`` and implement ``extract()``, ``transform()``,
    and ``load()`` for each medallion layer.

By Datanest Digital — https://datanest.dev
"""

from __future__ import annotations

import logging
import time
import uuid
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional

from pyspark.sql import DataFrame, SparkSession

# ---------------------------------------------------------------------------
# Metrics dataclass
# ---------------------------------------------------------------------------


@dataclass
class ETLMetrics:
    """Container for pipeline execution metrics."""

    batch_id: str = ""
    layer: str = ""
    pipeline_name: str = ""
    started_at: Optional[str] = None
    finished_at: Optional[str] = None
    duration_seconds: float = 0.0

`📅 Changelog`

v1.0.0 — Initial release.

Purchases include lifetime updates. Check the product page for the latest version.

`📄 Product Preview`

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview)⚙ Try Demo Tools📦 Download Free Sample

`❓ Frequently Asked Questions`

What license is this under?

MIT License. You can use it in commercial, client, and internal projects. See our license terms page for details.

How do I download after purchase?

Immediately after purchase, you'll be redirected to a download page. A download link is also sent to your email.

Do I get updates?

Yes. All products include lifetime updates at no extra cost. Re-download anytime for the latest version.

What if it doesn't work for me?

DM @DatanestDigital on X and we'll make it right.

Can I get a refund?

DM @DatanestDigital on X with your order number and we'll help.

Is there support?

DM @DatanestDigital on X. We respond within 1-2 business days.



      
        Buy Now — $29
        
        Back to Products