← Back to all products

Data Pipeline Testing Kit

$29

Comprehensive testing framework for PySpark data pipelines from unit tests to integration validation.

📁 17 files🏷 v1.0.0

Production-ready

✓ Instant download✓ Lifetime updates✓ MIT licensed✓ MIT license✓ Secure checkout (Stripe)

📋 What's Inside 17 files

✓ README.md—
✓ manifest.json—
✓ LICENSE—
✓ src/test_framework.py—
✓ src/data_generators.py—
✓ src/assertions.py—
✓ src/mock_utils.py—
✓ src/snapshot_testing.py—
✓ fixtures/sample_customers.json—
✓ fixtures/sample_orders.json—
✓ fixtures/expected_outputs/customer_summary.json—
✓ tests/conftest.py—
✓ tests/test_bronze_pipeline.py—
✓ tests/test_silver_pipeline.py—
✓ tests/test_gold_pipeline.py—

📁 File Structure 17 files

data-pipeline-testing/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│ ├── test_framework.py
│ ├── data_generators.py
│ ├── assertions.py
│ ├── mock_utils.py
│ ├── snapshot_testing.py
├── fixtures/
│ ├── sample_customers.json
│ ├── sample_orders.json
│ ├── expected_outputs/
│ │ ├── customer_summary.json
├── tests/
│ ├── conftest.py
│ ├── test_bronze_pipeline.py
│ ├── test_silver_pipeline.py
│ ├── test_gold_pipeline.py
├── configs/
│ ├── test_config.yaml
├── guides/
│ ├── testing-data-pipelines.md

📖 Documentation Preview README excerpt

Data Pipeline Testing Kit

Comprehensive testing framework for PySpark data pipelines — from unit tests to integration validation.

By [Datanest Digital](https://datanest.dev) | Version 1.0.0 | $39

---

What You Get

A complete testing toolkit for data pipelines running on Databricks and PySpark, including:

**Test Framework** — base classes and runners for PySpark unit/integration tests

**Data Generators** — realistic synthetic data factories for customers, orders, events

**Custom Assertions** — DataFrame-level assertions for schema, row count, nulls, uniqueness

**Mock Utilities** — helpers for mocking `spark`, `dbutils`, Delta tables, and external APIs

**Snapshot Testing** — golden-file comparison for pipeline output validation

**Sample Fixtures** — ready-to-use JSON test data (customers, orders, expected outputs)

**Pipeline Tests** — complete examples testing bronze, silver, and gold layers

File Tree


data-pipeline-testing/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│   ├── test_framework.py        # Base test classes and PySpark test runner
│   ├── data_generators.py       # Synthetic data factories
│   ├── assertions.py            # DataFrame assertion library
│   ├── mock_utils.py            # Spark/dbutils/Delta mocking helpers
│   └── snapshot_testing.py      # Golden-file snapshot comparison
├── fixtures/
│   ├── sample_customers.json    # 50 customer records
│   ├── sample_orders.json       # 100 order records
│   └── expected_outputs/
│       └── customer_summary.json
├── tests/
│   ├── conftest.py              # Shared pytest fixtures with SparkSession
│   ├── test_bronze_pipeline.py  # Bronze layer ingestion tests
│   ├── test_silver_pipeline.py  # Silver layer transformation tests
│   └── test_gold_pipeline.py    # Gold layer aggregation tests
├── configs/
│   └── test_config.yaml         # Test environment configuration
└── guides/
    └── testing-data-pipelines.md


Getting Started

1. Install Dependencies


pip install pyspark delta-spark pytest pyyaml


2. Use the Test Framework


... preview truncated, see full README in product download.

`📄 Code Sample .py preview`

src/test_framework.py"""
Data Pipeline Testing Kit — Test Framework
By Datanest Digital (https://datanest.dev) | Version 1.0.0

Base test classes and utilities for PySpark data pipeline testing.
Provides SparkTestCase with built-in DataFrame assertions, temp table
management, and Delta Lake test helpers.

Usage:
    from test_framework import SparkTestCase

    class TestMyPipeline(SparkTestCase):
        def test_transform(self):
            df = self.create_dataframe(
                [("Alice", 100), ("Bob", 200)],
                schema=["name", "amount"]
            )
            result = my_transform(df)
            self.assert_row_count(result, 2)
"""

from __future__ import annotations

import logging
import tempfile
import shutil
from pathlib import Path
from typing import Any, Dict, List, Optional, Sequence, Tuple, Union
from dataclasses import dataclass, field
from datetime import datetime

from pyspark.sql import SparkSession, DataFrame
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
import pytest

logger = logging.getLogger(__name__)


# =============================================================================
# Configuration

`📅 Changelog`

v1.0.0 — Initial release.

Purchases include lifetime updates. Check the product page for the latest version.

`📄 Product Preview`

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview)⚙ Try Demo Tools📦 Download Free Sample

`❓ Frequently Asked Questions`

What license is this under?

MIT License. You can use it in commercial, client, and internal projects. See our license terms page for details.

How do I download after purchase?

Immediately after purchase, you'll be redirected to a download page. A download link is also sent to your email.

Do I get updates?

Yes. All products include lifetime updates at no extra cost. Re-download anytime for the latest version.

What if it doesn't work for me?

DM @DatanestDigital on X and we'll make it right.

Can I get a refund?

DM @DatanestDigital on X with your order number and we'll help.

Is there support?

DM @DatanestDigital on X. We respond within 1-2 business days.



      
        Buy Now — $29
        
        Back to Products