PySpark Utils Library

$29

Battle-tested PySpark utility functions for transformations, data quality, SCD, schema evolution, and lineage.

🏷 v1.0.0

Production-ready

✓ Instant download✓ Lifetime updates✓ MIT licensed✓ MIT license✓ Secure checkout (Stripe)

📁 File Structure 28 files

pyspark-utils-library/
├── LICENSE
├── README.md
├── examples/
│ ├── usage_examples.py
├── free-sample.zip
├── guide/
│ ├── 01_what-s-inside.md
│ ├── 02_module-deep-dives.md
│ ├── 03_testing.md
│ ├── 04_support.md
├── index.html
├── interactive.html
├── pyproject.toml
├── pyspark_utils/
│ ├── __init__.py
│ ├── data_quality.py
│ ├── dedup.py
│ ├── diff.py
│ ├── logging_utils.py
│ ├── scd.py
│ ├── schema_utils.py
│ ├── transformations.py
├── setup.py
├── tests/
│ ├── __init__.py
│ ├── conftest.py
│ ├── test_data_quality.py
│ ├── test_dedup.py
│ ├── test_diff.py
│ ├── test_scd.py
│ ├── test_schema_utils.py
│ ├── test_transformations.py

📖 Documentation Preview README excerpt

PySpark Utils Library

Battle-tested utility functions for PySpark data engineering — transformations, data quality, SCD, schema evolution, logging, dedup, and DataFrame diffing.

Stop rewriting the same PySpark boilerplate on every project. This library gives you the production-ready building blocks that data engineering teams use daily — fully typed, tested, and documented.

---

What's Inside

Module	What It Does
transformations	15 reusable DataFrame transforms: column cleaning, casting, flattening, pivoting, hashing
data_quality	Chainable DQ validation framework with structured reports and severity levels
scd	SCD Type 1 (overwrite) and Type 2 (full history) merge utilities for Delta Lake
schema_utils	Schema comparison, evolution, DDL conversion, and compatibility checking
logging_utils	Structured pipeline logging with correlation IDs, metrics, and Delta table sink
dedup	Window-based, hash-based, and fuzzy deduplication strategies
diff	DataFrame comparison with row-level, column-level, and schema diffs

20 files — every one fully type-hinted, tested, and documented.

---

Quick Start

1. Install


pip install pyspark-utils-library


Or install from the source directory:


pip install -e .


For fuzzy deduplication support (Levenshtein distance):


pip install pyspark-utils-library[fuzzy]


For development (testing, linting, type checking):


pip install pyspark-utils-library[dev]


2. Import and Use


from pyspark_utils.transformations import clean_column_names, add_metadata_columns
from pyspark_utils.data_quality import DQValidator
from pyspark_utils.scd import scd2_merge
from pyspark_utils.logging_utils import PipelineLogger

Clean and enrich a raw DataFrame
df = spark.read.table("bronze.raw_orders")

... preview truncated, see full README in product download.

`📄 Code Sample .py preview`

examples/usage_examples.py"""
PySpark Utils Library — Usage Examples
======================================

This file demonstrates every module in the pyspark_utils library using a
Databricks notebook-style format.  Each section is self-contained and can
be run independently in a Databricks notebook by splitting at the ``# COMMAND``
markers.

Prerequisites:
    pip install pyspark-utils-library
    # or upload the wheel to your Databricks cluster

    # In Databricks, the SparkSession is available as `spark` by default.
"""

# COMMAND ----------
# Setup — Create SparkSession (not needed in Databricks)

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import (
    ArrayType,
    DoubleType,
    IntegerType,
    StringType,
    StructField,
    StructType,
)

spark = SparkSession.builder.appName("pyspark-utils-examples").getOrCreate()

# COMMAND ----------
# ============================================================
# 1. TRANSFORMATIONS MODULE
# ============================================================

from pyspark_utils.transformations import (
    add_metadata_columns,
    add_surrogate_key,

`📅 Changelog`

v1.0.0 — Initial release.

Purchases include lifetime updates. Check the product page for the latest version.

`📄 Product Preview`

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview)⚙ Try Demo Tools📦 Download Free Sample

`❓ Frequently Asked Questions`

What license is this under?

MIT License. You can use it in commercial, client, and internal projects. See our license terms page for details.

How do I download after purchase?

Immediately after purchase, you'll be redirected to a download page. A download link is also sent to your email.

Do I get updates?

Yes. All products include lifetime updates at no extra cost. Re-download anytime for the latest version.

What if it doesn't work for me?

DM @DatanestDigital on X and we'll make it right.

Can I get a refund?

DM @DatanestDigital on X with your order number and we'll help.

Is there support?

DM @DatanestDigital on X. We respond within 1-2 business days.



      
        Buy Now — $29
        
        Back to Products