PySpark Utils Library

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 27 files

pyspark-utils-library/ ├── LICENSE ├── README.md ├── examples/ │ └── usage_examples.py ├── free-sample.zip ├── guide/ │ ├── 01_what-s-inside.md │ ├── 02_module-deep-dives.md │ ├── 03_testing.md │ └── 04_support.md ├── index.html ├── pyproject.toml ├── pyspark_utils/ │ ├── __init__.py │ ├── data_quality.py │ ├── dedup.py │ ├── diff.py │ ├── logging_utils.py │ ├── scd.py │ ├── schema_utils.py │ └── transformations.py ├── setup.py └── tests/ ├── __init__.py ├── conftest.py ├── test_data_quality.py ├── test_dedup.py ├── test_diff.py ├── test_scd.py ├── test_schema_utils.py └── test_transformations.py

📖 Documentation Preview README excerpt

Battle-tested utility functions for PySpark data engineering — transformations, data quality, SCD, schema evolution, logging, dedup, and DataFrame diffing.

Stop rewriting the same PySpark boilerplate on every project. This library gives you the production-ready building blocks that data engineering teams use daily — fully typed, tested, and documented.

What's Inside

Module	What It Does
transformations	15 reusable DataFrame transforms: column cleaning, casting, flattening, pivoting, hashing
data_quality	Chainable DQ validation framework with structured reports and severity levels
scd	SCD Type 1 (overwrite) and Type 2 (full history) merge utilities for Delta Lake
schema_utils	Schema comparison, evolution, DDL conversion, and compatibility checking
logging_utils	Structured pipeline logging with correlation IDs, metrics, and Delta table sink
dedup	Window-based, hash-based, and fuzzy deduplication strategies
diff	DataFrame comparison with row-level, column-level, and schema diffs

20 files — every one fully type-hinted, tested, and documented.

Quick Start

1. Install


pip install pyspark-utils-library

Or install from the source directory:


pip install -e .

For fuzzy deduplication support (Levenshtein distance):


pip install pyspark-utils-library[fuzzy]

For development (testing, linting, type checking):


pip install pyspark-utils-library[dev]

2. Import and Use


from pyspark_utils.transformations import clean_column_names, add_metadata_columns
from pyspark_utils.data_quality import DQValidator
from pyspark_utils.scd import scd2_merge
from pyspark_utils.logging_utils import PipelineLogger

# Clean and enrich a raw DataFrame
df = spark.read.table("bronze.raw_orders")

*... continues with setup instructions, usage examples, and more.*

📄 Code Sample .py preview

setup.py """Setup configuration for pyspark-utils-library.""" from setuptools import setup, find_packages with open("README.md", "r", encoding="utf-8") as fh: long_description = fh.read() setup( name="pyspark-utils-library", version="1.0.0", author="DataStack Pro", author_email="support@datastackpro.com", description=( "Battle-tested PySpark utility library for transformations, " "data quality, SCD, schema evolution, logging, dedup, and diff." ), long_description=long_description, long_description_content_type="text/markdown", url="https://datastackpro.com/products/pyspark-utils-library", project_urls={ "Documentation": "https://datastackpro.com/docs/pyspark-utils", "Bug Tracker": "https://github.com/datastackpro/pyspark-utils-library/issues", "Source": "https://github.com/datastackpro/pyspark-utils-library", }, packages=find_packages(exclude=["tests", "tests.*", "examples"]), classifiers=[ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "Topic :: Software Development :: Libraries :: Python Modules", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Operating System :: OS Independent", "Typing :: Typed", ], python_requires=">=3.9", install_requires=[ "pyspark>=3.3.0", "delta-spark>=2.3.0", ], extras_require={ "dev": [ "pytest>=7.0", "pytest-cov>=4.0", "mypy>=1.0", "ruff>=0.1.0", ], # ... 17 more lines ...