Data Quality Framework

Data Quality Framework

Trust your data. A pluggable quality engine with built-in checks for completeness,

uniqueness, validity, freshness, and consistency — plus automated reporting to Slack,

HTML, and Delta Lake.

By Datanest Digital | Version 1.0.0 | $49

What You Get

• Quality Engine — Rule-based engine that loads checks from YAML, executes them

against any Spark DataFrame, aggregates results, and produces structured reports

• 6 Check Types — Completeness (null/empty), uniqueness (duplicates), validity

(regex, range, enum), freshness (staleness), consistency (cross-table), and custom

(arbitrary SQL expressions)

• 3 Reporters — Slack webhook notifications, standalone HTML reports, and Delta Lake

audit table writer for historical trending

• YAML Configuration — Define rules and thresholds in human-readable YAML; no code

changes needed to add new checks

• Databricks Notebook — Ready-to-run notebook for executing quality checks as a

scheduled job

• Strategy Guide — Best practices for implementing data quality at scale

File Tree

data-quality-framework/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│   ├── quality_engine.py              # Core engine: load, execute, report
│   ├── checks/
│   │   ├── completeness.py            # Null/empty field checks
│   │   ├── uniqueness.py              # Duplicate detection
│   │   ├── validity.py                # Regex, range, enum validation
│   │   ├── freshness.py               # Data staleness checks
│   │   ├── consistency.py             # Cross-table consistency
│   │   └── custom.py                  # Arbitrary SQL expression checks
│   └── reporters/
│       ├── slack_reporter.py          # Slack webhook notifications
│       ├── html_reporter.py           # Standalone HTML report
│       └── delta_reporter.py          # Delta Lake audit table writer
├── configs/
│   ├── quality_rules.yaml             # Rule definitions
│   └── thresholds.yaml                # Pass/warn/fail thresholds
├── notebooks/
│   └── run_quality_checks.py          # Databricks notebook
├── tests/
│   ├── conftest.py                    # Shared fixtures
│   └── test_quality_engine.py         # Unit tests
└── guides/
    └── data-quality-strategy.md       # Best practices guide

Getting Started

Follow this guide to get data quality framework up and running in your environment.

Getting Started

1. Define your quality rules

Edit configs/quality_rules.yaml to specify which checks to run:

rules:
  - name: "customer_email_not_null"
    table: "analytics.silver.customers"
    check_type: "completeness"
    columns: ["email"]
    threshold: 0.99  # 99% must be non-null

  - name: "order_id_unique"
    table: "analytics.silver.orders"
    check_type: "uniqueness"
    columns: ["order_id"]
    threshold: 1.0  # 100% unique

2. Run quality checks

from src.quality_engine import QualityEngine

engine = QualityEngine.from_config(
    rules_path="configs/quality_rules.yaml",
    thresholds_path="configs/thresholds.yaml",
)

# Execute all rules and get a report
report = engine.run_all()
print(report.summary())

# Check if all rules passed
if not report.passed:
    print(f"FAILED: {report.failed_count} of {report.total_count} checks failed")

3. Send notifications

from src.reporters.slack_reporter import SlackReporter
from src.reporters.delta_reporter import DeltaReporter

# Send Slack alert for failures
slack = SlackReporter(webhook_url="https://hooks.slack.com/services/T.../B.../xxx")
slack.send(report)

# Persist results to Delta Lake for trending
delta_reporter = DeltaReporter(audit_table="analytics.ops.quality_audit")
delta_reporter.write(report)

Requirements

• Databricks Runtime 13.3 LTS or later
• Apache Spark 3.4+
• Delta Lake 2.4+
• Python 3.10+
• requests (for Slack reporter)