Data Quality Monitoring Framework

$1490

Cross-cutting data-quality layer for any pipeline: freshness, completeness/documentation, and row-count anomalies with a standardized per-schema summary from system.information_schema. Ships a dependency-free checks library + offline tests. Works on any Unity Catalog workspace.

📁 17 files🏷 v1.0.0

Production-readyUnit-testedDatabricks Asset Bundle

✓ Instant download✓ Lifetime updates✓ MIT licensed✓ Secure checkout (Stripe)

⚙ Try the Live Demo interactive

Enter freshness, null rate and row counts and get a live data-quality score and status — the same checks the framework runs.

⚡ Open Quality Scorer

📋 What's Inside 17 files

✓ README.md
✓ LICENSE
✓ manifest.json
✓ databricks.yml
✓ resources/jobs.yml
✓ src/01_freshness_scan.sql
✓ src/02_completeness_scan.sql
✓ src/03_row_count_anomalies.sql
✓ src/04_quality_summary.sql
✓ lib/dq_checks.py
✓ tests/test_dq_checks.py
✓ conftest.py
✓ guide/01_what-you-get.md
✓ guide/02_getting-started.md
✓ guide/03_architecture.md
✓ guide/04_support.md
✓ guides/data-quality-methodology.md

📁 File Structure 17 files

gf-data-quality-framework/
├── README.md
├── LICENSE
├── manifest.json
├── databricks.yml
├── resources/
│   ├── jobs.yml
├── src/
│   ├── 01_freshness_scan.sql
│   ├── 02_completeness_scan.sql
│   ├── 03_row_count_anomalies.sql
│   ├── 04_quality_summary.sql
├── lib/
│   ├── dq_checks.py
├── tests/
│   ├── test_dq_checks.py
├── conftest.py
├── guide/
│   ├── 01_what-you-get.md
│   ├── 02_getting-started.md
│   ├── 03_architecture.md
│   ├── 04_support.md
├── guides/
│   ├── data-quality-methodology.md

📖 Documentation Preview README excerpt

Data Quality Monitoring Framework

A cross-cutting **quality layer** you can attach to any pipeline. Tracks

freshness, completeness/documentation, and row-count anomalies, and produces a

standardized summary per schema — built on `system.information_schema`, so it

works on **any** Unity Catalog workspace with no per-table wiring.

Fully generic and environment-agnostic: point it at a list of catalogs and

deploy.

What's inside

- **Databricks Asset Bundle** — a daily scan job (freshness, completeness,

row-count history → summary).

- **Metadata-driven scans** — no per-table SQL to maintain; monitored catalogs

are a single variable.

- **A dependency-free checks library** (`lib/dq_checks.py`): freshness/

completeness classification, null rates, z-score row-count anomalies, and a

0–100 quality score — with an offline test suite (7 tests).

Quickstart

```bash

pip install pytest && pytest tests/ -v # validate check logic offline

databricks bundle deploy -t dev -p <profile>

```

Set `target_catalog`, `target_schema`, `monitored_catalogs`, `warehouse_id`.

License

MIT — see `LICENSE`.

... preview truncated, see full README in product download.

📄 Code Sample .sql preview

src/01_freshness_scan.sql-- Data Quality: Freshness Scan — hours since each table was last updated. CREATE OR REPLACE TABLE IDENTIFIER(:target_catalog || '.' || :target_schema || '.f_freshness') AS SELECT table_catalog AS catalog_name, table_schema AS schema_name, table_name, last_altered AS last_modified_at, TIMESTAMPDIFF(HOUR, last_altered, CURRENT_TIMESTAMP()) AS staleness_hours, CASE WHEN TIMESTAMPDIFF(HOUR, last_altered, CURRENT_TIMESTAMP()) <= 24 THEN 'OK' WHEN TIMESTAMPDIFF(HOUR, last_altered, CURRENT_TIMESTAMP()) <= 72 THEN 'WARNING' ELSE 'CRITICAL' END AS freshness_status, CURRENT_TIMESTAMP() AS scanned_at FROM system.information_schema.tables WHERE table_catalog IN (SELECT TRIM(value) FROM (SELECT EXPLODE(SPLIT(:monitored_catalogs, ',')) AS value)) AND table_schema NOT LIKE '%__dlt_materialization_schema%' AND table_type IN ('MANAGED', 'EXTERNAL');

Buy Now — $1490 ⚙ Open Quality Scorer Back to Products