Pipeline Observability Framework

$1490

Job execution monitoring, health scoring, and lineage/impact analysis from system.lakeflow: run outcomes, durations, success-rate health classification, and downstream blast-radius. Ships a dependency-free library + offline tests. Works on any Databricks workspace.

📁 17 files🏷 v1.0.0

Production-readyUnit-testedDatabricks Asset Bundle

✓ Instant download✓ Lifetime updates✓ MIT licensed✓ Secure checkout (Stripe)

⚙ Try the Live Demo interactive

Enter job run outcomes and see success rate and a Healthy/Degraded/Unhealthy/Critical tier — the framework's health scoring, live.

⚡ Open Health Scorer

📋 What's Inside 17 files

✓ README.md
✓ LICENSE
✓ manifest.json
✓ databricks.yml
✓ resources/jobs.yml
✓ src/01_job_inventory.sql
✓ src/02_run_history.sql
✓ src/03_health_metrics.sql
✓ src/04_lineage_and_impact.sql
✓ lib/job_health.py
✓ tests/test_job_health.py
✓ conftest.py
✓ guide/01_what-you-get.md
✓ guide/02_getting-started.md
✓ guide/03_architecture.md
✓ guide/04_support.md
✓ guides/observability-methodology.md

📁 File Structure 17 files

pipeline-observability-framework/
├── README.md
├── LICENSE
├── manifest.json
├── databricks.yml
├── resources/
│   ├── jobs.yml
├── src/
│   ├── 01_job_inventory.sql
│   ├── 02_run_history.sql
│   ├── 03_health_metrics.sql
│   ├── 04_lineage_and_impact.sql
├── lib/
│   ├── job_health.py
├── tests/
│   ├── test_job_health.py
├── conftest.py
├── guide/
│   ├── 01_what-you-get.md
│   ├── 02_getting-started.md
│   ├── 03_architecture.md
│   ├── 04_support.md
├── guides/
│   ├── observability-methodology.md

📖 Documentation Preview README excerpt

Pipeline Observability Framework

Job execution monitoring, **health scoring**, and lineage/impact analysis from

Databricks `system.lakeflow` tables. Answers the three questions every data team

asks at 3am: **what ran, did it succeed, and what depends on it?**

Works on any Databricks workspace. Fully parameterized, no org-specific deps.

What's inside

- **Databricks Asset Bundle** — a 4-hourly refresh (inventory → run history →

health metrics).

- **Health scoring** — success rate and a Healthy/Degraded/Unhealthy/Critical

classification per job over a 30-day window.

- **Lineage + blast radius** — a job↔table lineage table and a downstream-impact

query that shows which consumers an unhealthy job affects.

- **A dependency-free library** (`lib/job_health.py`) mirroring the scoring +

impact logic, with an offline test suite (5 tests).

Health classification

| Status | Success rate | Action |

|---|---|---|

| Healthy | ≥ 99% | none |

| Degraded | 95–99% | monitor |

| Unhealthy | 80–95% | investigate |

| Critical | < 80% | fix now |

Quickstart

```bash

pip install pytest && pytest tests/ -v

databricks bundle deploy -t dev -p <profile>

```

Set `target_catalog`, `target_schema`, `warehouse_id`. Populate `d_job_lineage`

with your job↔table mappings to enable impact analysis.

License

MIT — see `LICENSE`.

... preview truncated, see full README in product download.

📄 Code Sample .sql preview

src/01_job_inventory.sql-- Pipeline Observability: Job Inventory — all active jobs, categorized. CREATE OR REPLACE TABLE IDENTIFIER(:target_catalog || '.' || :target_schema || '.d_jobs') AS SELECT job_id, name AS job_name, creator_id, run_as, create_time, CASE WHEN LOWER(name) LIKE '%dlt%' THEN 'DLT Pipeline' WHEN LOWER(name) LIKE '%ingest%' OR LOWER(name) LIKE '%bronze%' THEN 'Ingestion' WHEN LOWER(name) LIKE '%silver%' OR LOWER(name) LIKE '%transform%' THEN 'Transformation' WHEN LOWER(name) LIKE '%gold%' OR LOWER(name) LIKE '%model%' THEN 'Gold Model' WHEN LOWER(name) LIKE '%monitor%' OR LOWER(name) LIKE '%quality%' THEN 'Monitoring' ELSE 'Other' END AS job_category, CURRENT_TIMESTAMP() AS _load_timestamp FROM system.lakeflow.jobs WHERE delete_time IS NULL;

Buy Now — $1490 ⚙ Open Health Scorer Back to Products