Ingestion Latency Monitoring Framework

$990

End-to-end ingestion latency across medallion layers: batch-to-batch lag and source-to-landing time with p50/p95/p99 summaries and SLA status. Ships a dependency-free stats library + offline tests. Works for any streaming or micro-batch pipeline.

📁 16 files🏷 v1.0.0

Production-readyUnit-testedDatabricks Asset Bundle

✓ Instant download✓ Lifetime updates✓ MIT licensed✓ Secure checkout (Stripe)

⚙ Try the Live Demo interactive

Generate latency samples and see p50/p95/p99 and SLA status update live — the framework's percentile logic in your browser.

⚡ Open Latency Explorer

📋 What's Inside 16 files

✓ README.md
✓ LICENSE
✓ manifest.json
✓ databricks.yml
✓ resources/jobs.yml
✓ src/01_batch_latency.sql
✓ src/02_e2e_latency.sql
✓ src/03_latency_summary.sql
✓ lib/latency_stats.py
✓ tests/test_latency_stats.py
✓ conftest.py
✓ guide/01_what-you-get.md
✓ guide/02_getting-started.md
✓ guide/03_architecture.md
✓ guide/04_support.md
✓ guides/latency-methodology.md

📁 File Structure 16 files

ingestion-latency-framework/
├── README.md
├── LICENSE
├── manifest.json
├── databricks.yml
├── resources/
│   ├── jobs.yml
├── src/
│   ├── 01_batch_latency.sql
│   ├── 02_e2e_latency.sql
│   ├── 03_latency_summary.sql
├── lib/
│   ├── latency_stats.py
├── tests/
│   ├── test_latency_stats.py
├── conftest.py
├── guide/
│   ├── 01_what-you-get.md
│   ├── 02_getting-started.md
│   ├── 03_architecture.md
│   ├── 04_support.md
├── guides/
│   ├── latency-methodology.md

📖 Documentation Preview README excerpt

Ingestion Latency Monitoring Framework

Measures **end-to-end data ingestion latency** across medallion layers:

batch-to-batch lag ("how often is new data arriving?") and source-to-landing

time ("how fresh is it vs the source?"), with p50/p95/p99 summaries and SLA

status. Works for any streaming or micro-batch pipeline.

Fully generic and environment-agnostic — point it at your bronze/silver

catalogs and monitored tables.

What's inside

- **Databricks Asset Bundle** — a 2-hourly measure + summarize job.

- **Two latency measures** — batch-to-batch lag and end-to-end (source→landing).

- **Percentile summaries** — p50/p95/p99 per table/layer/day.

- **A dependency-free stats library** (`lib/latency_stats.py`): batch lags,

end-to-end latency, percentiles, summaries, and SLA status — with an offline

test suite (5 tests).

Quickstart

```bash

pip install pytest && pytest tests/ -v

databricks bundle deploy -t dev -p <profile>

```

Set the target/bronze/silver catalogs and edit the table references in the SQL

templates to your monitored tables.

License

MIT — see `LICENSE`.

... preview truncated, see full README in product download.

📄 Code Sample .sql preview

src/01_batch_latency.sql-- Ingestion Latency: Batch-to-batch lag — time between consecutive landings. -- Template for one monitored table; duplicate/UNION ALL per table you monitor. -- Requires an ETL load timestamp column (_etl_load_timestamp_utc). CREATE OR REPLACE TABLE IDENTIFIER(:target_catalog || '.' || :target_schema || '.f_batch_latency') AS WITH batches AS ( SELECT DISTINCT _etl_load_timestamp_utc FROM IDENTIFIER(:bronze_catalog || '.<schema>.<table>') -- set your table WHERE _etl_load_timestamp_utc >= DATEADD(DAY, -14, CURRENT_TIMESTAMP()) ), lagged AS ( SELECT _etl_load_timestamp_utc, LAG(_etl_load_timestamp_utc) OVER (ORDER BY _etl_load_timestamp_utc) AS previous_load_ts FROM batches ) SELECT '<schema>.<table>' AS monitored_table, 'bronze' AS layer, _etl_load_timestamp_utc AS batch_timestamp, (UNIX_MILLIS(_etl_load_timestamp_utc) - UNIX_MILLIS(previous_load_ts)) AS batch_lag_ms FROM lagged WHERE previous_load_ts IS NOT NULL;

Buy Now — $990 ⚙ Open Latency Explorer Back to Products