← Back to all products

Data Observability Setup

$29

Observability framework for Databricks with lineage tracking, anomaly detection, and SLA monitoring.

📁 17 files🏷 v1.0.0

Production-ready

✓ Instant download✓ Lifetime updates✓ MIT licensed✓ MIT license✓ Secure checkout (Stripe)

📋 What's Inside 17 files

✓ README.md—
✓ manifest.json—
✓ LICENSE—
✓ src/lineage_tracker.py—
✓ src/metric_collector.py—
✓ src/anomaly_detector.py—
✓ src/freshness_monitor.py—
✓ src/alert_manager.py—
✓ src/dashboard_data.py—
✓ configs/observability_config.yaml—
✓ configs/alert_rules.yaml—
✓ notebooks/observability_dashboard.py—
✓ notebooks/setup_observability.py—
✓ tests/conftest.py—
✓ tests/test_anomaly_detector.py—

📁 File Structure 17 files

data-observability-setup/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│ ├── lineage_tracker.py
│ ├── metric_collector.py
│ ├── anomaly_detector.py
│ ├── freshness_monitor.py
│ ├── alert_manager.py
│ ├── dashboard_data.py
├── configs/
│ ├── observability_config.yaml
│ ├── alert_rules.yaml
├── notebooks/
│ ├── observability_dashboard.py
│ ├── setup_observability.py
├── tests/
│ ├── conftest.py
│ ├── test_anomaly_detector.py
│ ├── test_freshness_monitor.py
├── guides/
│ ├── data-observability-guide.md

📖 Documentation Preview README excerpt

Data Observability Setup

Complete observability framework for Databricks data pipelines — lineage tracking, anomaly detection, SLA monitoring, and alerting.

> By [Datanest Digital](https://datanest.dev) | Version 1.0.0 | $49

---

What You Get

**Data Lineage Tracking** — Trace every record from source through transformations to target, stored in Delta

**Metric Collection** — Automated capture of row counts, durations, data volumes, and error rates

**Statistical Anomaly Detection** — Z-score, IQR, and moving average methods to catch pipeline drift

**Freshness Monitoring** — SLA-aware data freshness checks with breach detection

**Alert Routing** — Multi-channel alerting via Slack, PagerDuty, and email with severity-based routing

**Dashboard-Ready Data** — Pre-aggregated health scores and metrics for observability dashboards

File Tree


data-observability-setup/
├── README.md
├── manifest.json
├── LICENSE
├── src/
│   ├── lineage_tracker.py          # Source→transform→target lineage
│   ├── metric_collector.py         # Pipeline metric collection
│   ├── anomaly_detector.py         # Statistical anomaly detection
│   ├── freshness_monitor.py        # Data freshness & SLA monitoring
│   ├── alert_manager.py            # Multi-channel alert routing
│   └── dashboard_data.py           # Dashboard aggregation & health scores
├── configs/
│   ├── observability_config.yaml   # Main configuration
│   └── alert_rules.yaml            # Alert rule definitions
├── notebooks/
│   ├── observability_dashboard.py  # Health & lineage dashboard
│   └── setup_observability.py      # Initialize observability tables
├── tests/
│   ├── conftest.py                 # Shared fixtures
│   ├── test_anomaly_detector.py    # Anomaly detection tests
│   └── test_freshness_monitor.py   # SLA breach tests
└── guides/
    └── data-observability-guide.md # Observability strategy guide


Getting Started

1. Initialize Observability Tables

Run the setup notebook in your Databricks workspace to create the required Delta tables:


In Databricks — run notebooks/setup_observability.py
Creates: observability.lineage, observability.metrics,
         observability.alerts, observability.anomalies


2. Track Lineage in Your Pipelines


... preview truncated, see full README in product download.

`📄 Code Sample .py preview`

src/lineage_tracker.py"""
Lineage Tracker — Track data lineage from source through transformations to target.

Captures source→transformation→target relationships and persists them in a Delta
table for auditing, impact analysis, and regulatory compliance.

Author: Datanest Digital
"""

from __future__ import annotations

import logging
import uuid
from dataclasses import dataclass, field
from datetime import datetime
from enum import Enum
from typing import Any, Dict, List, Optional

from pyspark.sql import DataFrame, SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import (
    ArrayType,
    StringType,
    StructField,
    StructType,
    TimestampType,
)

logger = logging.getLogger(__name__)


# ---------------------------------------------------------------------------
# Data models
# ---------------------------------------------------------------------------

class NodeType(str, Enum):
    """Types of nodes in the lineage graph."""

    SOURCE = "source"
    TRANSFORMATION = "transformation"

`📅 Changelog`

v1.0.0 — Initial release.

Purchases include lifetime updates. Check the product page for the latest version.

`📄 Product Preview`

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview)⚙ Try Demo Tools📦 Download Free Sample

`❓ Frequently Asked Questions`

What license is this under?

MIT License. You can use it in commercial, client, and internal projects. See our license terms page for details.

How do I download after purchase?

Immediately after purchase, you'll be redirected to a download page. A download link is also sent to your email.

Do I get updates?

Yes. All products include lifetime updates at no extra cost. Re-download anytime for the latest version.

What if it doesn't work for me?

DM @DatanestDigital on X and we'll make it right.

Can I get a refund?

DM @DatanestDigital on X with your order number and we'll help.

Is there support?

DM @DatanestDigital on X. We respond within 1-2 business days.



      
        Buy Now — $29
        
        Back to Products