Overview

This chapter covers the core features and capabilities of Databricks Monitoring & Alerting Suite.

Overview

A comprehensive, production-ready monitoring and alerting toolkit for Databricks workspaces. This suite provides 8 SQL dashboards, configurable alert definitions, 50+ pre-built system table queries, webhook integrations, report templates, and operational runbooks -- everything you need to achieve full observability over your Databricks environment.

What's Included

Dashboards (8)

Alerts

• Alert Definitions -- SQL-based alert rules for job failures, cost spikes, idle clusters, SLA breaches, and more
• Webhook Templates -- Ready-to-use payload templates for Slack, Microsoft Teams, PagerDuty, and email

Queries

• System Table Library -- 50+ pre-built queries against system.billing, system.access, system.compute, and related tables covering billing analysis, access auditing, compute profiling, and operational diagnostics

Templates & Runbooks

• Monthly Health Report -- Markdown template with embedded query references for generating executive-level monthly reports
• Common Operations Runbook -- Step-by-step procedures for incident response, cost optimization, capacity management, and access review

Prerequisites

Complete these checks before importing dashboards or creating alerts. The suite reads Databricks system tables; it does not create monitoring data, repair account configuration, or grant its own access.

Workspace Capabilities

Use a Databricks workspace attached to a Unity Catalog metastore. Confirm the required system schemas are enabled and populated for the account and region you intend to monitor. The supplied SQL primarily references system.billing, system.access, system.compute, system.lakeflow, system.query, and system.information_schema. Availability and retention vary by system table, so test the queries against your workspace before promising a reporting period.

Provision a running Databricks SQL warehouse. Serverless or Pro is recommended for scheduled dashboards and alerts; choose a size and auto-stop policy that matches query volume. Enable Databricks SQL dashboards, alerts, and notification destinations in the target workspace.

Identity and Permissions

For interactive setup, use a named administrator or monitoring engineer. For scheduled automation, create a dedicated service principal rather than using a personal identity. Grant only:

• CAN USE on the SQL warehouse.
• USE CATALOG and USE SCHEMA on the required system catalog objects.
• SELECT on the system tables used by imported queries.
• Permission to create or manage queries, dashboards, alerts, and notification destinations where applicable.

An account administrator may need to enable system schemas or grant the initial access. Validate grants with the service principal itself, not only with an administrator.

Prefer OAuth machine-to-machine authentication for production automation. A personal access token (PAT) is acceptable for a short-lived local setup if organizational policy permits it. Never place client secrets or PATs in SQL, JSON templates, source control, screenshots, or the product directory. Store them in a secret manager and rotate them according to policy.

Local Python Environment

Python is optional for manual SQL import, but useful for connection testing and future automation. Use Python 3.10 or newer in an isolated environment:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install databricks-sql-connector databricks-sdk

Keep local dependency pins in your own automation repository; the suite itself remains SQL-first.

Connection Configuration

Copy the Server Hostname and HTTP Path from the SQL warehouse connection details. For a temporary PAT-based shell session, set environment variables without committing them:

export DATABRICKS_SERVER_HOSTNAME="adb-example.azuredatabricks.net"
export DATABRICKS_HTTP_PATH="/sql/1.0/warehouses/replace-me"
export DATABRICKS_TOKEN="<retrieve-from-secret-manager>"

The Databricks SDK and CLI can instead use a named profile in ~/.databrickscfg. In CI, inject equivalent host and OAuth client settings from the platform’s secret store. Ensure outbound HTTPS access to the workspace and any Slack, Teams, PagerDuty, or email notification endpoint you plan to configure. Replace placeholders in alerts/webhook_templates.json only in a protected deployment copy.

Verification Checklist

Before continuing, verify:

• [ ] The workspace is attached to Unity Catalog.
• [ ] A test query such as SELECT * FROM system.billing.usage LIMIT 1 succeeds.
• [ ] Queries against system.lakeflow.job_run_timeline and system.query.history succeed or are intentionally excluded.
• [ ] The monitoring identity can use the chosen SQL warehouse.
• [ ] The identity can create and share a test query, dashboard, and alert.
• [ ] Scheduled queries can start the warehouse and complete within the expected timeout.
• [ ] A test notification reaches a non-production destination.
• [ ] No credentials or real webhook URLs appear in tracked files.

Resolve failed checks before importing the full library; otherwise dashboards may look healthy simply because their source tables are inaccessible or empty.