Databricks Audit Toolkit

Databricks Audit Toolkit

A comprehensive, production-ready audit and inventory toolkit for Databricks

workspaces. Runs 12 independent audit modules covering security, governance,

compute costs, access control, and data lineage — then generates professional

HTML and JSON reports with risk scores and actionable recommendations.

Built by practitioners who manage large-scale Databricks environments daily.

Table of Contents

• Features
• Quick Start
• Prerequisites
• Installation
• Configuration
• Usage
• Run All Audits
• Run Specific Audits
• Output Formats
• List Available Audits
• Audit Modules
• 01 — Catalog Inventory
• 02 — Job Inventory
• 03 — Access Audit
• 04 — Secret Inventory
• 05 — Compute Analysis
• 06 — Security Findings
• 07 — Audit Logs
• 08 — External Storage
• 09 — Token Inventory
• 10 — Query History
• 11 — Data Lineage
• 12 — Table Permissions
• Reports
• HTML Report
• JSON Report
• Risk Scoring
• Architecture
• Project Structure
• BaseAudit Class
• API Client
• Finding Severity Levels
• Customization
• Adding a Custom Audit
• Adjusting Risk Thresholds
• Custom Report Templates
• Scheduling Automated Audits
• Security Considerations
• Troubleshooting
• License

Features

• 12 audit modules covering Unity Catalog, access control, secrets, compute,

tokens, query history, data lineage, external storage, and more

• Professional HTML reports with risk scores, severity distribution,

category breakdown, and actionable recommendations

• Machine-readable JSON output for SIEM ingestion, dashboards, and trend

analysis

• Risk scoring engine that aggregates findings into a 0-100 risk score

with CRITICAL/HIGH/MEDIUM/LOW ratings

• Standalone modules — run all 12 audits or pick specific ones
• Rate limiting and retry logic — respects Databricks API limits with

exponential backoff

• Paginated API support — handles workspaces with thousands of objects
• Zero hardcoded credentials — all authentication via environment variables
• Extensible architecture — add your own audit modules by inheriting from

BaseAudit

Quick Start

# 1. Clone or unzip the toolkit
cd databricks-audit-toolkit

# 2. Install Python dependencies
pip install -r requirements.txt

# 3. Set your credentials
export DATABRICKS_HOST='https://your-workspace.cloud.databricks.com'
export DATABRICKS_TOKEN='dapi...'

# 4. Run all audits
./run_audit.sh --all

# 5. Open the report
open output/audit_report.html

Prerequisites

Recommended Databricks Permissions

For the most comprehensive audit, the token should belong to a **workspace

admin** or a service principal with:

• Unity Catalog metastore admin or SELECT / USE grants
• Workspace admin group membership (for SCIM, token management, audit logs)
• CAN_MANAGE on clusters (for compute analysis)

If running with limited permissions, individual audit modules will gracefully

skip API calls they cannot access and report the limitation in their output.

Installation

Follow this guide to get databricks audit toolkit up and running in your environment.

Installation

# Option 1: pip install dependencies
pip install -r requirements.txt

# Option 2: pip install in a virtual environment
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Dependencies

Configuration

All configuration is via environment variables. No config files to manage,

no credentials in source code.

Required

Optional

Example .env file

# .env (source this before running)
export DATABRICKS_HOST='https://your-workspace.cloud.databricks.com'
export DATABRICKS_TOKEN='dapi...'
export DATABRICKS_WAREHOUSE_ID='abc123def456'
export AUDIT_OUTPUT_DIR='./output'
export AUDIT_LOG_LEVEL='INFO'
export AUDIT_COMPANY_NAME='Acme Corp'

source .env
./run_audit.sh --all

Usage

Run All Audits

./run_audit.sh --all

This runs all 12 audit modules sequentially and generates both HTML and JSON

reports in the output directory.

Run Specific Audits

# Run only catalog inventory and access audit
./run_audit.sh --audit 01 03

# Run compute analysis and token inventory
./run_audit.sh --audit 05 09

# Run security-focused audits
./run_audit.sh --audit 03 04 06 09 12

Output Formats

# HTML report only
./run_audit.sh --all --format html

# JSON report only
./run_audit.sh --all --format json

# Both formats (default)
./run_audit.sh --all --format both

Custom Output Directory

./run_audit.sh --all --output-dir /tmp/audit-$(date +%Y-%m-%d)

List Available Audits

./run_audit.sh --list

Output:

  ID    Module                          Description
  ---   ------------------------------  --------------------
  01    a01_catalog_inventory           Catalog Inventory
  02    a02_job_inventory               Job Inventory
  03    a03_access_audit                Access Audit
  04    a04_secret_inventory            Secret Inventory
  05    a05_compute_analysis            Compute Analysis
  06    a06_security_findings           Security Findings
  07    a07_audit_logs                  Audit Logs
  08    a08_external_storage            External Storage
  09    a09_token_inventory             Token Inventory
  10    a10_query_history               Query History
  11    a11_data_lineage                Data Lineage
  12    a12_table_permissions           Table Permissions

Audit Modules

01 — Catalog Inventory

Inventories all Unity Catalog objects: catalogs, schemas, tables, and views.

Identifies sensitive schemas, production catalogs, and governance gaps.

API endpoints: /api/2.1/unity-catalog/catalogs, /schemas, /tables

Findings: Schemas with sensitive keywords (PII, HIPAA, credentials),

untagged production catalogs, large catalog sprawl.

02 — Job Inventory

Comprehensive inventory of all Databricks jobs with run history analysis.

Identifies failing jobs, missing alerts, and scheduling issues.

API endpoints: /api/2.1/jobs/list, /api/2.1/jobs/runs/list

Findings: Jobs without failure notifications, high failure rates,

long-running jobs, paused jobs that may be abandoned.

03 — Access Audit

Audits workspace users, groups, service principals, and admin privileges.

Identifies over-provisioned access and inactive accounts.

API endpoints: /api/2.0/preview/scim/v2/Users, /Groups,

/ServicePrincipals

Findings: Excessive admin users, inactive accounts, users not in groups,

service principals with admin privileges.

04 — Secret Inventory

Inventories all Databricks secret scopes, secrets, and access control lists.

Identifies overly permissive secret access.

API endpoints: /api/2.0/secrets/scopes/list, /secrets/list,

/secrets/acls/list

Findings: Scopes granting MANAGE to all users, scopes without explicit

ACLs, secrets with broad read access.

05 — Compute Analysis

Analyzes all clusters and instance pools for cost optimization and security.

Estimates monthly compute costs.

API endpoints: /api/2.0/clusters/list, /instance-pools/list

Findings: Clusters without auto-termination, missing autoscaling, expensive

GPU instances, underutilized pools, estimated cost overruns.

06 — Security Findings

Meta-audit that aggregates findings from all other modules into a unified

security posture assessment. Calculates the overall risk score.

API endpoints: None (aggregates results from other audits)

Output: Risk score (0-100), risk rating, findings by severity and category,

prioritized recommendations.

07 — Audit Logs

Analyzes workspace audit log events for suspicious activity patterns. Detects

failed logins, permission changes, and unusual cluster creation.

API endpoints: Databricks system tables or audit log delivery

Findings: Failed authentication attempts, bulk permission changes,

after-hours activity, unusual cluster creation patterns.

08 — External Storage

Inventories Unity Catalog external locations and storage credentials. Identifies

overly broad storage access and credential types.

API endpoints: /api/2.1/unity-catalog/external-locations,

/storage-credentials

Findings: External locations accessible to all users, service principal

credentials (vs. managed identity), overlapping storage paths.

09 — Token Inventory

Inventories all personal access tokens using the Token Management API. Identifies

non-expiring tokens, old tokens, and admin-level tokens.

API endpoints: /api/2.0/token-management/tokens

Findings: Tokens without expiry dates, tokens older than 90 days,

admin-level tokens, excessive tokens per user.

10 — Query History

Analyzes SQL query history for performance patterns. Identifies slow queries,

heavy warehouse usage, and potential optimizations.

API endpoints: /api/2.0/sql/history/queries

Findings: Queries exceeding execution thresholds, heavily used tables

without optimization, warehouse capacity concerns.

11 — Data Lineage

Maps upstream and downstream lineage for Unity Catalog tables. Identifies

deep dependency chains and tables without lineage.

API endpoints: /api/2.1/unity-catalog/lineage/table-lineage,

/column-lineage

Findings: Deep dependency chains (6+ levels), tables with no upstream

lineage (potential data sources), high-impact tables (many downstream

consumers).

12 — Table Permissions

Audits table-level permissions and grants across Unity Catalog. Identifies

overly broad access and orphaned grants.

API endpoints: /api/2.1/unity-catalog/permissions/{securableType}/{fullName}

Findings: Tables granting access to all workspace users, SELECT * grants on

sensitive tables, orphaned grants for deleted principals.