Overview

This chapter covers the core features and capabilities of Compliance Automation Suite.

Overview

The Compliance Automation Suite provides production-ready tooling for organizations operating

on Databricks and Azure that must satisfy GDPR, SOC 2 Type II, ISO 27001, and the EU NIS2

Directive. Instead of manually assembling evidence and cross-referencing spreadsheets before

every audit cycle, this suite automates the heavy lifting: continuous control verification,

evidence collection, gap analysis, and audit-readiness reporting.

New in v2.0: The suite now includes a comprehensive operational audit module with 9

automated audit scripts covering catalog inventory, job analysis, secret management, compute

costs, storage security, token hygiene, query history, data lineage, and table permissions.

These operational audits use Python standard library only (no pip dependencies) and

produce HTML and JSON reports with findings mapped to compliance frameworks (GDPR, SOC 2,

ISO 27001, NIS2).

Every script is designed as a Databricks notebook or standalone Python module so it slots

directly into existing lakehouse architectures. The control mappings translate regulatory

language into concrete technical controls you can verify programmatically.

What's Included

Control Mappings (mappings/)

Compliance Audit Scripts (scripts/)

Operational Audit Module (audits/) — NEW in v2.0

A self-contained Python package with 9 automated audit scripts, a shared API client,

configuration module, and utility library. Zero pip dependencies — uses only Python

standard library (http.client, ssl, json, csv, logging, etc.).

Each audit produces structured Finding objects with:

• Severity levels (CRITICAL / HIGH / MEDIUM / LOW / INFO)
• Risk scores for prioritization
• Compliance framework references (e.g., "GDPR Art. 32", "SOC2 CC6.1", "ISO27001 A.8.2")
• Actionable remediation recommendations

Report Generators (reports/)

GDPR Toolkit (gdpr/)

Templates (templates/)

Dashboards (dashboards/)

Orchestration

Prerequisites

For Compliance Scripts (scripts/, gdpr/, dashboards/)

• Databricks Workspace (Premium or Enterprise tier recommended)
• Azure Subscription with Contributor or Reader access for resource enumeration
• Python 3.9+ with the following packages:
• databricks-sdk >= 0.20.0
• azure-identity >= 1.15.0
• azure-mgmt-resource >= 23.0.0
• pyspark (available by default in Databricks Runtime)
• cryptography >= 41.0.0
• requests >= 2.31.0
• Unity Catalog enabled (for fine-grained access auditing)
• Audit Log Delivery configured to a Delta table or storage account

For Operational Audits (audits/, reports/, run_audit.sh)

• Python 3.10+ (standard library only — no pip install required)
• Databricks workspace with a valid API token
• Unity Catalog enabled (recommended for catalog/lineage/permissions audits)
• Environment variables: DATABRICKS_HOST, DATABRICKS_TOKEN

Quick Start

Follow this guide to get Compliance Automation Suite up and running in your environment.

Quick Start

Running Operational Audits (New in v2.0)

The fastest way to get started is with the orchestrator script:

# Set required environment variables
export DATABRICKS_HOST='https://your-workspace.cloud.databricks.com'
export DATABRICKS_TOKEN='dapi_YOUR_TOKEN_HERE'
export AUDIT_COMPANY_NAME='Acme Corp'

# Run all 9 operational audits with both HTML and JSON output
chmod +x run_audit.sh
./run_audit.sh --all --format both

To run a specific audit:

# Run only the token inventory audit
./run_audit.sh --audit token_inventory --format json

# Run only compute analysis
./run_audit.sh --audit compute_analysis --format html

#### Available audit IDs for --audit:

#### Running from Python directly:

import os
import sys

# Ensure the suite root is on the path
sys.path.insert(0, "/path/to/compliance-automation-suite")

from audits.utils import DatabricksApiClient
from audits import ALL_AUDITS, AUDIT_REGISTRY

# Initialize the API client (reads DATABRICKS_HOST and DATABRICKS_TOKEN from env)
client = DatabricksApiClient()

# Run all audits
for audit_cls in ALL_AUDITS:
    audit = audit_cls(client=client)
    result = audit.execute()
    print(f"{audit.audit_name}: {len(result.get('findings', []))} findings")

# Or run a specific audit
TokenInventory = AUDIT_REGISTRY["token_inventory"]
audit = TokenInventory(client=client)
result = audit.execute()

#### Generating reports programmatically:

from reports.html_report import HtmlReportGenerator
from reports.json_report import JsonReportGenerator

# After running audits, generate reports
html_gen = HtmlReportGenerator(output_dir="./output")
html_gen.generate(audit_results=results, company_name="Acme Corp")

json_gen = JsonReportGenerator(output_dir="./output")
json_gen.generate(audit_results=results, company_name="Acme Corp")

Running Compliance Scripts

# In a Databricks notebook
%run ./scripts/access_control_audit

auditor = AccessControlAuditor(workspace_url="https://adb-xxxx.azuredatabricks.net")
report = auditor.run_full_audit()
report.to_delta("compliance.access_control_audit_results")

Automate with Workflows

Schedule each audit script as a Databricks Workflow job on a cadence that matches your

compliance requirements:

Compliance scripts:

• Daily: access_control_audit.py, audit_log_validation.py
• Weekly: encryption_verification.py, network_security_check.py
• Monthly: data_retention_check.py

Operational audits:

• Daily: token_inventory, compute_analysis
• Weekly: catalog_inventory, job_inventory, secret_inventory, external_storage, query_history, table_permissions
• Monthly: data_lineage

See templates/scheduling_guide.md for detailed cron, ADF, Databricks Workflows, and

GitHub Actions scheduling examples.

Build the Dashboard

Execute dashboards/compliance_dashboard.sql in Databricks SQL to create a real-time

compliance posture dashboard that aggregates findings from all audit scripts.

Handle DSRs

from gdpr.dsr_handler import DSRHandler

handler = DSRHandler(catalog="main", schemas=["production", "analytics"])
results = handler.handle_access_request(subject_id="user@example.com")
handler.export_portable_copy(results, format="json")

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                    COMPLIANCE AUTOMATION SUITE v2.0                      │
├──────────────────────────┬──────────────────────────────────────────────┤
│  Compliance Layer        │  Operational Audit Layer (NEW)               │
│                          │                                              │
│  scripts/                │  audits/                                     │
│  ├─ access_control       │  ├─ catalog_inventory                       │
│  ├─ encryption           │  ├─ job_inventory                           │
│  ├─ data_retention       │  ├─ secret_inventory                        │
│  ├─ audit_log            │  ├─ compute_analysis                        │
│  └─ network_security     │  ├─ external_storage                        │
│                          │  ├─ token_inventory                         │
│  gdpr/                   │  ├─ query_history                           │
│  ├─ dsr_handler          │  ├─ data_lineage                            │
│  ├─ consent_tracking     │  └─ table_permissions                       │
│  ├─ dpia_template        │                                              │
│  └─ ropa_generator       │  reports/                                    │
│                          │  ├─ html_report.py                           │
│  mappings/               │  ├─ json_report.py                           │
│  ├─ gdpr_controls        │  └─ templates/audit_report.html              │
│  ├─ soc2_controls        │                                              │
│  ├─ iso27001_controls    │  run_audit.sh (orchestrator)                 │
│  └─ nis2_checklist       │                                              │
├──────────────────────────┴──────────────────────────────────────────────┤
│  Shared Infrastructure                                                   │
│  ├─ Control Mappings → Framework references on every finding             │
│  ├─ Delta Lake Tables → Persistent findings store                        │
│  ├─ SQL Dashboard → Real-time compliance posture                         │
│  └─ Evidence Binder → Audit-ready documentation                          │
└─────────────────────────────────────────────────────────────────────────┘

How the layers work together

1. Operational audits discover your actual infrastructure state (what exists, how it's

configured, what's at risk)

2. Compliance scripts verify that infrastructure meets regulatory requirements (is

encryption enabled? are logs retained long enough?)

3. Control mappings tie both layers back to specific regulatory articles and controls

4. Reports present findings with compliance framework references for auditor consumption

5. Dashboard provides a real-time compliance posture view

Configuration

Operational Audit Configuration

The operational audit module is configured via environment variables. All settings have

sensible defaults. See audits/config.py for the complete reference.

Compliance Script Configuration

All compliance scripts accept configuration via environment variables or a

compliance_config.json file placed in the workspace root:

{
  "workspace_url": "https://adb-xxxx.azuredatabricks.net",
  "catalog": "main",
  "findings_schema": "compliance",
  "retention_policy_table": "compliance.retention_policies",
  "alert_webhook_url": "https://hooks.example.com/services/YOUR_WEBHOOK_HERE",
  "max_token_lifetime_days": 90,
  "required_encryption_algorithm": "AES-256",
  "audit_log_retention_days": 365
}