Unity Catalog Governance Pack

Unity Catalog Governance Pack

Production-ready governance templates for Databricks Unity Catalog.

Implement enterprise-grade data governance in hours, not months.




Table of Contents

1. Overview

2. Architecture

3. Quick Start

4. Module Reference

• RBAC (Role-Based Access Control)
• Data Classification
• Compliance (GDPR / CCPA)
• Security Patterns
• Automation
• Terraform Deployment

5. Configuration

6. Integration Patterns

7. Deployment Guide

8. Monitoring and Alerting

9. Troubleshooting

10. FAQ

11. Contributing

12. License

Overview

The Unity Catalog Governance Pack provides a complete, ready-to-deploy governance

framework for Databricks Unity Catalog environments. It includes:

• Role-Based Access Control (RBAC): A five-tier role hierarchy with SQL and Python

automation for setting up and auditing permissions across catalogs, schemas, and tables.

• Data Classification: A four-tier classification framework (Public, Internal,

Confidential, Restricted) with automated PII scanning, tagging, and coverage reporting.

• Compliance Templates: Production-ready implementations for GDPR (right to erasure,

data portability, consent tracking, subject access requests) and CCPA (opt-out tracking,

data disclosure) with detailed compliance checklists including SOC 2 Type II.

• Security Patterns: Row-level security, column masking, network isolation guides,

and encryption patterns for defense-in-depth.

• Automation: Tag propagation, governance violation scanning, policy enforcement,

and dashboard data generation for continuous governance monitoring.

• Terraform Module: Infrastructure-as-Code deployment of the entire governance

framework using the Databricks Terraform provider.

Who Is This For?

• Data Platform Engineers setting up governance from scratch
• Security/Compliance Teams implementing regulatory requirements
• Data Architects designing multi-team data mesh architectures
• DevOps Engineers automating governance with Terraform

What Problems Does It Solve?

Architecture

+------------------------------------------------------------------+
|                    Unity Catalog Metastore                         |
|                                                                    |
|  +------------------+  +------------------+  +------------------+ |
|  |  Catalog: prod   |  |  Catalog: dev    |  |  Catalog: staging| |
|  |  +------------+  |  |  +------------+  |  |  +------------+  | |
|  |  | Schema:    |  |  |  | Schema:    |  |  |  | Schema:    |  | |
|  |  | customers  |  |  |  | customers  |  |  |  | customers  |  | |
|  |  +------------+  |  |  +------------+  |  |  +------------+  | |
|  |  | Schema:    |  |  |  | Schema:    |  |  |  | Schema:    |  | |
|  |  | analytics  |  |  |  | analytics  |  |  |  | analytics  |  | |
|  |  +------------+  |  |  +------------+  |  |  +------------+  | |
|  +------------------+  +------------------+  +------------------+ |
+------------------------------------------------------------------+
         |                       |                       |
         v                       v                       v
+------------------------------------------------------------------+
|                     Governance Layer                               |
|                                                                    |
|  +-----------+  +----------------+  +-------------+  +----------+ |
|  |   RBAC    |  | Classification |  | Compliance  |  | Security | |
|  |  Roles &  |  | Tags & PII    |  | GDPR, CCPA  |  | RLS, Col | |
|  |  Groups   |  | Scanning      |  | Automation   |  | Masking  | |
|  +-----------+  +----------------+  +-------------+  +----------+ |
|                                                                    |
|  +-----------+  +----------------+  +---------------------------+ |
|  | Automation|  |   Terraform    |  |     Monitoring &          | |
|  | Scanner & |  |   IaC Module   |  |     Dashboards            | |
|  | Enforcer  |  |                |  |                           | |
|  +-----------+  +----------------+  +---------------------------+ |
+------------------------------------------------------------------+

Governance Data Flow

1. DEPLOY       -> Terraform provisions UC resources, groups, grants
2. CLASSIFY     -> PII Scanner identifies sensitive data, applies tags
3. SECURE       -> Row-level security + column masking applied
4. COMPLY       -> GDPR/CCPA handlers registered for data subject requests
5. MONITOR      -> Governance scanner runs on schedule, generates reports
6. ENFORCE      -> Policy enforcer blocks non-compliant operations
7. AUDIT        -> Permission auditor validates RBAC matrix continuously

Quick Start

Prerequisites

• Databricks workspace with Unity Catalog enabled
• Account admin or metastore admin privileges
• Python 3.9+ with databricks-sdk installed
• Terraform 1.5+ (for IaC deployment)

1. Install Dependencies

pip install databricks-sdk pyyaml pandas

2. Configure Environment

export DATABRICKS_HOST="https://your-workspace.cloud.databricks.com"
export DATABRICKS_TOKEN="your-personal-access-token"

3. Set Up RBAC

# Review and customize the role definitions
vi rbac/role_definitions.yaml

# Apply roles via SQL
databricks workspace import rbac/setup_roles.sql /Shared/governance/setup_roles

# Or use the Python automation
python rbac/setup_roles.py --catalog prod --dry-run
python rbac/setup_roles.py --catalog prod

4. Run PII Scanner

# Scan a specific catalog
python classification/pii_scanner.py --catalog prod --schema customers

# Scan everything
python classification/pii_scanner.py --catalog prod --all-schemas

5. Apply Classification Tags

-- Run in Databricks SQL
-- See classification/apply_tags.sql for full examples
ALTER TABLE prod.customers.profiles
  SET TAGS ('classification' = 'confidential', 'contains_pii' = 'true');

6. Deploy with Terraform

cd terraform/
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your values
terraform init
terraform plan
terraform apply

Module Reference

Follow this guide to get unity catalog governance pack up and running in your environment.

Module Reference

RBAC Module

Location: rbac/

The RBAC module implements a five-tier role hierarchy designed for enterprise

data platforms:

Role Inheritance: Each role inherits all permissions of the roles below it

in the hierarchy. Admins inherit engineer permissions, engineers inherit analyst

permissions, and so on.

Files:

• role_definitions.yaml - Complete role hierarchy with descriptions and member patterns
• permission_matrix.yaml - Granular permissions per role for every securable object type
• setup_roles.sql - Databricks SQL statements to create groups and apply grants
• setup_roles.py - Python SDK automation with dry-run support and rollback
• audit_permissions.py - Validates current state matches the expected permission matrix

Usage:

from rbac.setup_roles import RBACManager

manager = RBACManager(catalog="prod")
manager.setup_all_roles(dry_run=True)   # Preview changes
manager.setup_all_roles(dry_run=False)  # Apply changes

Classification Module

Location: classification/

Implements a four-tier data classification framework aligned with common

enterprise standards:

PII Detection Capabilities:

The pii_scanner.py detects the following PII types using a combination of

regex pattern matching and column name heuristics:

• Email addresses
• Social Security Numbers (SSN)
• Phone numbers (US/international)
• Credit card numbers (Visa, MasterCard, Amex, Discover)
• IP addresses (IPv4)
• Dates of birth
• Person names (via column name heuristics)

Files:

• classification_framework.yaml - Tier definitions, handling requirements, retention policies
• apply_tags.sql - SQL templates for applying classification tags
• pii_scanner.py - Automated PII detection engine
• pii_patterns.yaml - Configurable regex patterns and column name heuristics
• classification_report.py - Coverage report generator

Compliance Module

Location: compliance/

Production-ready compliance implementations for GDPR and CCPA, with audit

checklists for SOC 2 Type II.

#### GDPR (General Data Protection Regulation)

• right_to_deletion.py - Implements Article 17 (right to erasure) with Delta Lake

DELETE operations across all tables containing user data, with full audit logging

• data_portability.py - Implements Article 20 (data portability) by exporting all

user data in JSON and CSV formats

• consent_tracking.sql - Schema and queries for managing user consent records
• data_subject_request.py - Unified handler for all data subject access requests

#### CCPA (California Consumer Privacy Act)

• do_not_sell.py - Manages opt-out flags and ensures data is excluded from

third-party sharing pipelines

• data_disclosure.py - Generates comprehensive data disclosure reports showing

all data collected about a consumer

#### Compliance Checklists

• gdpr_checklist.md - 40+ item GDPR compliance checklist
• ccpa_checklist.md - 30+ item CCPA compliance checklist
• soc2_checklist.md - SOC 2 Type II controls mapping to Unity Catalog features

Security Module

Location: security/

Defense-in-depth security patterns for Unity Catalog:

• Row-Level Security (row_level_security.sql): Dynamic views that filter

rows based on the current user's group membership, supporting multi-tenant

data isolation patterns.

• Column Masking (column_masking.sql): Three masking strategies:
• Full hash (SHA-256)
• Redaction (replace with ***)
• Partial masking (show last 4 digits)

• Network Security (network_security.md): Guide for private endpoints,

IP access lists, VNet injection, and network peering.

• Encryption (encryption_guide.md): Patterns for encryption at rest

(customer-managed keys) and in transit (TLS, workspace encryption).

Automation Module

Location: automation/

Continuous governance monitoring and enforcement:

• tag_propagation.py - When a source table is tagged, automatically propagate

tags to all downstream tables discovered via Unity Catalog lineage.

• governance_scanner.py - Scans workspace for governance violations:
• Unclassified tables (missing classification tags)
• Overly permissive access (SELECT granted to ALL or public groups)
• Missing required tags (owner, classification, retention)
• Unused service principals (no queries in 90+ days)

• policy_enforcer.py - Blocks operations that violate governance policies:
• Prevents creation of unclassified tables
• Blocks granting excessive permissions
• Enforces naming conventions

• governance_dashboard.py - Generates JSON metrics for dashboarding:
• Classification coverage percentage
• PII exposure risk score
• Permission compliance rate
• Tag completeness metrics

Terraform Module

Location: terraform/

Infrastructure-as-Code module for deploying the complete governance framework:

module "governance" {
  source = "./terraform"

  catalogs         = ["prod", "staging", "dev"]
  admin_group      = "data-platform-admins"
  engineer_group   = "data-engineers"
  analyst_group    = "data-analysts"
  viewer_group     = "data-viewers"
  enable_audit_log = true
}

Resources Created:

• Unity Catalog catalogs and schemas
• Account-level groups with nested membership
• Permission grants following the RBAC matrix
• SQL warehouses for governance queries
• Cluster policies for secure compute

Configuration

All configuration is driven by YAML files that you customize before deployment:

rbac/role_definitions.yaml

roles:
  admin:
    description: "Full metastore and catalog management"
    members:
      - "platform-lead@your-org.com"

classification/pii_patterns.yaml

patterns:
  email:
    regex: "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}"
    column_hints: ["email", "e_mail", "email_address"]

terraform/terraform.tfvars.example

databricks_host  = "https://your-workspace.cloud.databricks.com"
catalogs         = ["prod", "staging", "dev"]

Integration Patterns

With Azure Data Factory

Use ADF to orchestrate governance scans on a schedule:

{
  "name": "GovernanceScanPipeline",
  "activities": [
    {
      "name": "RunGovernanceScanner",
      "type": "DatabricksNotebook",
      "linkedServiceName": "AzureDatabricks",
      "typeProperties": {
        "notebookPath": "/Shared/governance/governance_scanner"
      }
    }
  ]
}

With Databricks Workflows

Create a scheduled job for continuous governance monitoring:

# Schedule governance scanner as a Databricks job
job_config = {
    "name": "governance-scanner-daily",
    "schedule": {
        "quartz_cron_expression": "0 0 6 * * ?",
        "timezone_id": "UTC"
    },
    "tasks": [
        {
            "task_key": "scan",
            "python_wheel_task": {
                "package_name": "governance_pack",
                "entry_point": "scan"
            }
        }
    ]
}

With Slack/Teams Alerts

Configure the governance scanner to send alerts:

scanner = GovernanceScanner(catalog="prod")
violations = scanner.scan_all()

if violations:
    # Send to webhook
    webhook_url = dbutils.secrets.get("governance", "slack-webhook")
    send_alert(webhook_url, violations)

Deployment Guide

Phase 1: Foundation (Week 1)

1. Deploy Terraform module to create catalogs, groups, and base permissions

2. Run setup_roles.py to configure RBAC across all catalogs

3. Validate with audit_permissions.py

Phase 2: Classification (Week 2)

1. Run PII scanner across all schemas

2. Review findings and apply classification tags

3. Generate classification coverage report

4. Set up column masking for sensitive fields

Phase 3: Compliance (Week 3)

1. Deploy consent tracking schema

2. Configure GDPR right-to-deletion handlers

3. Set up CCPA opt-out tracking

4. Run through compliance checklists

Phase 4: Automation (Week 4)

1. Deploy governance scanner as a scheduled job

2. Enable tag propagation for lineage-based tagging

3. Activate policy enforcer for new table creation

4. Set up governance dashboard

Phase 5: Monitoring (Ongoing)

1. Review governance dashboard weekly

2. Address violations flagged by the scanner

3. Re-run PII scanner monthly for new data

4. Update compliance checklists quarterly