What You Get

This chapter covers the core features and capabilities of Databricks Notebook Framework.

What You Get

A complete, battle-tested framework for building Databricks notebooks that follow medallion architecture best practices. Stop reinventing the wheel on every project — start with production-ready templates that handle logging, error handling, data quality, configuration, and CI/CD out of the box.

Key Features

• Medallion Architecture Templates — Bronze, Silver, and Gold layer notebook templates with real-world patterns
• Ingestion Pipeline Templates — OOP-based pipeline classes for API, database, file, and streaming ingestion with merge/append/overwrite write modes, deduplication, and metadata columns
• Streaming Ingestion — Azure Event Hub and Apache Kafka ingestion with checkpoint management, OAuth/SASL auth, and configurable triggers
• Unity Catalog Setup — SQL scripts for catalog creation, storage credentials, external locations, and data governance policies
• Medallion Bootstrap — Programmatic catalog/schema/RBAC setup with configurable environments and group-based permissions
• Environment Detection — Auto-detect Databricks workspace environment (dev/staging/prod) with storage config, catalog prefix resolution, and Key Vault integration
• Incremental & Full-Refresh Modes — Built-in watermark-based incremental ingestion with fallback to full refresh
• Data Quality Framework — Null checks, uniqueness validation, referential integrity, and freshness monitoring
• SCD Type 2 Support — Slowly changing dimension logic ready to use in Silver layer transformations
• Structured Logging — Consistent, queryable logging across all notebooks with run context capture
• Configuration Management — Widget-based parameters with environment variable fallbacks
• Secrets Management — Unified interface for Databricks secret scopes and Azure Key Vault
• Testing Framework — Nutter-pattern testing with pytest integration and mock fixtures
• CI/CD Ready — Pre-commit hooks, DABs bundle configuration, and deployment templates
• Development Standards — Comprehensive notebook standards document for team alignment

File Listing

databricks-notebook-framework/
├── README.md                              # This file
├── manifest.json                          # Package manifest
│
├── notebooks/
│   ├── bronze_ingest_template.py          # Bronze layer ingestion template
│   ├── silver_transform_template.py       # Silver layer transformation template
│   └── gold_aggregate_template.py         # Gold layer aggregation template
│
├── ingestion_templates/
│   ├── base_pipeline.py                   # Abstract base pipeline class (merge/append/overwrite)
│   ├── api_ingestion.py                   # REST API ingestion with pagination & retry
│   ├── database_ingestion.py              # JDBC database ingestion (full/incremental)
│   ├── file_ingestion.py                  # File-based ingestion (CSV, JSON, Parquet, etc.)
│   └── streaming_ingestion.py             # Event Hub / Kafka streaming ingestion
│
├── unity_catalog_setup/
│   ├── setup_catalogs.sql                 # SQL to create Unity Catalog catalogs
│   ├── setup_credentials.sql              # Storage credential configuration
│   ├── setup_external_locations.sql       # External location definitions
│   └── data_governance_policies.md        # Data governance policy documentation
│
├── medallion_bootstrap/
│   ├── config.py                          # Bootstrap configuration (catalogs, schemas, groups)
│   ├── 01_create_catalogs.py              # Step 1: Create Unity Catalog catalogs
│   ├── 02_create_schemas.py               # Step 2: Create schemas in each catalog
│   └── 03_grant_permissions.py            # Step 3: Grant RBAC permissions to groups
│
├── utils/
│   ├── logging_utils.py                   # Structured logging module
│   ├── config_manager.py                  # Widget & environment config management
│   ├── quality_checks.py                  # Data quality validation functions
│   ├── secrets_manager.py                 # Secrets retrieval wrapper
│   └── environment.py                     # Environment detection & workspace config
│
├── testing/
│   ├── test_framework.py                  # Nutter-pattern notebook testing framework
│   └── conftest.py                        # Pytest fixtures with mock dbutils & spark
│
├── cicd/
│   ├── pre-commit-config.yaml             # Pre-commit hooks for notebook linting
│   └── databricks.yml                     # DABs bundle configuration template
│
├── standards/
│   └── NOTEBOOK_STANDARDS.md              # Notebook development standards
│
└── project-scaffold/
    └── README.md                          # Quick-start scaffold instructions

Getting Started

1. Clone Into Your Workspace

Copy this framework into your Databricks Repos or local project directory:

cp -r databricks-notebook-framework/ /path/to/your/project/

2. Configure Your Environment

Update utils/config_manager.py defaults for your catalog, schema, and environment naming conventions.

For multi-environment setups, update utils/environment.py with your workspace IDs:

from utils.environment import get_environment

env = get_environment()          # Auto-detect from workspace
env = get_environment("dev")     # Explicit override

print(env.bronze_catalog)        # "dev_bronze"
print(env.storage.raw_root)      # "abfss://raw@styourorgdev.dfs.core.windows.net"

3. Bootstrap Your Medallion Architecture

Use the medallion_bootstrap/ scripts to set up catalogs, schemas, and permissions:

# In a Databricks notebook — run in order:
%run ./medallion_bootstrap/config
%run ./medallion_bootstrap/01_create_catalogs
%run ./medallion_bootstrap/02_create_schemas
%run ./medallion_bootstrap/03_grant_permissions

Or use the Unity Catalog SQL scripts for a SQL-first approach:

-- Run in a SQL notebook or editor:
-- 1. setup_catalogs.sql
-- 2. setup_credentials.sql
-- 3. setup_external_locations.sql

4. Create Your First Notebook

Start from any template in notebooks/:

# Copy bronze_ingest_template.py
# Update widget defaults for your source
# Implement source-specific read logic in the marked section
# Run and verify

5. Use Ingestion Pipeline Templates

For production ingestion, use the OOP pipeline templates:

from ingestion_templates.api_ingestion import ApiIngestionPipeline
from ingestion_templates.base_pipeline import PipelineConfig

config = PipelineConfig(
    pipeline_name="my_api_pipeline",
    source_system="salesforce",
    source_class="accounts",
    target_catalog="prod_bronze",
    target_schema="salesforce",
    target_table="accounts",
    write_mode="merge",
)

pipeline = ApiIngestionPipeline(config=config, ...)
pipeline.run()

For streaming sources (Event Hub / Kafka):

from ingestion_templates.streaming_ingestion import (
    StreamingIngestionPipeline, EventHubConfig, StreamConfig,
)
from ingestion_templates.base_pipeline import PipelineConfig

pipeline = StreamingIngestionPipeline(
    config=pipeline_config,
    eh_config=eh_config,
    stream_config=stream_config,
)
pipeline.run_streaming()

6. Set Up Quality Checks

Import quality check functions into your Silver and Gold notebooks:

from utils.quality_checks import QualityChecker
checker = QualityChecker(spark)
results = checker.run_all_checks(df, config)

7. Enable CI/CD

1. Copy cicd/pre-commit-config.yaml to your repo root as .pre-commit-config.yaml

2. Customize cicd/databricks.yml for your DABs deployment

3. Run pre-commit install

Ingestion Templates

Follow this guide to get Databricks Notebook Framework up and running in your environment.

Ingestion Templates

The ingestion_templates/ directory contains production-ready OOP pipeline classes:

All templates inherit from BasePipeline and share a consistent PipelineConfig interface.

Unity Catalog Setup

The unity_catalog_setup/ directory contains SQL scripts for Unity Catalog infrastructure:

Medallion Bootstrap

The medallion_bootstrap/ directory provides a programmatic approach to setting up the medallion architecture:

Run these scripts in order (01 → 02 → 03) in a Databricks notebook.