← Back to all products

Databricks Notebook Framework

$59

Production-grade notebook development framework with structured project templates, reusable utility modules, testing patterns, and CI/CD integration for Databricks.

📁 29 files🏷 v2.0.0
PythonJSONMarkdownYAMLSQLAWSAzureDatabricksPySparkSpark

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 29 files

databricks-notebook-framework/ ├── README.md ├── cicd/ │ ├── databricks.yml │ └── pre-commit-config.yaml ├── ingestion_templates/ │ ├── api_ingestion.py │ ├── base_pipeline.py │ ├── database_ingestion.py │ ├── file_ingestion.py │ └── streaming_ingestion.py ├── medallion_bootstrap/ │ ├── 01_create_catalogs.py │ ├── 02_create_schemas.py │ ├── 03_grant_permissions.py │ └── config.py ├── notebooks/ │ ├── bronze_ingest_template.py │ ├── gold_aggregate_template.py │ └── silver_transform_template.py ├── project-scaffold/ │ └── README.md ├── standards/ │ └── NOTEBOOK_STANDARDS.md ├── testing/ │ ├── conftest.py │ └── test_framework.py ├── unity_catalog_setup/ │ ├── data_governance_policies.md │ ├── setup_catalogs.sql │ ├── setup_credentials.sql │ └── setup_external_locations.sql └── utils/ ├── config_manager.py ├── environment.py ├── logging_utils.py ├── quality_checks.py └── secrets_manager.py

📖 Documentation Preview README excerpt

Databricks Notebook Framework

Production-grade notebook development framework for Databricks Lakehouse

By [Datanest Digital](https://datanest.dev) | Version 2.0.0 | $59

What You Get

A complete, battle-tested framework for building Databricks notebooks that follow medallion architecture best practices. Stop reinventing the wheel on every project — start with production-ready templates that handle logging, error handling, data quality, configuration, and CI/CD out of the box.

Key Features

  • Medallion Architecture Templates — Bronze, Silver, and Gold layer notebook templates with real-world patterns
  • Ingestion Pipeline Templates — OOP-based pipeline classes for API, database, file, and streaming ingestion with merge/append/overwrite write modes, deduplication, and metadata columns
  • Streaming Ingestion — Azure Event Hub and Apache Kafka ingestion with checkpoint management, OAuth/SASL auth, and configurable triggers
  • Unity Catalog Setup — SQL scripts for catalog creation, storage credentials, external locations, and data governance policies
  • Medallion Bootstrap — Programmatic catalog/schema/RBAC setup with configurable environments and group-based permissions
  • Environment Detection — Auto-detect Databricks workspace environment (dev/staging/prod) with storage config, catalog prefix resolution, and Key Vault integration
  • Incremental & Full-Refresh Modes — Built-in watermark-based incremental ingestion with fallback to full refresh
  • Data Quality Framework — Null checks, uniqueness validation, referential integrity, and freshness monitoring
  • SCD Type 2 Support — Slowly changing dimension logic ready to use in Silver layer transformations
  • Structured Logging — Consistent, queryable logging across all notebooks with run context capture
  • Configuration Management — Widget-based parameters with environment variable fallbacks
  • Secrets Management — Unified interface for Databricks secret scopes and Azure Key Vault
  • Testing Framework — Nutter-pattern testing with pytest integration and mock fixtures
  • CI/CD Ready — Pre-commit hooks, DABs bundle configuration, and deployment templates
  • Development Standards — Comprehensive notebook standards document for team alignment

File Listing


databricks-notebook-framework/
├── README.md                              # This file
├── manifest.json                          # Package manifest
│
├── notebooks/
│   ├── bronze_ingest_template.py          # Bronze layer ingestion template
│   ├── silver_transform_template.py       # Silver layer transformation template
│   └── gold_aggregate_template.py         # Gold layer aggregation template
│
├── ingestion_templates/
│   ├── base_pipeline.py                   # Abstract base pipeline class (merge/append/overwrite)
│   ├── api_ingestion.py                   # REST API ingestion with pagination & retry
│   ├── database_ingestion.py              # JDBC database ingestion (full/incremental)
│   ├── file_ingestion.py                  # File-based ingestion (CSV, JSON, Parquet, etc.)
│   └── streaming_ingestion.py             # Event Hub / Kafka streaming ingestion
│
├── unity_catalog_setup/
│   ├── setup_catalogs.sql                 # SQL to create Unity Catalog catalogs
│   ├── setup_credentials.sql              # Storage credential configuration
│   ├── setup_external_locations.sql       # External location definitions
│   └── data_governance_policies.md        # Data governance policy documentation
│
├── medallion_bootstrap/
│   ├── config.py                          # Bootstrap configuration (catalogs, schemas, groups)
│   ├── 01_create_catalogs.py              # Step 1: Create Unity Catalog catalogs
│   ├── 02_create_schemas.py               # Step 2: Create schemas in each catalog
│   └── 03_grant_permissions.py            # Step 3: Grant RBAC permissions to groups

*... continues with setup instructions, usage examples, and more.*

📄 Code Sample .py preview

ingestion_templates/api_ingestion.py """ REST API Ingestion Pipeline ============================ Ingests data from REST APIs into the bronze layer of a Delta Lake medallion architecture. Supports: - Pagination (offset, cursor, and link-header based) - Configurable retry with exponential backoff - Content hashing for deduplication / change detection - Rate limiting with configurable delays - Request/response logging with redacted headers - Both full and incremental (date-range) load modes Usage (in a Databricks notebook): from ingestion_templates.api_ingestion import ApiIngestionPipeline, ApiConfig from ingestion_templates.base_pipeline import PipelineConfig api_config = ApiConfig( base_url="https://api.example.com", endpoint="/v2/customers", headers={"Authorization": "Bearer <token>"}, pagination_style="offset", page_size=500, ) pipeline_config = PipelineConfig( pipeline_name="customers_api", source_system="example_api", source_class="customers", target_catalog="dev_bronze", target_schema="example_api", target_table="customers", keys=["customer_id"], write_mode="merge", ) pipeline = ApiIngestionPipeline( config=pipeline_config, api_config=api_config, ) metrics = pipeline.run() Compatible with: Databricks Runtime 13.x+, Python 3.10+ """ from __future__ import annotations import hashlib # ... 413 more lines ...
Buy Now — $59 Back to Products