Medallion Pipeline Simulator

🔒 This demo processes a bundled 22-row sample. The full framework connects to your real data at any scale.

• notebooks/ — Bronze/Silver/Gold templates with incremental & full-refresh modes, watermark-based loading, SCD Type 2
• ingestion_templates/ — OOP pipeline classes for API, database, file, and streaming ingestion with merge/append/overwrite write modes
• utils/quality_checks.py — CheckResult + QualityReport framework: null, uniqueness, referential integrity, freshness, range validation with structured logging
• utils/config_manager.py — Widget-based parameters with environment variable fallbacks, typed access, validation
• utils/logging_utils.py — Structured, queryable logging with RunContext capture (notebook path, cluster, job ID)
• utils/environment.py — Auto-detect dev/staging/prod workspace environment with catalog prefix resolution
• utils/secrets_manager.py — Unified interface for Databricks secret scopes and Azure Key Vault
• unity_catalog_setup/ — SQL scripts for catalog, storage credentials, external locations, data governance policies
• medallion_bootstrap/ — Programmatic catalog/schema/RBAC setup with configurable environments and group-based permissions
• testing/ — Nutter-pattern notebook testing with pytest integration and mock dbutils + spark fixtures
• cicd/ — DABs bundle configuration, pre-commit hooks, deployment templates