Medallion Architecture Accelerator

$79

Production implementation of Bronze/Silver/Gold architecture on Databricks with metadata-driven pipeline generator and quality gates.

📁 34 files🏷 v2.0.0

PythonJSONMarkdownYAMLDatabricksPySparkSparkDelta Lake

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 34 files

medallion-architecture-accelerator/ ├── README.md ├── config/ │ ├── example_sources.yaml │ └── pipeline_generator.py ├── docs/ │ ├── architecture_guide.md │ ├── cheatsheets/ │ │ ├── layer_comparison.md │ │ ├── migration_checklist.md │ │ └── naming_conventions_cheatsheet.md │ ├── diagrams/ │ │ ├── data_flow.md │ │ ├── decision_tree.md │ │ └── medallion_overview.md │ ├── guide/ │ │ ├── 01_introduction.md │ │ ├── 02_decision_framework.md │ │ ├── 03_bronze_layer.md │ │ ├── 04_silver_layer.md │ │ ├── 05_gold_layer.md │ │ ├── 06_naming_conventions.md │ │ ├── 07_schema_evolution.md │ │ ├── 08_data_quality_gates.md │ │ ├── 09_anti_patterns.md │ │ └── 10_reference_architectures.md │ └── performance_benchmarks.md ├── framework/ │ ├── bronze/ │ │ ├── auto_loader_ingestor.py │ │ ├── base_ingestor.py │ │ └── jdbc_ingestor.py │ ├── gold/ │ │ ├── aggregation_builder.py │ │ └── fact_table_builder.py │ ├── naming_convention_generator.py │ ├── schema_migration.py │ └── silver/ │ ├── base_transformer.py │ └── scd_type2.py ├── optimization/ │ └── delta_optimization.py ├── quality/ │ └── quality_gates.py └── testing/ └── layer_tests.py

📖 Documentation Preview README excerpt

Medallion Architecture Accelerator

By [Datanest Digital](https://datanest.dev) | Version 2.0.0 | $79

Production-ready framework for implementing the Medallion Architecture (Bronze / Silver / Gold) on Databricks with Delta Lake. Eliminate weeks of boilerplate and enforce best practices from day one.

v2.0 adds a comprehensive 10-chapter implementation guide, quick-reference cheatsheets, architecture diagrams, plus naming convention and schema migration utilities.

What You Get

Production Framework (`framework/`)

#### Bronze Layer (Raw Ingestion)

Base Ingestor -- Schema evolution, exactly-once semantics, source lineage tracking
Auto Loader Ingestor -- Cloudfiles-based streaming ingestion with rescue columns
JDBC Ingestor -- Incremental extraction from relational databases with watermark tracking

#### Silver Layer (Conformed / Cleansed)

Base Transformer -- Deduplication, type casting, null handling, data conformance
SCD Type 2 -- Slowly-changing dimension management with merge-based upserts

#### Gold Layer (Business / Aggregated)

Aggregation Builder -- Business KPIs, time-series rollups, dimensional model support
Fact Table Builder -- Star-schema fact tables with conformed dimension joins

#### Utility Tools (New in v2.0)

Naming Convention Generator -- Enforce consistent table/column/path naming across your lakehouse
Schema Migration -- Track and apply schema changes across medallion layers with full audit trail

Pipeline Generation (`config/`)

Metadata-Driven Generator -- Define sources in YAML, generate all three layers automatically
Example Configs -- Five pre-built source configurations covering common patterns

Quality & Optimization

Quality Gates (quality/) -- Null checks, schema validation, business rules, inter-layer assertions
Delta Optimization (optimization/) -- OPTIMIZE, ZORDER, VACUUM, liquid clustering per layer tier
Layer Tests (testing/) -- Unit and integration testing framework for every layer

10-Chapter Implementation Guide (New in v2.0) (`docs/guide/`)

A complete, opinionated guide to designing, building, and operating a medallion lakehouse:

Chapter	Topic	What You'll Learn
01	Introduction	Why medallion, when to use it, core principles
02	Decision Framework	How to decide if medallion fits your use case
03	Bronze Layer	Ingestion patterns, schema evolution, exactly-once
04	Silver Layer	Cleansing, conformance, deduplication, SCD
05	Gold Layer	Aggregations, dimensional modeling, materialized views
06	Naming Conventions	Databases, tables, columns, paths -- consistent naming
07	Schema Evolution	Safe schema changes, migration strategies, compatibility
08	Data Quality Gates	Null checks, business rules, inter-layer assertions
09	Anti-Patterns	Common mistakes and how to avoid them
10	Reference Architectures	Three complete architectures (startup → enterprise)

Quick-Reference Cheatsheets (New in v2.0) (`docs/cheatsheets/`)

Layer Comparison -- Side-by-side comparison of Bronze vs Silver vs Gold responsibilities
Migration Checklist -- Step-by-step checklist for migrating existing pipelines to medallion

... continues with setup instructions, usage examples, and more.

📄 Code Sample .py preview

config/pipeline_generator.py # Databricks notebook source # MAGIC %md # MAGIC # Metadata-Driven Pipeline Generator # MAGIC **Medallion Architecture Accelerator** by [Datanest Digital](https://datanest.dev) # MAGIC # MAGIC Reads a YAML configuration file and generates complete Bronze, Silver, # MAGIC and Gold layer pipelines. Enables a declarative, config-driven # MAGIC approach to building medallion architectures. # COMMAND ---------- from __future__ import annotations import copy from dataclasses import dataclass, field from pathlib import Path from typing import Any, Dict, List, Optional import yaml from pyspark.sql import SparkSession # COMMAND ---------- # MAGIC %md # MAGIC ## Pipeline Configuration Model # COMMAND ---------- @dataclass class SourceConfig: """Parsed representation of a single source YAML configuration. This is an intermediate model that the generator converts into concrete layer configurations (Bronze, Silver, Gold). """ # Source identification. name: str = "" source_system: str = "" source_type: str = "auto_loader" # auto_loader | jdbc | custom # Bronze settings. source_format: str = "json" source_path: str = "" schema_hints: Dict[str, str] = field(default_factory=dict) bronze_catalog: str = "" bronze_database: str = "" bronze_table: str = "" # ... 307 more lines ...

Buy Now — $79 Back to Products