What You Get

This chapter covers the core features and capabilities of Medallion Architecture Accelerator.

What You Get

Production Framework (framework/)

#### Bronze Layer (Raw Ingestion)

• Base Ingestor -- Schema evolution, exactly-once semantics, source lineage tracking
• Auto Loader Ingestor -- Cloudfiles-based streaming ingestion with rescue columns
• JDBC Ingestor -- Incremental extraction from relational databases with watermark tracking

#### Silver Layer (Conformed / Cleansed)

• Base Transformer -- Deduplication, type casting, null handling, data conformance
• SCD Type 2 -- Slowly-changing dimension management with merge-based upserts

#### Gold Layer (Business / Aggregated)

• Aggregation Builder -- Business KPIs, time-series rollups, dimensional model support
• Fact Table Builder -- Star-schema fact tables with conformed dimension joins

#### Utility Tools (New in v2.0)

• Naming Convention Generator -- Enforce consistent table/column/path naming across your lakehouse
• Schema Migration -- Track and apply schema changes across medallion layers with full audit trail

Pipeline Generation (config/)

• Metadata-Driven Generator -- Define sources in YAML, generate all three layers automatically
• Example Configs -- Five pre-built source configurations covering common patterns

Quality & Optimization

• Quality Gates (quality/) -- Null checks, schema validation, business rules, inter-layer assertions
• Delta Optimization (optimization/) -- OPTIMIZE, ZORDER, VACUUM, liquid clustering per layer tier
• Layer Tests (testing/) -- Unit and integration testing framework for every layer

10-Chapter Implementation Guide (New in v2.0) (docs/guide/)

A complete, opinionated guide to designing, building, and operating a medallion lakehouse:

Quick-Reference Cheatsheets (New in v2.0) (docs/cheatsheets/)

• Layer Comparison -- Side-by-side comparison of Bronze vs Silver vs Gold responsibilities
• Migration Checklist -- Step-by-step checklist for migrating existing pipelines to medallion
• Naming Conventions -- At-a-glance naming rules for databases, tables, columns, and paths

Architecture Diagrams (New in v2.0) (docs/diagrams/)

• Medallion Overview -- High-level architecture diagram (ASCII art)
• Data Flow -- Detailed data flow through all three layers
• Decision Tree -- Visual decision tree for choosing ingestion and transformation patterns

Existing Documentation (docs/)

• Architecture guide with patterns and anti-patterns
• Performance benchmarks and sizing guidelines

Quick Start

1. Configure a Source

# config/sources/orders.yaml
source:
  name: orders
  type: auto_loader
  format: json
  path: "s3://raw-bucket/orders/"
  schema_hints:
    order_id: long
    customer_id: long
    order_date: timestamp

bronze:
  database: raw
  table: orders
  partition_columns: [ingestion_date]

silver:
  database: cleansed
  table: orders
  primary_keys: [order_id]
  dedup_columns: [order_id, order_date]

gold:
  - database: analytics
    table: daily_order_summary
    aggregation: daily_summary
    group_by: [order_date]
    metrics:
      - {column: order_total, function: sum, alias: total_revenue}
      - {column: order_id, function: count, alias: order_count}

2. Generate the Pipeline

from config.pipeline_generator import PipelineGenerator

generator = PipelineGenerator("config/sources/orders.yaml")
generator.generate_all()

3. Run Each Layer

# Bronze -- ingest raw data
bronze_pipeline.run()

# Silver -- cleanse and conform
silver_pipeline.run()

# Gold -- build business aggregations
gold_pipeline.run()

4. Use the Naming Convention Generator

from framework.naming_convention_generator import NamingConventionGenerator

gen = NamingConventionGenerator(environment="production")
table_name = gen.generate_table_name(layer="silver", domain="orders", entity="line_items")
# → "silver_orders_line_items"

5. Run Schema Migrations

from framework.schema_migration import SchemaMigrationManager

mgr = SchemaMigrationManager(catalog="main", environment="production")
mgr.plan_migration(source_schema, target_schema)
mgr.apply()

Project Structure

Follow this guide to get Medallion Architecture Accelerator up and running in your environment.

Project Structure

medallion-architecture-accelerator/
├── framework/
│   ├── bronze/
│   │   ├── base_ingestor.py              # Base ingestion class
│   │   ├── auto_loader_ingestor.py       # Auto Loader template
│   │   └── jdbc_ingestor.py              # JDBC incremental ingestor
│   ├── silver/
│   │   ├── base_transformer.py           # Silver transformation base
│   │   └── scd_type2.py                  # SCD Type 2 implementation
│   ├── gold/
│   │   ├── aggregation_builder.py        # Aggregation and KPI builder
│   │   └── fact_table_builder.py         # Fact table builder
│   ├── naming_convention_generator.py    # [NEW] Naming convention utility
│   └── schema_migration.py              # [NEW] Schema migration manager
├── config/
│   ├── pipeline_generator.py             # Metadata-driven generator
│   └── example_sources.yaml              # Example source configs
├── quality/
│   └── quality_gates.py                  # Inter-layer quality gates
├── optimization/
│   └── delta_optimization.py             # Delta optimization patterns
├── testing/
│   └── layer_tests.py                    # Testing framework
└── docs/
    ├── architecture_guide.md             # Architecture reference
    ├── performance_benchmarks.md         # Benchmarks and sizing
    ├── guide/                            # [NEW] 10-chapter implementation guide
    │   ├── 01_introduction.md
    │   ├── 02_decision_framework.md
    │   ├── 03_bronze_layer.md
    │   ├── 04_silver_layer.md
    │   ├── 05_gold_layer.md
    │   ├── 06_naming_conventions.md
    │   ├── 07_schema_evolution.md
    │   ├── 08_data_quality_gates.md
    │   ├── 09_anti_patterns.md
    │   └── 10_reference_architectures.md
    ├── cheatsheets/                      # [NEW] Quick-reference sheets
    │   ├── layer_comparison.md
    │   ├── migration_checklist.md
    │   └── naming_conventions_cheatsheet.md
    └── diagrams/                         # [NEW] Architecture diagrams
        ├── data_flow.md
        ├── decision_tree.md
        └── medallion_overview.md

Requirements

• Databricks Runtime 13.3 LTS or later
• Delta Lake 2.4+
• Unity Catalog (recommended) or Hive Metastore
• Python 3.10+
• PySpark 3.4+