Chapter 1: Introduction to Medallion Architecture

What Is Medallion Architecture?

Medallion architecture is a data design pattern that organizes a lakehouse into three logical layers — bronze, silver, and gold — each representing a progressively higher level of data quality, structure, and business meaning.

Raw Sources → [Bronze] → [Silver] → [Gold] → Consumers
               (raw)     (clean)    (business)

That's the one-sentence version. Here's what it actually means in practice:

• Bronze stores data exactly as it arrived from source systems. No transformations, no filtering, no opinions. If the source sent garbage, bronze has garbage — and that's correct.
• Silver applies cleaning, deduplication, schema enforcement, and conforming transformations. This is where data becomes trustworthy but remains general-purpose.
• Gold shapes data for specific business use cases — aggregations, metrics, feature stores, reporting views. Gold tables answer business questions directly.

The key insight is separation of concerns: each layer has exactly one job, and downstream layers never need to understand upstream source systems.

A Brief History

Before Medallion: The Data Warehouse Era

Traditional data warehouses (Kimball, Inmon) organized data into staging, integration, and presentation layers. The medallion pattern is this concept's spiritual successor, adapted for the lakehouse paradigm where:

• Storage is cheap (cloud object storage vs. expensive MPP databases)
• Compute is elastic (Spark clusters vs. fixed warehouse capacity)
• Data formats are open (Delta/Parquet vs. proprietary formats)
• Schema enforcement is optional (schema-on-read vs. schema-on-write)

The Databricks Origin

The term "medallion architecture" was popularized by Databricks around 2019-2020, but the pattern existed before the name. Teams running Spark on data lakes had independently converged on raw → cleaned → aggregated layering because it solved real problems:

1. Reproducibility: If a transformation has a bug, you fix it and re-run from bronze. The raw data is always there.

2. Auditability: You can trace any gold metric back through silver to the exact raw record.

3. Isolation: A broken ingestion pipeline doesn't corrupt your clean data. A bad business rule doesn't destroy your cleaned data.

Medallion in the Delta Lake Era

Delta Lake made medallion practical at scale by adding:

• ACID transactions: No more half-written directories of Parquet files
• Time travel: Query any layer at any point in history
• MERGE: Efficient upserts for silver/gold without full rewrites
• Schema enforcement/evolution: Catch schema drift at ingestion time
• OPTIMIZE + Z-ORDER: Physical layout optimization per layer

Without Delta Lake, medallion on a raw data lake was fragile. With it, the pattern became production-grade.

When Medallion Architecture Applies

Medallion is not always the right choice. It works best when:

Strong Fit

• Multiple source systems feeding into a shared analytical platform
• Data quality varies across sources (some clean APIs, some messy CSVs)
• Audit and lineage requirements exist (finance, healthcare, regulated industries)
• Multiple consumer teams need the same data at different levels of aggregation
• Batch or micro-batch processing is the primary pattern
• Team includes both engineers and analysts (engineers own bronze/silver, analysts own gold)

Weak Fit

• Single source, single consumer: The overhead of three layers isn't justified
• Pure real-time streaming: Sub-second latency requirements may need a streaming-first architecture
• Small data: If your entire dataset fits in a single DataFrame, you don't need a lakehouse architecture
• Exploratory/ad-hoc only: If nobody depends on the data downstream, just query the source

Signs You Need to Reconsider

If you find yourself in any of these situations, revisit the decision framework in Chapter 2:

• Your bronze and silver layers are identical (you're not actually cleaning anything)
• Every consumer reads from bronze anyway because silver is always behind
• You have more layers than actual transformations
• Your gold layer is just SELECT * FROM silver with a different name

Core Principles

1. Bronze Is Sacred

Bronze data is your insurance policy. It should be:

• Append-only (or at minimum, never modified destructively)
• Schema-preserving (store the source schema, even if it's messy)
• Complete (every record that arrived, including duplicates and errors)
• Timestamped (when did this record arrive in your system?)

If you lose bronze, you lose the ability to reprocess. Treat it accordingly.

2. Silver Is Where Opinions Live

Silver is where your data team makes decisions:

• Which records are duplicates?
• What's the canonical schema?
• How do we handle nulls?
• What's the primary key?
• Which SCD type for this dimension?

These are business decisions disguised as technical ones. Document them.

3. Gold Is Use-Case-Specific

Gold tables serve specific consumers. It's normal and expected to have:

• Multiple gold tables derived from the same silver tables
• Different gold layers for different teams (analytics gold, ML gold, reporting gold)
• Gold tables that join across multiple silver tables
• Gold tables with different refresh frequencies

Gold is not "one table to rule them all." It's the right shape for the right consumer.

4. Data Flows Downstream, Never Upstream

Bronze → Silver → Gold → Consumers
  ↑         ↑        ↑
  |         |        |
  NEVER reference a downstream layer from an upstream one

Silver never reads from gold. Bronze never reads from silver. Violations of this rule create circular dependencies that are nightmares to debug and maintain.

5. Each Layer Owns Its Quality Contract

If a consumer finds dirty data in a gold table, you don't fix it in gold — you fix it in silver and let it flow through.

Medallion vs. ETL/ELT

A common confusion: "Isn't this just ETL with fancy names?"

The distinction matters:

Traditional ETL/ELT is a process — Extract, Transform, Load (or Load then Transform). It describes *what you do*.

Medallion architecture is a structure — how you organize the *results* of those processes. It describes *where things live*.

You still do ETL within a medallion architecture:

• Bronze: Extract + Load (minimal or no transformation)
• Silver: Transform (clean, conform, deduplicate)
• Gold: Transform (aggregate, join, reshape for consumers)

The difference is that medallion gives you persistent, queryable layers at each stage. Traditional ETL often treats staging as ephemeral — data flows through and is discarded. In medallion, every layer is a first-class citizen that consumers can query.

Physical vs. Logical Layers

An important distinction that trips up beginners:

Logical Layers

The bronze/silver/gold layers are logical concepts. They represent different levels of data maturity, not necessarily different storage locations.

Physical Implementation

How you physically implement layers depends on your platform:

-- Option 1: Separate catalogs (recommended for Unity Catalog)
CREATE CATALOG bronze;
CREATE CATALOG silver;
CREATE CATALOG gold;

-- Option 2: Separate schemas within one catalog
CREATE SCHEMA main.bronze;
CREATE SCHEMA main.silver;
CREATE SCHEMA main.gold;

-- Option 3: Naming prefix within one schema (not recommended)
CREATE TABLE analytics.bronze_orders ...;
CREATE TABLE analytics.silver_orders ...;
CREATE TABLE analytics.gold_daily_revenue ...;

Chapter 6 covers naming conventions in detail. The key point: logical separation is mandatory. Physical separation is a governance and organizational choice.

What This Guide Covers

The rest of this guide walks through:

Each chapter includes:

• Conceptual explanation
• PySpark code examples
• Best practices and common mistakes
• Decision guidance for ambiguous situations

Quick Example: The Pattern in Action

Here's a minimal end-to-end example to ground the concepts:

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.getOrCreate()

# ── Bronze: Ingest raw JSON exactly as received ──
bronze_orders = (
    spark.readStream
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.schemaLocation", "/checkpoints/orders/schema")
    .load("/landing/orders/")
    .withColumn("_ingested_at", F.current_timestamp())
    .withColumn("_source_file", F.input_file_name())
)

bronze_orders.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option("checkpointLocation", "/checkpoints/orders/bronze") \
    .toTable("bronze.ecommerce.raw_orders")

# ── Silver: Clean, deduplicate, enforce types ──
silver_orders = (
    spark.readStream
    .table("bronze.ecommerce.raw_orders")
    .dropDuplicates(["order_id"])
    .filter(F.col("order_id").isNotNull())
    .withColumn("order_total", F.col("order_total").cast("decimal(10,2)"))
    .withColumn("order_date", F.to_date("order_date", "yyyy-MM-dd"))
    .drop("_source_file")
)

silver_orders.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option("checkpointLocation", "/checkpoints/orders/silver") \
    .toTable("silver.ecommerce.orders")

# ── Gold: Business aggregate ──
gold_daily_revenue = (
    spark.read
    .table("silver.ecommerce.orders")
    .groupBy("order_date")
    .agg(
        F.sum("order_total").alias("total_revenue"),
        F.countDistinct("order_id").alias("order_count"),
        F.avg("order_total").alias("avg_order_value"),
    )
)

gold_daily_revenue.write \
    .format("delta") \
    .mode("overwrite") \
    .saveAsTable("gold.analytics.daily_revenue")

This is deliberately simplified. The rest of this guide covers the production concerns that this toy example ignores: error handling, schema evolution, data quality gates, naming conventions, partitioning, optimization, and more.

Key Takeaways

1. Medallion is a data organization pattern, not a tool or product

2. Three layers (bronze/silver/gold) represent increasing data maturity

3. Bronze preserves raw data for reprocessing and auditing

4. Silver enforces quality through cleaning and conforming

5. Gold serves consumers with use-case-specific shapes

6. Data flows downstream only — never reference a downstream layer from upstream

7. It's not always the right choice — Chapter 2 helps you decide

*Next: Chapter 2 — Decision Framework*

Chapter 2: Decision Framework — Medallion vs. Alternatives

Overview

This chapter is the most important one in the guide. Before you invest weeks implementing medallion architecture, you need to confirm it's actually the right pattern for your situation.

Too many teams adopt medallion because "Databricks recommends it" without evaluating whether their specific data landscape, team composition, and requirements actually call for it. This chapter gives you a structured decision process.

The Five Architecture Patterns

Before we build the decision tree, let's clearly define the alternatives:

1. Medallion Architecture (Bronze → Silver → Gold)

Summary: Three persistent layers of increasing data quality.

Best for: Multi-source analytical platforms with mixed consumer types and varying source quality.

Sources → [Bronze: raw] → [Silver: clean] → [Gold: business] → Consumers

2. Data Vault 2.0

Summary: Hub-and-spoke model with Hubs (business keys), Links (relationships), and Satellites (descriptive attributes). Optimized for auditability and historical tracking.

Best for: Environments with complex many-to-many relationships, heavy regulatory requirements, and frequent source system changes.

Sources → [Raw Vault: Hubs + Links + Satellites] → [Business Vault] → [Information Marts]

3. One Big Table (OBT)

Summary: Denormalize everything into wide tables optimized for specific query patterns. Skip intermediate layers entirely.

Best for: Small teams, single-purpose analytics, or when query performance matters more than flexibility.

Sources → [Staging] → [One Wide Table per Use Case] → Consumers

4. ELT-Only (Flat Pipeline)

Summary: Load raw data, transform directly into final shape. No persistent intermediate layers.

Best for: Simple data flows with few sources, minimal cleaning needed, and small teams that can hold the full pipeline in their heads.

Sources → [Raw Landing] → [Transform] → [Final Tables] → Consumers

5. Streaming-First (Kappa Architecture)

Summary: Everything is a stream. Batch is just a special case of streaming. State is maintained in stream processors, not in persistent layers.

Best for: Real-time requirements (<1 second latency), event-driven architectures, systems where the latest value always wins.

Sources → [Stream Processor] → [Materialized Views / Serving Layer] → Consumers

The Decision Tree

Work through these questions in order. Each answer narrows your options.

Question 1: How many source systems do you have?

Q: How many distinct source systems feed your platform?

  [1-2 sources] → Medallion is likely overkill.
                   Consider ELT-Only or One Big Table.
                   
  [3-10 sources] → Medallion is a strong candidate.
                    Continue to Question 2.
                    
  [10+ sources]  → Medallion is almost certainly right.
                    Consider Data Vault if relationships are complex.
                    Continue to Question 2.

Why this matters: Medallion's value comes from consolidating diverse sources into a common cleaned layer. With one or two sources, the overhead of three layers often exceeds the benefit.

Question 2: How varied is your source data quality?

Q: How clean is the data when it arrives?

  [Uniformly clean]   → You might not need a silver layer.
    (APIs with schemas,  Consider ELT-Only with quality checks.
     validated upstream)
     
  [Mixed quality]      → Medallion is the right choice.
    (some clean APIs,    Silver layer earns its keep.
     some messy CSVs,    Continue to Question 3.
     some legacy feeds)
     
  [Uniformly messy]    → Medallion is essential.
    (legacy systems,     You need bronze for preservation
     manual uploads,     and silver for heavy cleaning.
     third-party feeds)  Continue to Question 3.

Question 3: What are your latency requirements?

Q: How fresh does data need to be for consumers?

  [Daily or slower]    → Batch medallion works perfectly.
                         Standard bronze → silver → gold with
                         scheduled jobs.
                         
  [Hourly to minutes]  → Micro-batch medallion with Structured
                         Streaming. Still three layers, but
                         streaming between them.
                         
  [Seconds]            → Hybrid: streaming for hot path,
                         medallion for warm/cold path.
                         Consider Kappa for the hot path.
                         
  [Sub-second]         → Streaming-First / Kappa architecture.
                         Medallion for historical analytics only.

Question 4: Who consumes the data?

Q: Who reads from your platform?

  [Engineers only]     → Any pattern works. Pick based on
                         complexity, not accessibility.
                         
  [Analysts + Engineers] → Medallion is ideal.
                           Analysts read gold (SQL-friendly).
                           Engineers own bronze/silver.
                           
  [ML + Analytics]     → Medallion with extended gold layer.
                         Separate gold schemas for analytics
                         and feature stores.
                         
  [External APIs]      → Gold layer becomes a serving layer.
                         Consider materialized views or
                         dedicated serving infrastructure.

Question 5: What are your regulatory requirements?

Q: Do you have audit, lineage, or compliance requirements?

  [Minimal/none]       → Any pattern works.
  
  [Standard audit]     → Medallion provides natural lineage.
    (who changed what,   Bronze = source of truth.
     data retention)     Delta time travel for history.
     
  [Heavy compliance]   → Medallion or Data Vault.
    (GDPR, HIPAA,       Data Vault if you need relationship-level
     SOX, regulatory     auditing. Medallion + Unity Catalog
     reporting)          lineage if Databricks-native is preferred.

Question 6: How large is your data engineering team?

Q: How many people build and maintain data pipelines?

  [1-2 people]         → Keep it simple. ELT-Only or OBT.
                         Medallion adds overhead that small
                         teams can't maintain.
                         
  [3-8 people]         → Medallion works well. Clear ownership:
                         some own ingestion (bronze), some own
                         transformation (silver/gold).
                         
  [8+ people]          → Medallion is strongly recommended.
                         Clear layer boundaries prevent teams
                         from stepping on each other.
                         Consider Data Vault if data modeling
                         is centralized.

Decision Matrix Summary

Hybrid Patterns

Real-world architectures are rarely pure. Here are legitimate hybrid approaches:

Medallion + Streaming Hot Path

                    ┌─ [Streaming] → [Real-time Serving] ─┐
Sources ──────────>─┤                                       ├→ Consumers
                    └─ [Bronze] → [Silver] → [Gold] ──────┘
                       (batch/micro-batch for historical)

When: You need both real-time dashboards AND historical analytics.

Medallion + Data Vault (Silver = Vault)

Sources → [Bronze: raw] → [Silver: Data Vault structure] → [Gold: marts]

When: You want medallion's simplicity for ingestion and serving, but Data Vault's modeling rigor for the integration layer.

Medallion + OBT (Gold = Wide Tables)

Sources → [Bronze: raw] → [Silver: clean] → [Gold: denormalized wide tables]

When: Your consumers need scan-optimized wide tables, but you still need the cleaning/auditing benefits of bronze and silver.

Common Decision Mistakes

Mistake 1: "We're on Databricks, so we must use medallion"

Databricks is a platform. Medallion is a pattern. You can run any of these patterns on Databricks. The platform doesn't mandate the architecture.

Mistake 2: "We'll start with all three layers and simplify later"

Starting with unnecessary complexity is expensive. It's easier to add a layer when you have a concrete need than to remove one that's woven into your pipelines.

Better approach: Start with ELT-Only. When you feel the pain of lost raw data or inconsistent cleaning, add bronze. When consumers need different shapes, add gold.

Mistake 3: "Data Vault is more rigorous, so it must be better"

Data Vault is more rigorous. It's also significantly more complex to implement and maintain. Unless you have specific requirements around relationship-level auditing and very frequent source system changes, medallion gives you 80% of the benefit at 40% of the cost.

Mistake 4: "Streaming-first is the future, so let's go all-in"

Streaming adds operational complexity: checkpoint management, state store tuning, exactly-once semantics, watermark configuration. If your consumers are happy with hourly or daily data, the complexity isn't justified.

Mistake 5: "We need all data in one big table for performance"

OBT optimizes read performance at the cost of write complexity and storage. With Delta Lake's Z-ORDER and liquid clustering, you can get excellent read performance from normalized silver tables without the maintenance burden of massive denormalized tables.

Real-World Decision Examples

Example A: Mid-Size SaaS Company

Context: 5 source systems (product database, billing system, user events, support tickets, marketing platform). Team of 4 data engineers, 3 analysts. Data refreshed every 30 minutes.

Decision: Medallion. Multiple sources with varying quality, mixed consumer types, and a team large enough to maintain three layers.

Example B: IoT Startup

Context: 1 source (device telemetry via message queue). Team of 2 engineers. Need real-time anomaly detection AND daily reports.

Decision: Hybrid — Streaming + simplified medallion. Streaming hot path for real-time detection. Bronze + gold (skip silver, since telemetry from one source doesn't need heavy cleaning) for daily aggregates.

Example C: Regulated Financial Institution

Context: 20+ source systems. Team of 15. Heavy SOX and regulatory reporting requirements. Complex entity relationships across systems.

Decision: Medallion with Data Vault silver layer. Bronze for raw preservation (regulatory requirement). Silver as Data Vault for relationship-level auditing. Gold as reporting marts. Unity Catalog for lineage.

Example D: Solo Analyst with a Dashboard

Context: 1 PostgreSQL database. 1 analyst. Needs a BI dashboard with daily refresh.

Decision: ELT-Only. Extract from Postgres, transform directly into dashboard-ready tables. Adding bronze/silver/gold triples the maintenance burden with zero benefit.

Example E: E-Commerce Platform

Context: 8 source systems (orders, inventory, payments, shipping, customers, reviews, ads, web analytics). Team of 6. Need both real-time inventory updates and weekly business reviews.

Decision: Medallion + streaming hot path. Full medallion for the analytical platform. Separate streaming pipeline for real-time inventory that feeds directly into an operational serving layer.

Evaluating Your Situation

Use this scorecard to quantify your decision:

Interpretation:

• 5-10: Start with ELT-Only or OBT. Add layers when pain emerges.
• 11-17: Medallion is a strong fit. Proceed with confidence.
• 18-25: Medallion is essential. Consider Data Vault hybrid for the silver layer if relationship complexity is high.

Key Takeaways

1. Medallion is not always the answer — it's the answer for a specific set of conditions

2. Team size matters as much as technical requirements — a pattern nobody can maintain is worse than no pattern

3. Hybrid approaches are legitimate — don't force everything through three layers

4. Start simple, add layers when pain emerges — YAGNI applies to data architecture too

5. Revisit the decision periodically — as your data landscape evolves, your architecture should too

*Next: Chapter 3 — Bronze Layer Deep-Dive*