Overview

The Multi-Cloud Lakehouse Blueprint is a reference implementation for Databricks across Microsoft Azure and Amazon Web Services. It combines cloud-native storage and networking with consistent governance and delivery. It gives teams common controls, portable data interfaces, and tested recovery paths.

Architecture at a Glance

Azure hosts a VNet-injected Databricks workspace with Private Link access to ADLS Gen2. AWS hosts a workspace in a private VPC, with PrivateLink, controlled NAT egress, and Delta tables on S3. Compute remains close to data: Azure clusters read Azure storage, and AWS clusters read S3. Cross-cloud raw-data movement adds latency, egress cost, and failure risk.

Unity Catalog governs catalogs, schemas, tables, permissions, and auditing. Where metastore sharing is unsuitable, Delta Sharing publishes governed products across the boundary. Azure AD and AWS IAM identities are federated through the corporate identity provider and provisioned with SCIM.

Reference Architecture in Text

Read the reference diagram top to bottom. Users and service principals enter through the identity provider. The governance plane maps groups to Unity Catalog permissions and records access. Two private execution zones follow: Azure Databricks in a VNet and AWS Databricks in a VPC. Each contains job compute, private endpoints, monitoring, and local Delta storage—ADLS Gen2 or S3. Delta Sharing connects selected datasets without exposing storage accounts. VPN or dedicated ExpressRoute and Direct Connect paths carry approved traffic. CI/CD deploys both zones through separate credentials and approval gates.

Key Design Decisions

Storage: keep a system of record for each dataset and document residency. Replicate only products required for recovery or cross-cloud consumption. Use Delta format in both clouds, encryption with cloud-native keys, lifecycle policies, and explicit recovery-point objectives.

Compute: use independent, autoscaling Databricks compute per cloud. Standardize runtime versions, cluster policies, tags, and job definitions, but choose provider-specific instance families based on workload and price. Avoid permanently active duplicate compute unless the recovery objective requires active-active service.

Catalog: design catalog boundaries around environments, domains, and data classifications—not cloud names alone. Assign workspaces to the appropriate Unity Catalog metastore, define ownership centrally, and expose cross-cloud products through governed shares.

Governance: federate identity, apply least privilege, use private networking, centralize audit export, and test data-residency controls. A common policy must still map to Azure Policy, AWS controls, key management, logging, and regional requirements.

Multi-Cloud or Single-Cloud?

Choose multi-cloud when regulation requires provider separation, an acquisition has durable workloads in both ecosystems, customers demand cloud-local delivery, or the business funds cross-cloud disaster recovery. It can support a data mesh already spanning clouds.

Prefer single-cloud when the motivation is only negotiating leverage, hypothetical resilience, or fear of lock-in. A second cloud doubles skills, identity paths, networking, security evidence, observability, and incident procedures. If one provider meets residency, availability, and commercial needs, multi-region deployment within that provider is usually simpler and less expensive.

What the Blueprint Covers

The package includes Azure and AWS Terraform modules, private network patterns, identity federation guidance, Unity Catalog and Delta Sharing configuration, a unified Azure DevOps deployment pipeline, compliance mapping, migration guidance, architecture decision records, a DBU cost comparison tool, and active-passive disaster-recovery runbooks. Use the ADRs to confirm the business case, deploy isolated foundations, configure governance, publish selected shares, and rehearse failover.

The blueprint accelerates infrastructure and operating-model design; it does not remove organization-specific work. You must still choose regions, identity ownership, recovery objectives, data classifications, cost limits, and accountable service owners before production deployment.

What's Included

Follow this guide to get Multi-Cloud Lakehouse Blueprint up and running in your environment.

What's Included

Documentation

Terraform Modules

CI/CD

Tools

Disaster Recovery

Architecture Overview

┌─────────────────────────────────────────────────────────────────────┐
│                      Unity Catalog (Metastore)                      │
│                   Cross-Cloud Governance Layer                      │
├────────────────────────────┬────────────────────────────────────────┤
│                            │                                        │
│    ┌───────────────────┐   │   ┌───────────────────┐               │
│    │  Azure Databricks │   │   │  AWS Databricks   │               │
│    │  Workspace        │   │   │  Workspace        │               │
│    │                   │   │   │                   │               │
│    │  ┌─────────────┐  │   │   │  ┌─────────────┐  │               │
│    │  │ Compute     │  │   │   │  │ Compute     │  │               │
│    │  │ Clusters    │  │   │   │  │ Clusters    │  │               │
│    │  └─────────────┘  │   │   │  └─────────────┘  │               │
│    └────────┬──────────┘   │   └────────┬──────────┘               │
│             │              │            │                           │
│    ┌────────▼──────────┐   │   ┌────────▼──────────┐               │
│    │  ADLS Gen2        │   │   │  S3 + Glue        │               │
│    │  (Delta Lake)     │◄──┼──►│  (Delta Lake)     │               │
│    └───────────────────┘   │   └───────────────────┘               │
│         Delta Sharing      │        Delta Sharing                   │
│                            │                                        │
├────────────────────────────┴────────────────────────────────────────┤
│                    Cross-Cloud Network Fabric                       │
│              (VPN Gateway / ExpressRoute / Direct Connect)          │
└─────────────────────────────────────────────────────────────────────┘