This chapter covers the core features and capabilities of Cloud Data Platform Blueprint.
This blueprint provides a complete reference architecture and deployable infrastructure for building a modern cloud data platform. It follows the medallion architecture (Bronze/Silver/Gold) with real-time and batch processing paths, comprehensive data governance, and a self-service analytics layer.
Whether you're building a new data platform from scratch or modernizing a legacy data warehouse, this blueprint gives you the IaC templates, processing patterns, and governance framework to deliver production-quality data infrastructure.
| File | Description |
|---|---|
| Infrastructure as Code | |
cloudformation/data-lake-storage.yaml | S3 data lake with medallion zones, lifecycle policies, encryption |
cloudformation/streaming-ingestion.yaml | Kinesis Data Streams + Firehose for real-time ingestion |
cloudformation/batch-processing.yaml | Glue ETL jobs, crawlers, and job bookmarks for batch pipelines |
cloudformation/serving-layer.yaml | Athena workgroups, Redshift Serverless, QuickSight config |
terraform/data-lake-foundation.tf | Complete data lake foundation in Terraform |
terraform/governance-layer.tf | Lake Formation permissions, data catalog, audit logging |
terraform/variables.tf | Shared variables for all Terraform configs |
| Documentation | |
docs/architecture.md | Full reference architecture with Mermaid diagrams |
docs/data-governance-framework.md | Data classification, access control, lineage, quality |
docs/scaling-guide.md | Scaling strategies for each layer with cost benchmarks |
| Scripts | |
scripts/data_quality_validator.py | Data quality rules engine with 20+ built-in checks |
scripts/cost_estimator.py | Data platform cost modeling tool |
| Examples | |
examples/glue-etl-job.py | Sample Glue ETL job (Bronze → Silver transformation) |
examples/athena-queries.sql | Common analytical queries for the Gold layer |
examples/lake-formation-permissions.json | Example Lake Formation permission grants |
LICENSE | MIT License |
┌─────────────────────────────────────────────────────────────┐
│ DATA SOURCES │
│ Databases │ APIs │ IoT Streams │ Files │ SaaS │
└──────┬──────┴───┬────┴──────┬────────┴────┬────┴─────┬─────┘
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
┌─────────────────────────────────────────────────────────────┐
│ INGESTION LAYER │
│ Kinesis Streams │ Kinesis Firehose │ AWS DMS │ SFTP │
└──────────────────────────┬──────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ STORAGE LAYER (S3) │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ BRONZE │→│ SILVER │→│ GOLD │ │
│ │ (Raw) │ │(Cleansed) │ │(Business) │ │
│ └──────────┘ └──────────┘ └──────────┘ │
└──────────────────────────┬──────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ PROCESSING LAYER │
│ Glue ETL (batch) │ Glue Streaming │ EMR (heavy) │
└──────────────────────────┬──────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ SERVING LAYER │
│ Athena (ad-hoc) │ Redshift (BI) │ QuickSight (viz) │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ GOVERNANCE LAYER │
│ Lake Formation │ Glue Catalog │ CloudTrail │ Macie │
└─────────────────────────────────────────────────────────────┘Follow this guide to get Cloud Data Platform Blueprint up and running in your environment.
aws cloudformation deploy \
--template-file cloudformation/data-lake-storage.yaml \
--stack-name data-lake-storage \
--parameter-overrides Environment=production OrganizationPrefix=acme \
--capabilities CAPABILITY_NAMED_IAMaws cloudformation deploy \
--template-file cloudformation/streaming-ingestion.yaml \
--stack-name data-lake-streaming \
--parameter-overrides DataLakeBucketName=acme-data-lake-productionaws cloudformation deploy \
--template-file cloudformation/batch-processing.yaml \
--stack-name data-lake-batch \
--parameter-overrides DataLakeBucketName=acme-data-lake-productionaws cloudformation deploy \
--template-file cloudformation/serving-layer.yaml \
--stack-name data-lake-serving \
--capabilities CAPABILITY_NAMED_IAMpython3 scripts/data_quality_validator.py \
--rules examples/quality-rules.json \
--report quality-report.json| Component | Monthly Cost (1 TB data, moderate queries) |
|---|---|
| S3 Storage (3 zones) | ~$30-50 |
| Kinesis Data Streams (1 shard) | ~$15 |
| Kinesis Firehose | ~$10-30 (per GB delivered) |
| Glue ETL (10 DPU-hours/day) | ~$130 |
| Glue Catalog | ~$1-5 |
| Athena Queries | ~$25-100 (per TB scanned) |
| Redshift Serverless (8 RPU) | ~$200-500 |
| Lake Formation | Free (included) |
| Total | ~$400-800/month |
Cost optimization tips:
Get the full Cloud Data Platform Blueprint and unlock everything.
Get the complete guide with every chapter unlocked, including code samples, diagrams, and best practices.
Access all interactive tools with complete data, all workload profiles, and the full scenario library.
Downloadable source code, configuration files, and working examples from every chapter.
Free updates for life. Every new chapter, tool, and improvement included.