← Back to all products
$49
Cloud Data Platform Blueprint
End-to-end cloud data platform designs: ingestion, storage, processing, serving, and governance layers.
JSONPythonTerraformSQLYAMLMarkdownAWSSpark
📄 Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample📁 File Structure 22 files
cloud-data-platform-blueprint/
├── LICENSE
├── README.md
├── cloudformation/
│ ├── batch-processing.yaml
│ ├── data-lake-storage.yaml
│ ├── serving-layer.yaml
│ └── streaming-ingestion.yaml
├── docs/
│ ├── architecture.md
│ ├── data-governance-framework.md
│ └── scaling-guide.md
├── examples/
│ ├── athena-queries.sql
│ ├── glue-etl-job.py
│ └── lake-formation-permissions.json
├── free-sample.zip
├── guide/
│ ├── 01_overview.md
│ ├── 02_prerequisites.md
│ └── 03_faq.md
├── index.html
├── scripts/
│ ├── cost_estimator.py
│ └── data_quality_validator.py
└── terraform/
├── data-lake-foundation.tf
├── governance-layer.tf
└── variables.tf
📖 Documentation Preview README excerpt
Cloud Data Platform Blueprint
End-to-end cloud data platform designs covering ingestion, storage, processing, serving, and governance layers — with production-ready IaC templates for AWS.
Overview
This blueprint provides a complete reference architecture and deployable infrastructure for building a modern cloud data platform. It follows the medallion architecture (Bronze/Silver/Gold) with real-time and batch processing paths, comprehensive data governance, and a self-service analytics layer.
Whether you're building a new data platform from scratch or modernizing a legacy data warehouse, this blueprint gives you the IaC templates, processing patterns, and governance framework to deliver production-quality data infrastructure.
What's Included
| File | Description |
|---|---|
| Infrastructure as Code | |
cloudformation/data-lake-storage.yaml | S3 data lake with medallion zones, lifecycle policies, encryption |
cloudformation/streaming-ingestion.yaml | Kinesis Data Streams + Firehose for real-time ingestion |
cloudformation/batch-processing.yaml | Glue ETL jobs, crawlers, and job bookmarks for batch pipelines |
cloudformation/serving-layer.yaml | Athena workgroups, Redshift Serverless, QuickSight config |
terraform/data-lake-foundation.tf | Complete data lake foundation in Terraform |
terraform/governance-layer.tf | Lake Formation permissions, data catalog, audit logging |
terraform/variables.tf | Shared variables for all Terraform configs |
| Documentation | |
docs/architecture.md | Full reference architecture with Mermaid diagrams |
docs/data-governance-framework.md | Data classification, access control, lineage, quality |
docs/scaling-guide.md | Scaling strategies for each layer with cost benchmarks |
| Scripts | |
scripts/data_quality_validator.py | Data quality rules engine with 20+ built-in checks |
scripts/cost_estimator.py | Data platform cost modeling tool |
| Examples | |
examples/glue-etl-job.py | Sample Glue ETL job (Bronze → Silver transformation) |
examples/athena-queries.sql | Common analytical queries for the Gold layer |
examples/lake-formation-permissions.json | Example Lake Formation permission grants |
LICENSE | MIT License |
Architecture at a Glance
┌─────────────────────────────────────────────────────────────┐
│ DATA SOURCES │
│ Databases │ APIs │ IoT Streams │ Files │ SaaS │
└──────┬──────┴───┬────┴──────┬────────┴────┬────┴─────┬─────┘
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
┌─────────────────────────────────────────────────────────────┐
│ INGESTION LAYER │
│ Kinesis Streams │ Kinesis Firehose │ AWS DMS │ SFTP │
└──────────────────────────┬──────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ STORAGE LAYER (S3) │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ BRONZE │→│ SILVER │→│ GOLD │ │
│ │ (Raw) │ │(Cleansed) │ │(Business) │ │
│ └──────────┘ └──────────┘ └──────────┘ │
└──────────────────────────┬──────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
*... continues with setup instructions, usage examples, and more.*
📄 Code Sample .py preview
examples/glue-etl-job.py
"""
Sample AWS Glue ETL Job — Bronze to Silver Transformation
This script demonstrates a complete Bronze-to-Silver ETL transformation
using the AWS Glue PySpark runtime. It reads raw JSON data from the Bronze
zone, applies data quality rules, deduplicates, conforms the schema, and
writes optimized Parquet to the Silver zone.
This is a TEMPLATE — adapt the schema, transformations, and quality rules
to your specific datasets.
Usage:
Deploy to Glue via the included CloudFormation/Terraform templates.
The job reads parameters from Glue job arguments (--bronze-database, etc.)
Note: This file runs in the AWS Glue runtime, which pre-installs PySpark
and the awsglue library. It is NOT meant to be run locally with pip.
"""
import sys
from datetime import datetime, timezone
# -----------------------------------------------------------------------
# AWS Glue imports (available in Glue runtime, not locally)
# These imports will show as unresolved in local IDEs — that's expected.
# -----------------------------------------------------------------------
from awsglue.context import GlueContext
from awsglue.dynamicframe import DynamicFrame
from awsglue.job import Job
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from pyspark.sql import functions as F
from pyspark.sql.types import (
DoubleType,
IntegerType,
StringType,
StructField,
StructType,
TimestampType,
)
# -----------------------------------------------------------------------
# Job initialization
# -----------------------------------------------------------------------
# Parse job arguments passed via CloudFormation/Terraform
args = getResolvedOptions(sys.argv, [
"JOB_NAME",
"bronze-database",
# ... 246 more lines ...