The ingestion layer handles data acquisition from all source systems. It supports both real-time streaming and batch loading patterns.

graph LR
    subgraph "Real-Time Path"
        Producer[Data Producer] -->|PUT Record| KDS[Kinesis Data Streams]
        KDS --> KDF[Kinesis Firehose]
        KDF -->|Parquet + Partition| S3[S3 Bronze]
    end

    subgraph "Batch Path"
        Source[(Source DB)] --> DMS[AWS DMS]
        DMS -->|Full/CDC| S3
        Upload[File Upload] -->|S3 API| S3
    end

Pattern	Use When	Latency	Cost Profile
Kinesis Streams + Firehose	Continuous event data, IoT telemetry	Sub-minute	Per-shard + per-GB
AWS DMS	Database replication (full load + CDC)	Minutes	Per-instance-hour
S3 Direct Upload	File drops, batch exports	Hours	Per-request only
AWS Transfer Family	Partner SFTP/FTPS file drops	Hours	Per-protocol + transfer

Design decisions:

Kinesis over Kafka: Kinesis is fully managed, scales automatically, and integrates natively with Firehose for S3 delivery. Use MSK (Kafka) only if you have existing Kafka producers or need Kafka-specific features.
Firehose Parquet conversion: Converting to Parquet at ingestion time (even in Bronze) reduces downstream Athena costs by 90%. However, this requires a schema upfront — for schema-agnostic ingestion, land as JSON and convert in the Bronze→Silver ETL.
Dynamic partitioning: Firehose partitions by year/month/day/hour automatically. This is critical for Athena query performance — without partitions, every query scans the entire dataset.

Included with Cloud Data Platform Blueprint

Ready for the complete Cloud Data Platform Blueprint?