Airflow DAG Templates

Airflow DAG Templates

Production-ready Apache Airflow DAG templates for modern data pipelines.

Skip the boilerplate. Start with 14 battle-tested DAGs covering ETL, data quality, ML pipelines, warehouse loading, CDC streaming, database replication, SLA monitoring, dynamic task mapping, and more.

What You Get

Total: 29 files, 7,000+ lines of production-quality code and documentation.

File Tree

airflow-dag-templates/
├── README.md                           # This file
├── manifest.json                       # Product manifest
├── LICENSE                             # MIT License
│
├── dags/                               # 14 production DAG templates
│   ├── etl_pipeline_dag.py             # Full ETL: extract → transform → quality → load
│   ├── data_quality_dag.py             # Data quality checks with Slack alerts
│   ├── dbt_orchestration_dag.py        # dbt run → test → docs generation
│   ├── api_ingestion_dag.py            # REST API → S3 → Spark processing
│   ├── cdc_streaming_dag.py            # Debezium → Kafka → Delta Lake CDC
│   ├── backfill_dag.py                 # Parameterized historical backfill
│   ├── file_sensor_dag.py              # S3 file watching with format routing
│   ├── data_warehouse_load_dag.py      # Multi-target warehouse loading (Snowflake/Redshift)
│   ├── ml_pipeline_dag.py              # ML training with evaluation gates
│   ├── database_replication_dag.py     # Full & incremental database replication
│   ├── sla_monitoring_dag.py           # SLA checks with tiered alerting
│   ├── cleanup_dag.py                  # Table cleanup, temp files, retention policies
│   ├── cross_dag_dependency_dag.py     # ExternalTaskSensor & TriggerDagRun patterns
│   └── dynamic_task_mapping_dag.py     # Airflow 2.3+ dynamic task mapping
│
├── plugins/                            # Custom Airflow plugins
│   ├── operators/
│   │   ├── spark_submit_operator.py    # Databricks Spark job submission
│   │   ├── data_quality_operator.py    # Configurable data quality checks
│   │   ├── databricks_notebook_operator.py  # Databricks notebook execution
│   │   └── delta_table_sensor.py       # Delta Lake table freshness sensor
│   ├── hooks/
│   │   ├── slack_webhook_hook.py       # Slack webhook notifications
│   │   └── teams_webhook_hook.py       # Microsoft Teams webhook notifications
│   └── sensors/
│       └── s3_key_sensor_extended.py   # S3 sensor with size/age validation
│
├── configs/                            # Configuration templates
│   ├── connections.yaml                # Airflow connection definitions
│   ├── variables.yaml                  # Airflow variable definitions
│   └── dag_factory_config.yaml         # Config-driven DAG generation
│
├── tests/                              # Test suite
│   └── test_dags.py                    # DAG integrity & plugin tests
│
└── guides/                             # Documentation
    └── airflow-best-practices.md       # Comprehensive best practices guide

DAG Overview

Core Pipeline DAGs

Streaming & Replication DAGs

Advanced Pattern DAGs

Operational DAGs

Getting Started

Follow this guide to get airflow dag templates up and running in your environment.

Getting Started

1. Copy to your Airflow deployment

# Copy DAGs
cp dags/*.py $AIRFLOW_HOME/dags/

# Copy plugins (preserves directory structure)
cp -r plugins/ $AIRFLOW_HOME/plugins/

# Copy configs (for reference / import)
cp -r configs/ $AIRFLOW_HOME/configs/

2. Configure connections and variables

Use the YAML templates in configs/ to set up connections:

# Import connections
airflow connections import configs/connections.yaml

# Import variables
airflow variables import configs/variables.yaml

Or configure manually via the Airflow UI under Admin → Connections and Admin → Variables.

3. Customize the templates

Each DAG has clearly marked # TODO: sections for project-specific customization:

# Example: customize the ETL DAG
DAG_CONFIG = {
    "source_table": "raw.events",          # TODO: your source table
    "target_table": "analytics.events_clean",  # TODO: your target table
    "schedule": "@daily",
    "owner": "data-team",
}

4. Choose which DAGs to deploy

You don't need all 14 DAGs. Pick the ones that match your use case:

• Starting a new data platform? → etl_pipeline_dag + data_quality_dag + cleanup_dag
• Adding dbt? → dbt_orchestration_dag
• Ingesting APIs? → api_ingestion_dag + file_sensor_dag
• Building ML pipelines? → ml_pipeline_dag
• Need CDC/replication? → cdc_streaming_dag + database_replication_dag
• Multi-DAG orchestration? → cross_dag_dependency_dag + sla_monitoring_dag
• Learning Airflow 2.3+? → dynamic_task_mapping_dag

Plugin Reference

Operators

Hooks

Sensors

Configuration Reference

configs/connections.yaml

Pre-configured connection templates for common data infrastructure:

• AWS S3 / Redshift
• Databricks workspace
• Snowflake
• PostgreSQL / MySQL
• Slack / Teams webhooks
• Kafka / Schema Registry

configs/variables.yaml

Environment-aware variable templates:

• Pipeline configuration (tables, schedules, thresholds)
• Notification settings (channels, severity levels)
• Feature flags (enable/disable DAG features)

configs/dag_factory_config.yaml

Configuration-driven DAG generation for teams managing many similar pipelines:

• Define pipelines in YAML instead of writing Python
• Supports schedule, source/target, quality checks, notifications
• Generates fully functional DAGs at import time