← Back to all products
$29
Airflow DAG Templates
Production-ready Airflow DAG templates for modern data pipelines with error handling and monitoring.
JSONMarkdownPythonYAMLAWSDatabricksSparkDelta LakeRedisPostgreSQL
📄 Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample📁 File Structure 29 files
airflow-dag-templates/
├── LICENSE
├── README.md
├── configs/
│ ├── connections.yaml
│ ├── dag_factory_config.yaml
│ └── variables.yaml
├── dags/
│ ├── api_ingestion_dag.py
│ ├── backfill_dag.py
│ ├── cdc_streaming_dag.py
│ ├── cleanup_dag.py
│ ├── cross_dag_dependency_dag.py
│ ├── data_quality_dag.py
│ ├── data_warehouse_load_dag.py
│ ├── database_replication_dag.py
│ ├── dbt_orchestration_dag.py
│ ├── dynamic_task_mapping_dag.py
│ ├── etl_pipeline_dag.py
│ ├── file_sensor_dag.py
│ ├── ml_pipeline_dag.py
│ └── sla_monitoring_dag.py
├── guides/
│ └── airflow-best-practices.md
├── plugins/
│ ├── hooks/
│ │ ├── slack_webhook_hook.py
│ │ └── teams_webhook_hook.py
│ ├── operators/
│ │ ├── data_quality_operator.py
│ │ ├── databricks_notebook_operator.py
│ │ ├── delta_table_sensor.py
│ │ └── spark_submit_operator.py
│ └── sensors/
│ └── s3_key_sensor_extended.py
└── tests/
└── test_dags.py
📖 Documentation Preview README excerpt
Airflow DAG Templates
Production-ready Apache Airflow DAG templates for modern data pipelines.
Skip the boilerplate. Start with 14 battle-tested DAGs covering ETL, data quality, ML pipelines, warehouse loading, CDC streaming, database replication, SLA monitoring, dynamic task mapping, and more.
What You Get
| Category | Count | Highlights |
|---|---|---|
| DAG templates | 14 | ETL, ML, CDC, dbt, warehouse, replication, SLA, cleanup |
| Custom operators | 4 | Spark submit, data quality, Databricks notebook, Delta sensor |
| Custom hooks | 2 | Slack webhook, Microsoft Teams webhook |
| Custom sensors | 1 | Extended S3 key sensor with size/age checks |
| Config files | 3 | Connections, variables, DAG factory config |
| Test suite | 1 | DAG integrity, plugin validation, config checks |
| Best practices guide | 1 | 450+ lines covering TaskFlow, dynamic mapping, SLAs |
Total: 29 files, 7,000+ lines of production-quality code and documentation.
File Tree
airflow-dag-templates/
├── README.md # This file
├── manifest.json # Product manifest
├── LICENSE # MIT License
│
├── dags/ # 14 production DAG templates
│ ├── etl_pipeline_dag.py # Full ETL: extract → transform → quality → load
│ ├── data_quality_dag.py # Data quality checks with Slack alerts
│ ├── dbt_orchestration_dag.py # dbt run → test → docs generation
│ ├── api_ingestion_dag.py # REST API → S3 → Spark processing
│ ├── cdc_streaming_dag.py # Debezium → Kafka → Delta Lake CDC
│ ├── backfill_dag.py # Parameterized historical backfill
│ ├── file_sensor_dag.py # S3 file watching with format routing
│ ├── data_warehouse_load_dag.py # Multi-target warehouse loading (Snowflake/Redshift)
│ ├── ml_pipeline_dag.py # ML training with evaluation gates
│ ├── database_replication_dag.py # Full & incremental database replication
│ ├── sla_monitoring_dag.py # SLA checks with tiered alerting
│ ├── cleanup_dag.py # Table cleanup, temp files, retention policies
│ ├── cross_dag_dependency_dag.py # ExternalTaskSensor & TriggerDagRun patterns
│ └── dynamic_task_mapping_dag.py # Airflow 2.3+ dynamic task mapping
│
├── plugins/ # Custom Airflow plugins
│ ├── operators/
│ │ ├── spark_submit_operator.py # Databricks Spark job submission
│ │ ├── data_quality_operator.py # Configurable data quality checks
│ │ ├── databricks_notebook_operator.py # Databricks notebook execution
│ │ └── delta_table_sensor.py # Delta Lake table freshness sensor
│ ├── hooks/
│ │ ├── slack_webhook_hook.py # Slack webhook notifications
│ │ └── teams_webhook_hook.py # Microsoft Teams webhook notifications
│ └── sensors/
│ └── s3_key_sensor_extended.py # S3 sensor with size/age validation
│
*... continues with setup instructions, usage examples, and more.*
📄 Code Sample .py preview
dags/api_ingestion_dag.py
"""
API Ingestion DAG — Airflow DAG Templates
============================================
Ingests data from a paginated REST API, stages to S3, and triggers
Spark processing. Handles rate limiting and pagination automatically.
By Datanest Digital — https://datanest.dev
"""
from __future__ import annotations
import json
import time
from datetime import datetime, timedelta
from typing import Any, Dict, List, Optional
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.empty import EmptyOperator
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
DEFAULT_ARGS: Dict[str, Any] = {
"owner": "data-engineering",
"depends_on_past": False,
"retries": 3,
"retry_delay": timedelta(minutes=5),
"retry_exponential_backoff": True,
"execution_timeout": timedelta(hours=2),
}
API_CONFIG: Dict[str, Any] = {
"base_url": "https://api.example.com/v2",
"endpoint": "/events",
"auth_conn_id": "api_auth",
"page_size": 100,
"max_pages": 1000,
"rate_limit_delay": 0.5,
"s3_bucket": "my-data-lake",
"s3_prefix": "raw/api_events/",
}
# ---------------------------------------------------------------------------
# Task callables
# ---------------------------------------------------------------------------
def fetch_api_pages(**context: Any) -> Dict[str, Any]:
"""
Fetch all pages from the REST API and return metadata.
# ... 110 more lines ...