Apache Airflow

Apache Airflow is the industry-standard workflow orchestration platform. It allows you to programmatically author, schedule, and monitor workflows as Directed Acyclic Graphs (DAGs). Originally developed at Airbnb in 2014, it became an Apache top-level project and is now maintained by the community with commercial support from Astronomer, Google, and AWS.

Core Concepts

DAG (Directed Acyclic Graph)

A DAG is a collection of tasks with defined dependencies. The DAG defines the order of execution and relationships between tasks. Airflow executes tasks in the order specified by the DAG, respecting upstream and downstream dependencies.

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.bash import BashOperator

default_args = {
    "owner": "kiersten",
    "depends_on_past": False,
    "retries": 2,
    "retry_delay": timedelta(minutes=5),
}

with DAG(
    dag_id="daily_data_pipeline",
    start_date=datetime(2025, 1, 1),
    schedule="@daily",
    catchup=False,
    default_args=default_args,
    description="Extract, transform, load daily orders",
) as dag:

    extract = BashOperator(
        task_id="extract_orders",
        bash_command="python /scripts/extract.py --date {{ ds }}",
    )

    transform = PythonOperator(
        task_id="transform_orders",
        python_callable=transform_fn,
        op_kwargs={"execution_date": "{{ ds }}"},
    )

    load = BashOperator(
        task_id="load_to_warehouse",
        bash_command="dbt run --models +fct_orders --vars '{"date": "{{ ds }}"}'",
    )

    extract >> transform >> load

Operators

Operators define what a single task does. Airflow ships with dozens of built-in operators and the community provides hundreds more:

• PythonOperator - Execute a Python callable
• BashOperator - Run a bash command
• PostgresOperator - Execute SQL on Postgres
• S3CopyObjectOperator - Copy objects in S3
• SnowflakeOperator - Run queries on Snowflake
• KubernetesPodOperator - Run a task as a K8s pod
• BranchPythonOperator - Branch execution based on conditions

Sensors

Sensors are a special kind of operator that wait for a condition to be met before proceeding. Common sensors:

• S3KeySensor - Wait for a file to land in S3
• ExternalTaskSensor - Wait for another DAG's task to complete
• SqlSensor - Wait for a SQL query to return results

Airflow Architecture

Airflow consists of several components running together:

• Scheduler - Continuously scans DAGs, triggers tasks, and manages dependencies. The scheduler is the brain of Airflow.
• Web Server - The UI for viewing DAGs, task logs, and triggering manual runs.
• Worker - Executes tasks. Workers can run on Celery, Kubernetes, or local executors.
• Database - Stores DAG metadata, task states, and connection info. Typically Postgres or MySQL.
• Executor - Defines how tasks are run. Options: LocalExecutor, CeleryExecutor, KubernetesExecutor, SequentialExecutor.

Best Practices

• Keep DAGs idempotent - running the same DAG twice should produce identical results
• Use catchup=False unless you specifically need backfill behavior
• Avoid putting heavy computation in PythonOperators; use KubernetesPodOperator for resource-intensive work
• Set meaningful retries and retry delays for transient failures
• Use Airflow variables and connections for configuration, never hard-code secrets
• Break large DAGs into focused, manageable pieces - one DAG per domain
• Add SLA monitoring for critical data delivery deadlines

Resources

• Apache Airflow Documentation
• Astronomer Documentation
• Airflow Summit Talks