Data Engineer · Step 6 of 8 · 8 min
Orchestration and DAGs
Understand how pipelines are scheduled, ordered, retried and monitored.
Why you need an orchestrator
One script on a timer works until you have twenty. Then tasks depend on each other, some fail, some need rerunning for last Tuesday, and you need to know what ran and when. An orchestrator handles this.
Apache Airflow is the best known, and Airflow 3 is the current major version. Dagster and Prefect are popular alternatives, and cloud providers offer managed versions.
The DAG
Orchestrators describe work as a DAG, a directed acyclic graph. Directed means each arrow points one way: extract before transform. Acyclic means there are no loops, so the work always ends.
Each box in the graph is a task. The orchestrator runs a task when the tasks before it succeed, and can run independent tasks in parallel.
Retries, backfills and alerts
Networks fail and APIs time out, so tasks should retry a few times with a pause. A backfill reruns a pipeline for past dates, for example after fixing a bug. Backfills only work safely if your tasks are idempotent and take the date as an input instead of using today's date.
Alerts tell a person when something fails for good. Send them where people actually look, and keep them few, or they will be ignored.
Keep tasks small and plain
Let the orchestrator schedule and track work. Keep heavy processing in the warehouse or a processing engine, not inside the orchestrator itself. Small tasks with clear inputs and outputs are easier to retry and to debug.
Try it
Draw and review a DAG
- 1.List the tasks for your pipeline from step 3, one per line.
- 2.Draw arrows for which task must finish before another starts.
- 3.Ask a chatbot the prompt below and compare its DAG with yours.
Here are the tasks in my daily pipeline: [list]. Arrange them as a DAG, show which can run in parallel, and say for each task how many retries it should have and whether it is safe to backfill. Then show a minimal Apache Airflow 3 DAG in Python, and tell me which parts I should check against the official docs.