Data orchestration decides what runs, in what order, under what conditions, and what happens when a step fails. It turns a set of independent jobs into a pipeline somebody can reason about.
Teams often acquire orchestration by accident. One scheduled script becomes three. Someone adds a delay so the second does not start before the first finishes. Within a year the real execution order lives in a spreadsheet and two people's heads.
The cost surfaces at recovery time. When a source is late, the question is not only which job failed. It is what ran on bad inputs afterwards, what has to be rebuilt, and in what order. A system that cannot answer that is scheduled, not orchestrated.
Apache Airflow's documentation states the model directly: a DAG encapsulates a workflow, its schedule, its tasks and the dependencies between them. See the Airflow DAG concepts guide.
- Scheduling starts jobs at a time; orchestration runs a dependency graph with state and recovery.
- Idempotent tasks and explicit run windows are what make retries and backfills safe.
- Time-driven, event-driven and data-aware triggering solve different problems and often coexist.
- Design recovery before throughput: the question is always what to rebuild, and in what order.
- Ownership boundaries matter as much as the graph once more than one team ships pipelines.
A Worked Example: Meridian's Nightly Graph
The same retailer runs through this article. Meridian Retail loads about 4 million orders a month from 12 marketplace regions, a Postgres order system, a payments feed and clickstream events. The nightly window runs 01:00 to 03:00. Month-end close lands on working day three.
Region files arrive between 00:20 and 01:40. One region is routinely late. The finance mart also depends on an FX rate feed that refreshes at 00:45.
A scheduler handled this for two years. Each job had a start time chosen to sit after the previous one. The padding between those times was never written down anywhere.
- The failure An FX feed outage delayed the rate refresh to 01:20. The finance mart job started at 01:00 on stale rates and completed successfully.
- The detection Nobody noticed for two days. Finance queried a margin figure that looked wrong during close preparation.
- The cost Three downstream models had to be rebuilt for a 30-day range, during close week, under time pressure.
- The fix Move the mart from a start time to a dependency on the FX feed's output. The job now waits instead of running on stale input.
That change is the difference between scheduling and orchestration in one pipeline. The rest of this article generalises it.
Start times encode dependencies nobody wrote down, and the encoding fails silently.
What Data Orchestration Actually Does
Orchestration is responsible for execution, not transformation. It decides when work becomes eligible, enforces the order implied by dependencies, tracks run state, and decides what happens on failure.
- Resolve dependencies so a task starts only when the data it needs is present.
- Maintain run state, so a partial failure is distinguishable from a run that never started.
- Apply retry, timeout and alerting policy consistently rather than per script.
- Support backfills over a historical window without corrupting current results.
- Expose execution lineage: which run produced which output, from which inputs.
Execution lineage is distinct from data lineage. When a figure is disputed, data lineage says which columns fed it. Execution lineage says which run produced it, on which code version, from which input partition. Resolving a dispute usually needs both.
Orchestration does not know whether the numbers a task produced are correct. It knows the task exited successfully. A transformation can complete cleanly against an empty input and report success, which is why monitoring and observability sits alongside orchestration rather than inside it.
Orchestration owns execution, state and recovery, not the logic inside each task.
Scheduling Is Not Orchestration
The upgrade from one to the other is a common data platform project, and a common one to get wrong. The table sets out what actually differs.
How time-based scheduling differs from dependency-driven orchestration
| Concern | Scheduler | Orchestrator |
|---|---|---|
| Trigger | A clock time | Upstream completion, data arrival, or a clock time |
| Ordering | Implied by chosen start times | Declared as a dependency graph |
| Run state | Exit codes in a log | Tracked per task and per run, and queryable |
| Failure handling | Retry the script, or a human intervenes | Policy-driven retries, with downstream work held back |
| Late data | The run proceeds on whatever is present | The task stays ineligible until its input exists |
| Backfill | Manual re-run with edited parameters | A first-class operation over a date range |
Meridian's FX failure is row five. The scheduler had no way to express that the mart needed the rate feed, so the mart ran on whatever was there.
A scheduler is not wrong for a small estate. Three independent daily jobs with no shared inputs do not need a dependency graph, and adding one creates operational surface for no return.
The threshold is coupling. Once one job's output is another's input, start times become an undocumented contract. The padding between them quietly becomes the system's error budget.
Move when jobs start depending on each other, not when the estate reaches some size.
Dependencies and the Execution Graph
Orchestrators model work as a directed acyclic graph. Tasks are nodes, dependencies are edges, and there are no cycles. Getting that graph right is most of the design work.
Declare data dependencies, not timing ones
A task should depend on what it consumes, not on a job that happens to finish first. Timing dependencies encode today's schedule into the graph. They break the first time a source gets slower or a step gets faster.
Where the dependency is genuinely an external arrival, express it as a sensor or an availability check rather than a delay.
Choose task granularity deliberately
Too coarse, and a failure halfway through forces a re-run of work that already succeeded. Too fine, and scheduling overhead outweighs the work. A practical rule is to split where you would want to resume.
Granularity also decides what can run in parallel. Meridian's 12 region files are 12 tasks, not one. A late region blocks only its own branch and the mart, not the other eleven.
Keep the graph readable
A graph only its author can interpret is a liability at three in the morning. Consistent naming, grouped sub-graphs per domain and a visible critical path are worth more than clever generation nobody can trace.
Dynamic generation is not wrong. For a graph fanning out over hundreds of partitions it is the only sensible option. The test is whether an on-call engineer can answer two questions quickly: what produced this output, and what breaks if this task fails. Stable task identifiers pass that test. Identifiers keyed on run time do not, because yesterday's failure no longer has a name today.
Model dependencies on data, split where you would resume, keep the graph legible.
Idempotency, Retries and Backfills
Retries are safe only if running a task twice produces the same result as running it once. That property is designed in, not configured.
The usual mechanism is an explicit run window plus a deterministic write. A task is parameterised by the period it owns. It replaces that period's output rather than appending. Re-running then overwrites the same partition with the same result.
Appending without a key breaks this. A retried append doubles a day's rows. The run eventually succeeded, so nothing in the orchestrator flags it. The failure surfaces later as a total that looks wrong to somebody in finance.
Backfills apply the same mechanism over a range. Meridian's 30-day FX rebuild was bounded because every task was window-parameterised. Rebuilding was a re-run, not a reconstruction.
Backfills need concurrency limits. A graph that handles one day comfortably can saturate a warehouse when asked for ninety at once. Meridian caps concurrent backfill tasks at four so the nightly close is never starved. Transformation tools expose the same distinction; dbt's incremental model documentation separates an incremental run from a full refresh for exactly this reason.
- Parameterise every task by the window it owns, never by wall-clock now.
- Replace the target partition rather than appending, so a retry cannot duplicate.
- Cap backfill concurrency so historical rebuilds cannot starve production runs.
- Bound retries and escalate, rather than retrying indefinitely into a broken source.
Idempotent, window-parameterised tasks turn retries and backfills from risk into routine.
Triggering Patterns
Three triggering models dominate. Mature estates usually run all three rather than picking one.
- Time-driven A task becomes eligible on a schedule. Predictable and easy to reason about, but it assumes inputs arrive before the clock does.
- Event-driven Arrival of a file, a message or a completion signal makes work eligible. Removes guesswork about upstream timing.
- Data-aware Tasks declare the datasets they produce and consume. Eligibility follows dataset freshness rather than task completion.
Data-aware triggering decouples teams. A consumer declares the dataset it needs rather than the job that builds it, so the producer can restructure its own pipeline without renegotiating.
The trade-off is conceptual overhead. Teams have to think in assets and freshness rather than jobs and schedules. That transition is usually harder than the tooling change.
Meridian layers all three. The nightly graph is time-driven at its root, because the business day starts whether or not a partner file arrived. Region branches wait on file-arrival events. Shared downstream models are data-aware, so they rebuild when inputs actually change.
Trigger on data availability where teams are separate, and on time only where inputs are reliable.
Where Orchestration Sits in the Platform
Orchestration is a control plane. It sits beside storage, governance and integration rather than inside any one of them.
It should not become where business logic accumulates. That boundary erodes easily. A transformation gets expressed as orchestration configuration because it was faster that day. A year later a metric definition lives in a scheduler rather than in version-controlled code.
How control planes relate to domains, storage and consumption is an architecture question, set out in modern enterprise data architecture.
Keep the orchestrator invoking logic, never holding it.
Orchestrating ETL and ELT Workloads
The transformation model changes the shape of the graph. Pre-load transformation concentrates work in a processing tier. The graph is a short chain with a heavyweight middle step, and sequencing extraction and load around it is the main job.
In-platform transformation produces the opposite shape. Many small interdependent model layers sit inside the warehouse, each depending on the one beneath. The graph is wider and deeper. Most of the orchestrator's effort goes into layer order and concurrency against shared compute.
Cost control becomes an orchestration concern in that second case. Rebuilding every layer on every run is simple and expensive. Rebuilding only what changed requires the graph to know which inputs moved, which is what data-aware triggering provides.
Meridian's mixed pipeline shows both. The tokenisation step is a single heavyweight task before load. Everything downstream is a layered model graph inside the warehouse.
Deciding where transformation belongs is covered in ETL vs ELT for enterprise data pipelines. The storage platform that constrains both is compared in data warehouse vs data lake vs data lakehouse.
ELT estates need concurrency and selective rebuilds; ETL estates need sequencing around a heavy step.
Streaming and Continuous Pipelines
Continuously running stream processors are not orchestrated task by task. They are long-lived services. The orchestrator's role shifts to lifecycle: deployment, version rollout, state migration and controlled restart.
Batch work around the stream still needs orchestration. Reconciliation jobs, daily aggregates over landed events and replay after a processor fix are ordinary graph tasks. Their inputs happen to be produced continuously.
Latency targets, delivery guarantees, ordering and replay belong to streaming design. They are covered in real-time data architecture.
Orchestrate the lifecycle of stream processors and the batch work around them.
Failure, Recovery and Service Levels
Design recovery before throughput. During an incident the operative question is what has to be rebuilt and in what order. A graph that cannot answer it turns a two-hour fix into a two-day reconciliation.
Distinguish failure classes. A transient infrastructure error deserves an automatic retry. A late source deserves waiting and then alerting. A contract violation should stop immediately, because retrying produces the same wrong answer faster. Resource exhaustion needs a different fix and will recur until capacity changes.
Blocking downstream work on failure should be the default. Letting dependents run on stale inputs converts one visible failure into several quiet ones. That is precisely what happened to Meridian's finance mart.
Express service levels per pipeline. A regulatory extract and an internal dashboard justify different recovery effort. A single blanket target either over-engineers the dashboard or under-protects the extract.
- Classify failures as transient, late-input or contract violations, with a distinct response for each.
- Hold downstream tasks by default rather than letting them run on stale inputs.
- Define per-pipeline freshness and recovery targets, tiered by business consequence.
- Rehearse a rebuild, so the recovery path is known before it is needed under pressure.
Failure classes drive responses, and blocking downstream work is the safer default.
Governance and Multi-Team Ownership
Orchestration becomes a governance surface as soon as more than one team ships pipelines into it. The graph spans ownership boundaries. An unowned task inside someone else's critical path is a recurring source of unresolved incidents.
Three things need to be explicit. Every task needs a named owner and an escalation path. Access to trigger, retry or backfill needs scoping, because a backfill is a write with real cost. Credentials belong in a secret store rather than in orchestration configuration, where they tend to appear in run logs.
Cross-team dependencies should be dataset contracts rather than direct task references. A consumer depending on a producer's internal task is coupled to that team's implementation. A consumer depending on a published dataset is coupled only to its shape and freshness.
Name an owner for every task, scope who can rerun, and couple teams through datasets.
Migrating from Cron and Embedded Schedulers
Orchestration programmes are usually migrations rather than greenfield builds. The hard part is discovery.
Start by recovering the real graph. Scheduled jobs encode dependencies as start times, so the actual order has to be reconstructed from run logs, job configuration and the people who operate it. Expect undocumented dependencies, and at least one job whose output nothing consumes.
Migrate by dependency cluster rather than one job at a time. Jobs that share inputs and outputs move as a unit. Splitting a cluster across two systems means neither can enforce the dependency between them.
Run both systems in parallel during transition, with legacy writes directed somewhere harmless, then compare outputs before cutting over. Sequencing, validation gates and rollback criteria are the same ones set out in legacy data pipeline modernization.
Recover the real dependency graph first, then migrate whole clusters.
Choosing an Orchestration Approach
Match the approach to the estate, not to the tool the team used before.
A decision framework for choosing an orchestration approach
| Question | Points to a scheduler | Points to a full orchestrator |
|---|---|---|
| Do jobs depend on each other's output? | No, they are independent | Yes, and the chain is growing |
| How often is a backfill needed? | Rarely, and manually is fine | Regularly, over defined ranges |
| How many teams ship pipelines? | One | Several, with shared datasets |
| What does a failure cost? | A delayed internal report | A wrong number in front of a customer or regulator |
| Is execution history needed for audit? | No | Yes, per run and per input |
| Who operates it out of hours? | The team that wrote it | A rota needing legible graphs and runbooks |
Meridian answers the orchestrator column on five of six. The FX incident answered the fourth on its own.
Between managed and self-hosted orchestration, the deciding factor is rarely features. It is whether the organisation wants to own upgrades, scaling and availability of the control plane. A self-hosted orchestrator is a production service with its own on-call implications.
Record the decision and revisit it when the estate changes shape. The threshold that justified a scheduler stops applying the moment a second team consumes your outputs.
Let coupling, blast radius and audit needs decide, then treat a self-hosted control plane as production.
Frequently Asked Questions
How Is Orchestration Different From Scheduling?
A scheduler starts jobs at fixed times and infers order from those times. An orchestrator runs an explicit dependency graph, so a task becomes eligible when its inputs exist rather than when the clock reaches a value. It also holds downstream work back when something upstream fails.
When Should A Team Move Beyond Cron?
When jobs start depending on each other's output. At that point the padding between start times becomes an undocumented contract and the system's error budget. A late source then produces wrong results rather than a visible failure.
What Makes A Task Safe To Retry?
Idempotency. The task must be parameterised by the window it owns and must replace that window's output rather than appending. Running it twice then produces the same result as running it once. Appending without a key is the common way this breaks.
How Should Backfills Be Handled?
As a first-class operation over a date range, using the same idempotent tasks as normal runs, with a concurrency cap. Uncapped backfills can saturate shared warehouse compute and starve production pipelines while history is rebuilt.
Should Downstream Tasks Run If An Upstream Task Fails?
By default, no. Allowing dependents to proceed on stale inputs turns one visible failure into several quiet inaccuracies that are harder to detect and explain. Continuing should be a deliberate exception for a named pipeline.
Should Business Logic Live In The Orchestrator?
No. The orchestrator should invoke logic, not contain it. Transformation rules expressed as orchestration configuration end up outside version control and outside review, which is how a metric definition ends up living in a scheduler.
