Runbook
Investigating a slow DAG
Separate queue delay, job duration, and dependency wait.
Audience: Engineers diagnosing latency in scheduled or triggered pipelines.
Procedure
1. Measure wall time vs job time
Compare total DAG duration to sum of job runtimes. Large gaps usually mean queue wait or blocked dependencies.
2. Inspect the critical path
Find the longest dependency chain. A single slow job on that path dominates end-to-end time.
3. Check concurrency limits
Worker saturation and per-queue concurrency caps look like 'everything is slow' when they are actually gated.
4. Compare definition versions
If slowness started after a deploy, diff the execution definition: new steps, heavier inputs, removed parallelism.
5. Act with a hypothesis
Scale workers, split a hot job, or fix a dependency — one change at a time, with a before/after run to compare.
Do not
- Do not raise every timeout without understanding the critical path.
- Do not treat alert noise as root cause.