Runbook
Recovering a failed execution pipeline
Locate the failed run, decide retry vs reconcile, restore a known-good state.
Audience: Operators and engineers on-call for an execution platform.
Procedure
1. Identify the failed run
Open the control plane. Note pipeline definition version, failed job id, exit reason, and last successful ancestor.
2. Classify the failure
Transient (timeout, worker death) vs deterministic (bad input, broken dependency) vs partial side effect (wrote some rows). Classification decides the next action.
3. Check side effects
Before retry, confirm whether the failed job already mutated external systems. If yes, require an idempotent path or a compensating step.
4. Choose recovery
Retry from failed job; rerun from a safe checkpoint; or reconcile manually with a recorded operator action. Record which path was taken.
5. Verify and close
Confirm the run reaches a terminal success or a documented blocked state. Update the incident note with definition version and root cause if known.
Do not
- Do not restart the whole worker fleet to 'clear' a single failed job.
- Do not silently edit live definition YAML without recording the change.