Runbook
Rolling back an execution definition
Return a pipeline to a previous known-good definition without losing auditability.
Audience: Engineers who own pipeline definitions in production.
Procedure
1. Identify the last good version
Use the audit trail of definition changes. Prefer an explicit version id over 'whatever was running last week'.
2. Stop or drain in-flight work
Decide whether running jobs may finish under the old definition or must be cancelled. Document the choice.
3. Publish the rollback
Deploy the previous definition through the same path as a forward change. Do not hot-patch a single worker.
4. Smoke-test with a controlled run
Trigger a non-destructive or canary run. Confirm logs and outputs match expectations.
5. Record the rollback
Leave a note: why, which versions, who approved. Future investigations depend on this.
Do not
- Do not leave two conflicting definitions active for the same pipeline name.
- Do not delete history to 'clean up' after a bad release.