Ops-heavy data platform with accidental orchestration

Context

A product company runs a data platform that grew from a few ETL scripts into dozens of background processes: nightly syncs, reconciliation, report generation, and a new AI enrichment pipeline. The web application still triggers much of the work.

Problem

Operators cannot explain how jobs interact. Failures require manual recovery. Engineers spend more time debugging operations than shipping features. Changing a pipeline is risky because dependencies are tribal knowledge.

Constraints

• Cannot freeze product delivery for a full rewrite.
• Existing warehouse and some pipelines must keep running.
• Team is small; the solution must be operable by the same people.

Architecture

Introduce a dedicated execution plane. Move long-running work out of the web app. Keep the warehouse; make pipeline definitions versioned and inspectable. Add per-run logs and a recovery path for failed jobs. Dagychu is a natural fit for the execution plane; the same boundary could be implemented with other tools if constraints demand it.

Implementation

• Map critical workflows and mark which still run inside the app.
• Extract the highest-risk jobs first (money, customer-facing data).
• Define retry and idempotency requirements per job type.
• Publish runbooks for failure and rollback before expanding scope.

Operational result

Expected outcome of this shape of work: operators can answer what is running and how to recover; engineers change definitions with an audit trail; the product app no longer doubles as an orchestrator. Specific timelines and metrics depend on the real system and are not invented here.

Lessons learned

• Execution reliability is an architecture problem, not a ticket queue.
• Visibility and recovery must land before 'more pipelines'.
• A product (Dagychu) that encodes the boundary reduces repeated invention.