Engineering guide ยท Raideria Software
How to Run, Monitor, and Recover Production Python Jobs
A practical operating model for recurring Python and Bash jobs: schedules, execution state, logs, outputs, retries and safe recovery on your own infrastructure.
Running a script is not the whole operating model
A recurring import, export or report needs more than a process that starts on time. Someone must know which input was used, whether the run finished, what it produced and what can safely be repeated after a failure.
Dagychu is a self-hosted execution-control platform developed by Raideria. It provides pipeline and task operations, scheduling, workers, logs and recovery controls. Business logic stays in your job code; execution is operated through a separate control plane.
For one low-risk script, cron and a useful failure alert may be sufficient. Evaluate a separate execution layer when dependencies, investigation and recovery create recurring operational work.
What an operator needs for every execution
Preserve one task identity across dispatch, execution and investigation. A successful process exit is useful evidence, but it does not by itself prove that a report was delivered or a downstream record was updated correctly.
| Question | Evidence to retain |
|---|---|
| What started the work? | Task identity, input and trigger context |
| What is happening now? | Task and job-run state |
| Which attempt failed? | Attempt-specific error and logs |
| What did it produce? | Outputs linked to the execution |
| What may be repeated? | Retry policy and side-effect rules |
| Did recovery work? | The subsequent attempt and verified business result |
Example: a report pipeline with a recoverable failure
Consider an illustrative three-step workflow: extract rows, generate a report, then publish it. This is a workflow design, not copy-and-paste Dagychu configuration; use the versioned examples from the distribution repository for the exact YAML schema.
If publishing fails after the destination accepted a report, a blind retry may deliver it twice. Define an idempotency key or reconciliation check in the business logic. A scheduler cannot infer the semantics of an external side effect.
| Step | Output | Recovery question |
|---|---|---|
| Extract rows | A snapshot or a stable reference to the input | Will another attempt read the same reporting period? |
| Generate report | A report artifact associated with the task | Can this step replace an incomplete artifact safely? |
| Publish report | A destination reference or delivery receipt | How will you prevent a duplicate delivery? |
Where Dagychu fits
Dagychu uses YAML-defined pipelines and local or Docker-based job execution. Its web UI exposes pipeline and task operations, execution logs, scheduling and recovery controls. It stores execution metadata in your PostgreSQL instance; job logs live on worker storage.
- Keep business logic in Python or Bash jobs and define dependencies in the pipeline.
- Inspect failed job runs and logs before choosing a recovery action.
- Use a controlled rerun to check both execution state and the resulting business data.
- Treat worker permissions, credentials, backups and any Docker socket access as infrastructure responsibilities.
Start with one existing workflow
Do not migrate every background job at once. Keep existing tooling where it already meets your operational requirements.
- Select a non-critical import, export or report with a known failure path.
- Install Community in an isolated environment using the repository instructions. Review prerequisites and release-specific configuration.
- Configure, validate and connect the project before running its jobs.
- Run a successful execution and record the expected output.
- Introduce a controlled failure using synthetic data; ask an operator to find the failed job and explain its state.
- Recover it using the workflow's safe-rerun rules, and check for missing or duplicate effects.
Frequently asked questions
Can Dagychu run existing Python scripts without changing their code?
Yes. An existing Python or Bash script can run as a standalone Dagychu job without source-code changes. You still configure its project runtime and pipeline YAML. Change the script or add a thin adapter only when it must consume structured JSON input from Dagychu or emit structured output_json for downstream jobs and reports.
Does a retry guarantee exactly-once business effects?
No. Design idempotency or reconciliation for writes to external systems. Execution retries and business correctness are separate concerns.
Is Dagychu Community open source?
No. Community is free to self-host from runtime images. The public repository contains distribution materials; the application source is not public.
Does self-hosting mean no telemetry?
No. Community product telemetry is enabled by default and can be disabled by an administrator. Review the current README for the exact aggregate fields and settings.
Evaluate one workflow
Start with one non-critical Python or Bash workflow. Inspect its successful and failed runs, then test recovery against your own acceptance criteria.
Community is free to self-host from container images. The public repository contains installation materials, documentation and examples, not the application source. External Execution API (/ext/*) and RBAC are Enterprise capabilities; confirm the edition boundary for the release you evaluate.
Open the live Dagychu Enterprise demo to inspect a running product before installing Community.
Related guides
Sources and further reading
This guide is written by Raideria, the maker of Dagychu. Product documentation and release-specific examples take precedence over conceptual examples here.