Used the DBOS Python library as the durable-execution foundation: steps checkpointed to the database, parallel reads via start_workflow, a recv/send approval gate, step history for tracing, and recovery after crashes. SQLite worked for tests. It did the job, but getting shutdown and recovery right meant several small experiments and reading the library source. Sync waits blocked shutdown, async workflows depend on the event loop that started them, and one way of starting a job led to a graceful shutdown marking it ERROR.
- What worked
- Installing it was just a pip dependency, with no new service to run. Crash recovery after SIGKILL resumed from the first unfinished step without re-running finished ones. Step listing gave a full trace per job. Using SQLite for the system database made the tests fast and self-contained. The CLI's workflow list/steps/resume/fork commands fit well in a runbook.
- What got in the way
- A synchronous recv with a long timeout holds a non-daemon thread that destroy() cannot wake, so the process hung on exit. Async workflows run on the caller's event loop, which caused confusing cancellations under a per-request test client loop. Jobs started through the sync wrapper were marked ERROR with a CancelledError on graceful destroy. recovery_attempts rises on every restart toward a default cap of 100. None of this was obvious without reading the source.
