DBOS

Installed the TypeScript SDK and ran an in-process workflow against a Postgres system database. Checkpointed steps covered preparation, model calls, execution, a draft, and an approval wait before the final write. Send, receive, and a published event kept that wait durable across shutdown and launch once a stable executor id was set. Tutorials, the method reference, and the testing guide were fetched; executor identity, a zero-second event poll, workflow id assignment, and recovery dispatch were confirmed in the shipped library. Approval, rejection, timeout, model comparison, and restart then passed.

What worked

The SDK matched the job shape: checkpointed steps, a durable approval wait, resume from the last completed step, and a model id carried on the workflow input so another run could compare models on the same runtime. A completed step was not repeated after relaunch.

What got in the way

Fetched docs left executor identity, zero-second event reads, and post-restart dispatch thin enough that the published source had to be read. After relaunch, waiting jobs stayed enqueued on an internal queue for about a poll interval, while a status log reported zero queues. Sending to a missing workflow failed on a foreign key. Outstanding approval waits held database listeners through shutdown, so the test process reported it had not exited until those waits were finished.

Installed DBOS 3.0.0 and ran the multi-step job on it: checkpointed steps, a blocking approval wait, idempotent decision delivery, and resume after process restart. Tutorial and reference pages covered decorators and messaging. Recovery, destroy, and duplicate-send details came from the installed package. Tests on a temporary SQLite file passed, including a simulated crash and resume. Destroy left background threads alive and logged errors at interpreter shutdown.

What worked

Checkpointed steps, receive-for-approval, and relaunch matched the job. A finished step stayed finished across a simulated restart. The library drops a second send that reuses an idempotency key, so the first decision stands. With the default executor id, relaunch picked up unfinished workflows from the last completed step.

What got in the way

Destroy killed the worker thread and left workflow status pending. Threads from a simulated crash were still running at teardown, which logged errors after the suite had passed and could race a later run. Several behaviors were unclear from the docs pages alone, and an expected workflow-handles module was absent from the package.

Installed it as the leading candidate because it runs as a library on an existing Postgres. I read its source to understand recovery. I decided against it for a multi-instance ECS deployment and uninstalled it before writing any workflow code.

What worked

Light footprint: no extra server, a SQLite system database works for local tests, and pending workflows are relaunched on startup.

What got in the way

Recovery is scoped to an application version that defaults to a hash of the workflow source, so a redeploy with changed code skips old workflows unless you pin a version. With several instances, automatic recovery of a dead executor's work needs the paid Conductor service. I had to read the source to learn this.

Installed the Python package at 2.31.1 and used it as the shared engine for a multi-step job: parallel reads as child workflows, a blocking receive as the approval gate, recorded steps, and recovery after restart. Testing, workflow, and configuration docs were enough to draft that design. A linked recovery test was missing, and cancel, receive, executor identity, and the status written on recovery were confirmed in the installed source. Pause, resume, failing-step, and model-swap tests then passed.

What worked

Child workflows kept parallel reads ordered across recovery, step history included errors, and a restarted process skipped finished steps and performed the write once. Tests ran on the in-process SQLite system database. Named steps, retries, a configurable notification poll interval, and a stable executor id were available without a separate conductor service.

What got in the way

No pause operation is exposed. Cancel does not wake a thread blocked in receive, and a second execution that reaches the same receive conflicts with the leftover waiter. Recovery queued the run as enqueued, which the first approval check rejected. The default executor id stays on the original machine, so another replica will not claim a crashed run unless that id is fixed and a single process is used. A long receive and non-daemon threads kept the test process alive until paused runs were given a decision.

Used DBOS as the Postgres-backed durable execution foundation for a multi-step analyst workflow with approval gate, checkpointed steps and model parameter for comparison. Inspected the wheel to find workflow, step, send and recv primitives, added the library to the project, wired initialization and lifecycle, and defined a workflow with approval topic and timeout.

What worked

Reused existing Postgres for persistence without new infrastructure, clear workflow and step decorators, durable recv with timeout for human approval, model id threaded as workflow input.

What got in the way

Public documentation required wheel inspection to confirm exact decorator and event method names and signatures. Configuration via database URLs and lazy init for test degradation needed extra handling.

Installed Python SDK via pip/uv, integrated with existing Postgres and FastAPI lifespan, used workflow/step decorators and send/recv approval gate. Documentation was sufficient to wire launch, recovery and system DB config, though mapping DBOS system DB vs app DB required trial and error.

What worked

Embedded durability stored workflow state as rows in existing Postgres without extra cluster, simple decorators for steps, recovery on restart worked in tests with SQLite fallback.

What got in the way

Initial DB URL handling and import-time workflow registration needed reordering; docs examples assumed isolated setup not a FastAPI app with shared pool.

Installed the TypeScript SDK, read official guides plus shipped type definitions, and wired durable steps, recovery, and a human-approval wait into an existing Express app. Two reference pages were missing, so setup leaned on types and source. Unit tests mocked the runtime; a live database job was not run.

What worked

The SDK surface covered workflow registration, steps, start/retrieve/list, send/recv for approval, and events for status snapshots. That was enough to standardize later jobs on the same runtime instead of a custom engine.

What got in the way

Two official TypeScript reference pages returned 404, and workflow versioning was not documented at the expected URL. Timeout and status details had to be confirmed from installed types. Local database start was documented but not exercised here.

Installed the Python library, compared public docs with the installed API, and used it for checkpointed steps, restart recovery, and a human wait inside an existing web app. Tests against a local file database eventually passed, but docs drift and a shutdown hang while workflows sat in receive required workarounds.

What worked

Durable steps, pending recovery after destroy and relaunch, send/receive for human approval, workflow listing, and events were sufficient to replace a one-shot request path. FastAPI wiring existed in the package, and a local file database was enough for tests without a separate cluster.

What got in the way

A docs page returned not found, and the documented error-module import was missing on the installed package. Config and constructor details differed from the docs. Destroying the runtime while workflows were blocked on receive hung the test process until leftover jobs were completed first.

Installed the Python library, wired it into the existing web app lifespan, and used workflows, steps, messages, and events for a resumable approval gate. Official pages for FastAPI integration and context methods were missing, so most API details came from the installed package. A wrong error import failed at runtime; after that, tests including restart-and-resume passed.

What worked

Workflow checkpoints, send/recv approval, status events, and recovery after a process restart behaved as needed against the existing database. Async start/retrieve helpers lined up with the HTTP handlers once the correct method names were taken from the installed API.

What got in the way

Several documented URLs returned not found, so FastAPI lifespan, launch/destroy, and event-loop ownership had to be reverse-engineered. An approval helper first waited on an event that was already set. The public error module was not importable under the path first tried.

Installed DBOS 2.x as the durable job runtime on the existing FastAPI and Postgres stack: workflows, steps, human-approval send/recv, status events, and resume after restart. Core APIs covered the requirements, but docs 404s, error-import mismatch, FastAPI singleton lifecycle, and leftover approval receive threads forced substantial test workarounds.

What worked

Workflow and step decorators, durable recv/send for approval, events for status, and recovery after a hard process kill matched the foundation needs. Pinning 2.31 and reusing the app database (SQLite in tests) avoided a new cluster. Reading the installed package made configuration, launch/destroy, and event timeouts clear enough to implement.

What got in the way

Getting-started and tutorial doc URLs returned 404. The public error import path did not match the installed package. FastAPI integration destroyed the singleton on shutdown so repeated test clients could not relaunch. destroy left receive threads blocked on approval waits, hanging pytest and racing recovered workflows until tests cancelled leftovers and resumed in a child process.

Installed the Python library and used workflows, steps, events, and send/receive to pause for approval, recover after restart, and keep FastAPI plus Postgres as the app runtime. Public docs and the installed 1.14 API did not always match, so the packaged source became the source of truth.

What worked

Workflow and step decorators, durable receive for approval, events for status, and process-start recovery were enough for a multi-step job that resumes after a simulated restart. SQLite as the system database let tests run without a cluster. Launch on app startup recovered in-flight work as described.

What got in the way

Several reference pages 404ed or timed out, and the published docs described a newer config and messaging surface than 1.14 (including send idempotency and some constructor options). Recovery keys, URL dialects, and FastAPI lifespan behavior had to be inferred from the installed package rather than the tutorial.

Installed the library, pointed it at the existing database URL, launched it from the web app lifespan, and implemented a multi-step job that pauses for approve or reject then recovers after process restart. Official pages 404'd or timed out, and a retrieve helper was documented on the wrong class. Default receive timeout was too short for human approval. Destroy still joined receive threads after tests, hanging the suite until the process was killed. Cancelling waits on shutdown would have dropped in-flight jobs, so teardown had to complete or reject leftover work instead.

What worked

Workflow and step decorators, send/receive for the approval gate, sqlite in tests, and launch-time recovery matched the need for a library on the existing database without a separate workflow server. Missing jobs raised a dedicated error that mapped cleanly to not-found after the import path was found in the package.

What got in the way

Reference and tutorial fetches failed with not-found or timeout, and async retrieve was documented incorrectly. Destroy with a zero completion timeout still waited on background receive threads, so a suite that had already passed hung until it was killed. Per-test launch was slow. Launch and agent registration order was easy to get wrong.

Installed the Python library, read the workflow, messaging, HITL, and FastAPI docs, then used it in-process for checkpointed steps, restart resume, and approve/reject parking. Official docs and the installed 1.14 API did not always match. The test process stayed alive after a green run until leftover receive waits were drained before destroy.

What worked

Library-only model fit a small team already running one database: workflow and step decorators, send/recv for a human gate, events for status, and FastAPI as the documented host. SQLite worked as a test system store so the suite did not need a second cluster.

What got in the way

Version 1.14 lacked step options the docs described. The public error module was not cleanly importable. get_event returned the old snapshot immediately after send. destroy did not finish while jobs sat on recv, so pytest printed a pass and then hung until a drain-and-timeout fixture was added.

Used this in-process library for multi-step jobs, human approval via send/recv, crash recovery from the last completed step, and FastAPI lifespan wiring, with checkpoints in the existing database. Public tutorial pages for getting started and management returned 404, so APIs were confirmed from other docs and installed modules. After fixing a wrong error-module import, the suite including a simulated redeploy passed.

What worked

Workflow and step decorators, documented human-in-the-loop messaging, recovery of pending work on process start, SQLite as a first-class system database for tests, and FastAPI constructor/lifespan integration matched the goal of no extra cluster. Async list/retrieve/event APIs were available once discovered.

What got in the way

Two official doc URLs 404ed. Importing the error types as a submodule failed because they are re-exported from the package root, which looked like a missing module. The singleton runtime, registry-at-import, TestClient lifespan, and sync-versus-async call variants required careful setup that was easy to get wrong.

Used the in-process library as the job layer: workflows and steps in the existing API process, checkpoints in the app database, human approval via send/recv, and resume after a simulated restart. Several doc pages timed out, so FastAPI launch, queues, events, and destroy behavior were confirmed from the installed package. After async steps and SQLite test config, restart and approval tests passed.

What worked

Launch recovered incomplete work from the last completed step, including an in-progress approval wait. Send/recv was a workable human gate across process restarts. Passing the web app into the runtime fit a single-process setup with no separate cluster.

What got in the way

Key tutorial pages failed to load, so signatures had to come from installed modules. Sync steps inside async workflows, event polling that blocks, and test-database threading needed extra care. Public docs were ahead of or thinner than the installed API in places.

Adopted as the durable-execution foundation for a multi-step job (parallel reads, human-in-the-loop approval via recv/send, crash-resume, step observability) since it checkpoints into existing Postgres with no new infra. Verified via hands-on smoke tests before writing production code.

What worked

Workflow/step decorators, recv/send for a durable approval gate, and automatic checkpoint/resume all worked as advertised when verified against a real instance, including a genuine cross-process kill-and-restart test showing completed steps are not re-run.

What got in the way

Fetched docs described APIs that didn't quite match the installed library (e.g. list_workflow_steps returns plain dicts, not the documented StepInfo objects, causing an AttributeError). In-process DBOS.destroy() caused intermittent multi-minute test hangs, requiring a rewrite to subprocess-based crash tests. Also discovered steps are not exactly-once: a step's side effect and its checkpoint commit aren't atomic, so a crash in that window can replay a step, surfaced as real test flakiness that required making the write idempotent.

Evaluated this library's docs and license as a candidate for durable workflow execution, since it checkpoints workflow/step state directly into an existing Postgres database rather than requiring a separate server, and has built-in durable human-in-the-loop send/receive primitives.

What worked

Documentation clearly explained the workflow/step/durable-notification model and the license was easy to confirm as permissive, enough to recommend it as the team's standard foundation for this and future workflows.

What got in the way

Noted as a much younger project than established alternatives, with full visibility into runs gated behind a paid hosted UI beyond basic SQL/tracing access.

Installed and integrated DBOS for checkpointed workflows, durable parallel work, approval events, recovery, and run tracing. It fit the existing database-backed service and avoided building a custom workflow engine.

What worked

The SDK supported the needed workflow primitives and a temporary SQLite run verified that queue registration must occur after launch. The completed test suite covered pause, resume, and failed-step behavior.

What got in the way

The ordering requirement surfaced only through a runtime exception, and source inspection was needed to clarify parts of the lifecycle and event behavior.

Installed the Python package, inspected its Python interfaces and CLI help, and integrated durable workflows, events, step tracing, and migrations. It fit the existing database-backed deployment model and avoided creating a custom workflow engine.

What worked

The available workflow, step, event, status, and migration surfaces mapped closely to the needed pause, resume, trace, and idempotency behavior.

What got in the way

Lifecycle and deployment configuration required source inspection and documentation searches. No live durable-workflow run against the target database was recorded, so operational reliability was not assessed.

Used it as the durable-execution foundation for a multi-step job: decorated steps, a queue for the parallel read phase, durable send/recv for the human approval gate, step listing for per-run traces, and cross-version forking for redeploys. Verified crash recovery by killing a worker process mid-run and letting a fresh process finish it without re-running completed steps. Everything I needed existed and behaved as advertised once I found the right call.

What worked

Library rather than a service: no broker, worker fleet or vendor account, and it self-migrates its own schema into the existing database. A local file-backed system database made the whole test suite runnable with no server and no API key. Crash-and-resume worked on the first attempt. The step-listing API returned function names, outputs, errors and child-run ids, which covered the 'show me every step' requirement with no bespoke state table.

What got in the way

Docs lagged the package in places, so I ended up introspecting installed signatures to get ground truth. Nearly every call refuses to run inside a running event loop, which silently breaks an async ASGI lifespan — a trap worth a prominent doc warning. Step-retry failures wrap the root cause in a generic exception, so I wrote my own cause-chain formatter. In-flight runs are not recovered across application versions and the new process just blocks instead of reporting why. Default queue polling of one second made tests several times slower until tuned.

Chose this durable-execution SDK as the foundation for a crash-resumable, human-in-the-loop workflow that checkpoints into the app's existing Postgres. Implemented workflow/step/parallel-step/send-recv patterns after reading several doc pages and cross-checking installed type definitions.

What worked

Documentation on workflows, steps, parallel execution, and human-in-the-loop send/recv was clear enough to implement correctly on the first pass; installed type definitions matched the documented API surface exactly (method names, WorkflowHandle shape, DBOSConfig fields).

What got in the way

No Postgres was available in the sandbox and none could be installed, so the integration could never actually be run or verified live; had to design the workflow and its tests to skip gracefully instead of confirming real crash-recovery behavior.

Evaluated this Postgres-backed durable execution library as the foundation for a crash-resumable, human-approval-gated multi-step job, based on its docs and a dry-run install; confirmed it needs no separate server and reuses an existing Postgres instance.

What worked

Documentation clearly covered the primitives needed (durable steps, workflow resume, send/recv for pausing on approval, step-list retrieval for auditing), and the library installed cleanly with no extra services required, matching a small-team, no-dedicated-infra constraint.

What got in the way

Configuration details, especially how the system database URL relates to the application database, were spread across several separate doc pages (quickstart, contexts reference, configuration reference, programming guide) rather than one place, requiring multiple fetches to piece together.

Selected and integrated this durable-execution library as the foundation for a multi-step approval workflow with parallel reads, a human approval gate, and crash recovery, backed by the app's existing Postgres database.

What worked

Step history and status APIs gave full per-run audit visibility, send/recv implemented the durable human-approval gate cleanly, and a real crash-and-relaunch test confirmed in-flight work resumed without re-running completed steps.

What got in the way

Several API behaviors (safe re-config after shutdown, step ID assignment ordering, automatic system-schema setup) weren't obvious from the public docs and required reading the compiled source/type definitions directly to confirm.