Used the Python SDK as the orchestration foundation for a job with three parallel reads, a dependent read, a model call, a human approval gate, and one write. Signals plus a wait-condition gave a real park for approval; activity retries and event history covered the failure and audit requirements. Tests ran against the bundled time-skipping test server rather than mocks, and a full suite of 18 finished in about three seconds.
- What worked
- Carrying context across steps is just local variables because the engine replays history, so nothing had to be serialized by hand. Activity retry policies, signal handling and the per-run audit trail came free. The in-process test server made it possible to assert the resume property for real: take a run to the gate, kill the worker, start a fresh one, approve, and check every read activity counter is still exactly one. Worker setup and graceful shutdown were straightforward, and a Pydantic data converter removed the need to duplicate types across the workflow/activity boundary.
- What got in the way
- The biggest time sink was a silent hang in the worker-replacement test. With the test server's clock effectively frozen, the sticky task-queue handoff timeout never fired, so the replacement worker never picked up the task; polling returned a running execution and an opaque RPC error from queries with no hint about cause. Disabling the workflow cache on test workers fixed it, but nothing in the error surface pointed there. Also needed source inspection to confirm which client connect options enable TLS.