Selected it as the buffer for a few-hundred-per-second telemetry feed with multi-thousand reconnect bursts, then wrote the producer path, a shard poller with external checkpoints, and the infrastructure definition. The per-key ordering guarantee and multi-day replay were exactly the two properties the workload needed, and independent readers let a fourth consumer join without touching the existing three. Never executed against the real service in this environment.
- What worked
- Ordering per partition key plus retention-based replay solved both the ordering state machine and the new-consumer-from-the-beginning requirement that a plain queue could not. Shard sizing rules are simple enough to reason about capacity and cost up front. Sequence numbers are durable, which made an external checkpoint store straightforward.
- What got in the way
- The built-in lag metric is per stream and per shard, not per reader, so with several shared-throughput consumers it cannot distinguish which one is behind; I had to emit a custom per-consumer lag metric to get usable alerting. Shard iterators expiring on a short timer forces extra bookkeeping on top of checkpoints. Batch writes can fail partially, so the producer needs its own per-record retry logic rather than a simple success or failure.
