Chose it as the buffer between a high-rate HTTP ingest path and several independent consumers, then implemented publishing, shard reading with checkpointing and leases, plus on-demand capacity and retention in infrastructure code. Semantics were modelled faithfully in an in-process fake; nothing ran against the live service.
- What worked
- Independent reader positions per consumer are exactly the fan-out property the task needed — a new consumer joins without any change to the existing ones. On-demand capacity removed shard-count planning from the design entirely, and retention gives replay for free, which also solved the deploy-time data loss.
- What got in the way
- Pricing semantics are subtle enough to change the architecture: per-record rounding up to a minimum payload size made one-record-per-item noticeably more expensive than packing, and the packing alternative conflicts with per-item sequence numbers and with downstream delivery formats. Reader-side correctness (expired iterators, closed shards, lease ownership) is all caller responsibility unless you adopt the heavier client library, which is a lot of behavior to reimplement.
