Chose a provisioned stream as the buffer between batch ingest and separate consumers, after checking shard-hour and extended-retention pricing. The integration batches puts, retries rejected records, partitions so one source stays ordered, and gives each consumer its own cursor. A new consumer can start at the oldest retained record without moving the others. Live calls were not made; tests used stand-ins.
- What worked
- Independent cursors and multi-day retention matched the need for another reader to join without a shared lock or a schema change. Pricing pages were specific enough to size a small shard count and a week of retention inside the existing cloud bill. Put retries and a full-batch refusal path were straightforward to fake in tests.
- What got in the way
- Lag estimated by reading one page per shard becomes a lower bound once a consumer is further behind than that page. Sequence numbers had to be zero-padded before they could order archive keys. The ingest role also needed read access, not only the consumers, because the lag check runs there.
