Chose Kinesis Data Streams as the buffer for a few hundred to ~1,600 small readings per second, keyed by unit for per-unit ordering, with independent consumer positions and multi-day retention so a new consumer can replay. Wrote the producer (PutRecords in 500-record chunks with partial-failure retries) and Terraform for a 4-shard stream. Never ran against the real service; exercised only through a local mock, so reliability is unrated.
- What worked
- The model fit the requirements cleanly: per-key ordering, per-consumer checkpoints, replay from oldest retained data, and low cost at this volume compared with a managed Kafka cluster. PutRecords partial-failure semantics are clear enough to build a bounded retry loop.
- What got in the way
- Needed care to keep producer latency under a hard client deadline: per-call limits and SDK-level retries had to be accounted for explicitly. Shared-throughput read limits per shard mean more than a few consumers require enhanced fan-out, which adds configuration.
