Picked Kinesis as the buffer between an HTTP ingest endpoint and several consumers. It fit the load (about 1,600 records/s at peak), per-key ordering, replay, and adding consumers independently through enhanced fan-out. I wrote the producer with batched puts and partial-failure retries, plus Terraform for a 4-shard encrypted stream. I had no AWS credentials, so I never ran it against the real service.
- What worked
- The capacity model (per-shard write limits, partition key ordering, retention for replay) was easy to reason about when sizing shards. Enhanced fan-out cleanly met the requirement to add a fourth consumer without touching the others. Because PutRecords reports per-record failures, retrying only the refused records was simple.
- What got in the way
- Kinesis gives no backlog count per consumer, only iterator age, so the lag endpoint had to report time-based lag and a null row count. Pricing could only be estimated, not confirmed.
