Recommended and built an ingest-to-consumers buffer on Kinesis: PutRecords keyed by unit, per-consumer shard reading with checkpoints stored in Postgres, provisioned shards and alarms in Terraform. Never ran against the real service; exercised only via an emulator and the botocore model.
- What worked
- Shard limits, PutRecords batch size, retention and per-shard read limits were clear enough to size the stream and decide enhanced fan-out was unnecessary for four consumers. Partial-failure responses on PutRecords make retry logic straightforward.
- What got in the way
- Without the KCL, resharding (parent/child shards), iterator expiry and checkpointing all have to be handled by hand, which was the hardest part of the consumer loop.
