Recommended Kinesis and built against it: partition key per trailer, 7-day retention, and enhanced fan-out so each consumer reads on its own. Wrote the writer, polling and fan-out readers, and Terraform. I never ran it against the real service, only against a local emulator and fake clients, so I can't speak to its reliability.
- What worked
- The model matched the requirements well. Partition-key ordering, extended retention for replay, and up to 20 fan-out consumers per stream make adding a new consumer cheap. It is also cheaper and simpler for a small team than running brokers.
- What got in the way
- There is no first-class Python consumer library (KCL), so I had to write checkpointing, shard handling and resubscription myself. A partial PutRecords failure that gets retried can break ordering, so I grouped all of a unit's readings into one record.
