Chose a managed Kafka cluster as the ingest buffer so independent consumer groups, per-key order, and week-long replay did not share a database queue. Wrote cluster, topic, retention, and IAM-auth settings plus client wiring, using public pricing to size brokers. Never applied the stack or opened a live broker.
- What worked
- The log model mapped cleanly onto a new subscriber without schema or poller changes, keyed partitions for ordering, and retention-based replay. Existing compute could pass bootstrap settings into tasks.
- What got in the way
- Identity ARNs, SASL/IAM settings, and whether unauthenticated access was already disabled had to be pieced together. Broker sizing and cost were estimated from search results, not a live cluster.
