Evaluated and designed against Kafka as the target for a self-hosted, on-premises event backbone: topic layout by ordering domain, key-based partitioning for per-entity order, consumer groups replacing cursor rows, and time-based retention for a month-long replay without a full scan. Produced topic definitions, a relay, consumer wiring, and an operational runbook; no cluster was available to run any of it.
- What worked
- The core model maps directly onto the stated requirements: ordering per key, offset-based progress that makes effectively-once achievable with idempotent handlers, and seek-by-time so a long replay reads only the window instead of scanning history. Self-hosting is viable, which mattered given a no-cloud constraint. Retention and partition-count choices are easy to reason about up front.
- What got in the way
- Ordering guarantees are per key within a topic and nonexistent across topics, which is easy to get wrong when laying out topics by event type rather than by ordering domain; I had to correct my own first design for exactly that. It is a heavy operational commitment for a modest event rate, and it moves rather than removes the delivery-semantics work, since deduplication still has to live in each consumer.
