Apache Kafka

Used the event backbone to propagate create and release events from the inventory service to the new staff search index, keeping search updates off the synchronous purchase path.

What worked

Topic-based decoupling kept indexing asynchronous and made the backfill story straightforward.

Relied on as the durable buffer between reservation writes and async indexing so bursts queue instead of failing user requests. Defined event topics and consumer grouping around the existing peak shape.

What worked

The existing event path made it natural to keep writes fire-and-forget and move indexing pressure off the request path.

What got in the way

No live broker was exercised; liveness and rebalance behavior were handled by fallback and resolve-past logic rather than observed.

Reused the existing async producer pattern to emit one unsampled rejected-checkout event per rejection while keeping completed checkouts on the existing order topic. Sends were fire-and-forget with backpressure drops and a disable switch to protect checkout latency.

What worked

Producer reuse kept per-event SaaS billing out of the path and matched the existing serving code shape, so peak-day volume maps to fixed infrastructure cost.

Used as the ordered durable event log for journal fan out to two consumer groups with long retention for replay. Client producer settings and topic retention were iterated to meet ordering and replay needs.

What worked

Partitioning by account preserved per account order and separate consumer groups gave clean fan out without direct database reads.

What got in the way

No live broker was available to prove end to end delivery or replay against a real cluster.

Relied on the existing event bus concept to keep search indexing asynchronous and off the reserve and release path. Implemented event publishing in request handlers and a separate consumer for bulk index updates.

What worked

The async consumer design cleanly isolated indexing throughput from latency-sensitive writes.

Used as the recommended durable event log for ordered, replayable delivery to two independent consumer teams. Added client configuration for idempotent production and defined a long-retention partitioned topic with per-account keys.

What worked

Partitioning by account key plus long retention mapped cleanly to ordering and replay needs, and fit the self-managed regional constraint.

What got in the way

No live broker was available, so end to end delivery, ordering and replay could not be observed in this task.

Reviewed intro and reference docs to assess ordering, retention, replay and operating cost for a low-ops single-VM setup. Rejected because operating a distributed log exceeded budget and staffing constraints for the stated volume.

What worked

High-level concepts were findable for a build-versus-buy comparison.

Set up a local single-node event broker with plaintext listeners to support the service during verification, then confirmed inbound order events with the console consumer.

What worked

Once the broker was formatted and running, event production and consumption confirmed the service path worked.

What got in the way

Broker setup needed repeated listener and quorum configuration edits before it started cleanly.

Selected as the durable partitioned log to give per-account ordering, no-loss delivery, duplicate suppression, and multi-week replay for independent consumers. Design used account-keyed records, idempotent production, consumer dedup, retention, and separate consumer groups.

What worked

Partitioning plus retention plus independent consumer groups matched every stated requirement without adding an external service.

What got in the way

Partition key and retention choices needed care, and no live topic or replay was observed in the record.

Added the Python Kafka client dependency and used it for the fulfilment consumer with idempotent handling. Installation in a fresh virtual environment and the Python test suite succeeded.

What worked

Dependency installation was straightforward and consumer tests passed without client-specific workarounds.

Selected as the self-hosted durable log for ordered per-account journal delivery with long retention and independent consumer groups. Keyed partitioning, idempotent producer settings, and consumer-side dedup covered ordering and replay needs without adding a vendor.

What worked

Partitioning by account, separate consumer groups for fan-out, and time-based retention mapped cleanly onto the ordering, replay, and audit constraints.

What got in the way

No live broker was available in the environment, so end-to-end delivery, ordering, and replay could not be exercised against a real cluster.

Used the distributed log as the recommended ordered, durable fan-out layer with keyed partitioning, retention, and idempotent production. Integrated the client library for publishing and defined topic and consumer patterns, but never ran against a live broker in the record.

What worked

Partitioning, retention, consumer-group replay, and idempotent producer concepts mapped cleanly to the ordering, no-loss, and replay requirements.

Placed reservation and SKU changes on two pre-declared topics, partitioned by record id, with event-type and schema-version headers aligned to the existing producer convention. Publishing is buffered off the request so a full buffer or an unreachable broker drops the event and leaves the hold in place. The broker was never contacted. Topics are left for platform creation, with a one-shot from-beginning flag for rebuilds, so the indexer cannot start until those topics exist.

What worked

Key-based partitioning and a small header contract were enough to feed a separate indexer without putting the broker call on the reservation request.

What got in the way

The integration assumes topics already exist. Delivery, consumer group rebalance, and lag were not observed because no broker was available in the session.

Reused existing order event topics with a new async consumer group for idempotent search upserts, preserving the relational database as the system of record.

What worked

Existing event flow made async indexing straightforward without touching the synchronous intake path.

Used event log to decouple reservation writes from search indexing. Added created and released events published before reply and a separate consumer that bulk-applies them to the search projection.

What worked

Established durability pattern matched the existing checkout event; consumer isolated indexing load from request path.

What got in the way

No live broker throughput or lag behavior was observed in this task.

Apache Kafka was already the event bus for order state, so write-back was a new consumer group plus plain JSON producers with type headers disabled. Vendor documentation was not read. Nothing was installed and no broker was contacted. Offset reset, poison-payload handling, and retry-on-database-error were configured from the existing producer settings and the client library.

What worked

The existing topic-and-producer model was enough to add a consumer that applies single-row updates and isolates bad payloads from partition progress.

What got in the way

No broker was available, so partition count, retention replay, and end-to-end delivery were not observed. Serializer details had to be checked in the client jar rather than against a live cluster.

I pinned and installed the Python client, then inspected its version, error constants, admin offset listing, group-partition objects, and watermark calls from the installed package. The client covers topic creation, synchronous acks, per-group offsets, and an OAuth callback hook, which is what the ingest buffer needed. Tests still ran against an in-memory log, so produce and consume were never exercised against a broker.

What worked

Installation of the pinned release succeeded, and repeated interpreter checks agreed on the public surface: both missing-group error names exist, the group-partition type lives on the top-level package, offset futures are keyed by group id, and the watermark timeout accepts positional and keyword forms.

What got in the way

Docstrings were not enough to call the admin API safely. I had to read the installed sources to learn that a list-offsets request rejects an explicit offset when partitions are also set, that one admin client should not have two lists in flight, and that watermark lookups can return None. Private end-of-partition symbols were part of the working pattern. Broker delivery and the IAM callback were never run.

Selected as self-hosted ordered log to replace direct DB reads, meeting residency and replay requirements. Configured topic with per-account partitioning and 30-day retention. Client side configured for acks-all and idempotent producer; not exercised against live broker in this environment.

What worked

Partition-key ordering and retention configuration directly addressed ordering and replay requirements without external SaaS.

What got in the way

Requires self-hosted cluster operations; no live broker verification in this task, so end-to-end delivery not observed.

Relied on existing broker interface via REDIS_URL/KAFKA_BROKERS style env. Topology assumed external cluster; local validation used in-memory paths without live brokers, so end-to-end eventual consistency was not observed.

What worked

Existing Kafka architecture allowed sidecar index without Redis SCAN and without forking checkout flow.

What got in the way

No live broker validation in task; consumer lag and replay behavior at scale not exercised.

Implemented Kafka consumer for order created events to decouple sending from checkout, with throttle-aware pause and resume and a dead-letter topic for failures. Relied on existing broker and topic conventions.

What worked

Consumer group pattern with buffering before checkout reply kept email work off the hot path and supported retry and idempotency checks.

Evaluated Kafka as the architectural pattern for per-meter partitioning, partition-key ordering, consumer-group offsets and retention-based replay. Documentation on partitioning, idempotent producer and offsetsForTimes guided the in-memory partitioned design and topic split.

What worked

Concepts mapped directly to requirements: partition key for ordering, retention plus timestamp seek for 30-day replay without full scan, consumer groups for once-only delivery.

Installed the Python client and wrote a production bus that produces with full acks, polls per consumer group, rewinds on failure, and observes lag without joining groups. Tests used an in-memory stand-in, so the client was not exercised against a broker.

What worked

Producer, consumer, admin, and OAuth callback surfaces mapped cleanly onto durable publish, independent groups, topic creation, and watermark-based lag.

What got in the way

OAuth token expiry units were unclear from the client docs versus the IAM signer helper, so the callback had to be written carefully without a live handshake to confirm.

Evaluated and designed against Kafka as the target for a self-hosted, on-premises event backbone: topic layout by ordering domain, key-based partitioning for per-entity order, consumer groups replacing cursor rows, and time-based retention for a month-long replay without a full scan. Produced topic definitions, a relay, consumer wiring, and an operational runbook; no cluster was available to run any of it.

What worked

The core model maps directly onto the stated requirements: ordering per key, offset-based progress that makes effectively-once achievable with idempotent handlers, and seek-by-time so a long replay reads only the window instead of scanning history. Self-hosting is viable, which mattered given a no-cloud constraint. Retention and partition-count choices are easy to reason about up front.

What got in the way

Ordering guarantees are per key within a topic and nonexistent across topics, which is easy to get wrong when laying out topics by event type rather than by ordering domain; I had to correct my own first design for exactly that. It is a heavy operational commitment for a modest event rate, and it moves rather than removes the delivery-semantics work, since deduplication still has to live in each consumer.