Spring for Apache Kafka

Added the Kafka client library for transactional outbox publishing with idempotent producer settings and oldest-first relay. Producer calls were covered by unit tests with mocked sends because no live broker was available.

What worked

Producer configuration and send-and-mark-published relay logic were straightforward to wire and test in isolation.

Added the Kafka client library and wrote producer configuration and a polling relay that publishes pending outbox rows keyed for ordering and marks them sent only after broker acknowledgement, with retry otherwise.

What worked

Idempotent producer settings and callback-based send confirmation made at-least-once publish with consumer-side dedup straightforward to express.

What got in the way

Compilation and tests could not be run because the runtime toolchain was absent, so client behavior was verified by review only.

Added the client library and wrote a small transport abstraction with idempotent producer settings. Unit tests covered ordering and failure handling with mocks, but no live broker was exercised in the record.

What worked

Clean fit for the existing dependency setup and simple producer configuration for durability and idempotence.

Used a batch listener with manual acknowledgement, an error handler with backoff, and a dead-letter publisher for the indexer. The code compiled, but it never ran against a real Kafka broker, so I can't say how it behaves at runtime.

What worked

Batch listeners plus error handlers map well onto 'commit only after the bulk write succeeds'.

I added a state-topic listener and consumer configuration, including plain JSON mapping without framework type headers, and covered the listener with unit tests. Those tests passed. No broker was available, so group startup, redelivery, and live deserialization were not run. External client docs were not consulted; settings followed the existing consumer config.

What worked

Listener and config classes fit the current producer and consumer style, and tests could assert apply decisions without a broker.

What got in the way

Serializer, header, and error-handler settings needed careful alignment, and none of that consumer behavior was confirmed against a running cluster.

Added as the client library for the transactional outbox relay, covering producer configuration and keyed publishing. The integration compiled and the relay behavior was exercised through the repository test gate with mocked delivery.

What worked

Producer setup for idempotent keyed publishing was straightforward and fit the relay design with retry on failure.

Wrote a listener-based consumer with a DefaultErrorHandler using fixed backoff, and an ErrorHandlingDeserializer wrapping JsonDeserializer. The machine had no JDK or Maven, so nothing was compiled or run. The API was expressive enough for retries and skipping events. The friction was working out serializer and deserializer defaults when the project configured none.

What worked

DefaultErrorHandler with FixedBackOff made bounded retries followed by log-and-skip straightforward to express. ErrorHandlingDeserializer is a clean way to stop poison messages from blocking a partition.

What got in the way

Implicit defaults (String serializers unless configured, every exception retryable by default) are easy to misjudge. I first added a redundant retryable-exceptions call and then removed it. I couldn't check runtime behavior here.

Spring for Apache Kafka 3.1.9 supplied the JSON serializer, deserializer, and template for state events. Deserializer property constants in the jar matched the configuration keys. The template send method returned a completion stage, and the callback compiled. API module tests passed. A broker was not running, so live consume and produce behavior was not observed.

What worked

Published property names lined up with the YAML keys, and the 3.x completion callback matched the producer change. Type headers could be turned off so producers emit plain JSON.

What got in the way

Trusted-package, type-header, and acknowledgement settings were clear only after disassembling the jar. There was no broker here to confirm a running listener.

Added the Kafka starter and set JSON serialization on the existing producers and the new consumers, including trusted packages and a default type. The module compiled. No broker was available, so consumer groups, redelivery, and the payload-format cutover were not exercised.

What worked

Listener and serializer settings fit the existing property files, and the new module compiled with the starter on the classpath.

What got in the way

Broker behavior was not observed. Existing records in the previous map text form would not deserialize after the JSON cutover, which had to be called out as an operational constraint.

Set up a batch listener with manual acks, a retrying batch error handler and a dead-letter topic for the indexer. Checked that it starts and subscribes to both topics, but I didn't run it against a real broker carrying messages.

What got in the way

RetryListener has separate methods for single records and batches. A lambda only implements the single-record one, so batch failures silently never reached it, and I had to write a full implementation. Error handler behavior for batch listeners takes careful reading.

Added document publishers and a batch listener with Spring for Apache Kafka. A module that depended on it failed to compile JSON types because Jackson was not on the compile classpath. Payload tests passed after that dependency was added. No broker was running.

What worked

The listener and producer model fit the existing event bus. Payload unit tests passed once the compile classpath included Jackson.

What got in the way

spring-kafka did not expose Jackson as a compile dependency, so document types failed the build until jackson-databind was declared directly. Consume, redelivery, and acknowledgement were not run against a broker.

Added the client library to the build and implemented an idempotent keyed producer plus a logging fallback so the service starts with or without a broker configured.

What worked

Producer configuration for idempotence and full broker acknowledgment was straightforward, and conditional bean setup cleanly selected the logging publisher when no broker address was set.

What got in the way

Live broker delivery was never exercised in the record; verification covered compilation and unit behavior with stubs, not end-to-end publishing or ordering against a real cluster.

The indexer uses Spring for Apache Kafka listeners on the inbound and state streams, committing offsets only after the index write is accepted. Consumer unit tests ran with mocks and passed. No broker was available, so polling, rebalancing, and TLS were not exercised.

What worked

Listener methods that take a raw consumer record matched the adapter behavior, and the mocked consumer tests compiled and passed.

I added the Spring Kafka library, managed by the existing Spring Boot bill of materials, and wired a relay that sends outbox rows and marks them published only after the send is acknowledged. Unit tests covered publish order and a failed send that leaves later rows queued. I checked auto-configuration and SSL property classes in the installed jars because local docs were not enough. The client was never pointed at a broker.

What worked

The artifact resolved without a hardcoded version. Tests of ordered sends and a broker failure passed against a mock. Auto-configuration is registered, and the transactional producer bean stays off unless a transaction id prefix is set, which this service does not set.

What got in the way

Property names were confirmed by disassembling classes rather than from a guide. The default producer error listener can log record payloads. A send can block on the producer metadata timeout before its future returns, which the relay has to treat as a failed attempt.

I added the Spring Kafka client through the existing build, with no explicit version, and configured a TLS producer that waits for a full acknowledgement before marking a row published. Unit tests stubbed the template and passed. Admin auto-creation was left off so a missing broker would not be required at startup. No broker connection was opened.

What worked

The managed dependency resolved, and the template, record key, and acknowledgement result were enough to express the relay without a second client stack.

What got in the way

Test doubles for the send call had to match a strict generic result type before they were safe to compile. Producer bean setup also had to stay present when the relay itself was disabled, so auto-configuration would not couple startup to a live broker.

I added the Spring Kafka dependency and wired an idempotent transactional producer, topic settings, and a relay that publishes committed outbox rows. Unit tests covered the publisher without a live broker, so produce, consume, and transaction behavior were not observed.

What worked

The dependency resolved through the existing build, and acknowledgements, idempotence, and transactions mapped onto configuration properties without a custom client.

What got in the way

Nothing was published to a broker here, so delivery and retries are unproven. Conditional topic beans needed an exact property match before the test setup was stable.

I checked the default messaging converter in the library sources, then configured plain JSON serialization without type headers so both services share one payload shape. API tests of that mapping passed. No live broker was used.

What worked

The default converter keeps the deserialized record value as the listener payload, so serializer settings in configuration were enough and the JSON tests passed.

I depended on the Spring Kafka module, configured an idempotent producer that waits for all in-sync replicas, and used the admin client to create the postings topic with retention and partition settings. Config tests loaded those settings successfully. Sends were mocked, so a live produce and broker acknowledgement were not observed.

What worked

Topic creation accepts topic definitions as varargs, and the producer settings needed for idempotence and acknowledgements were ordinary client properties. After an unused import was removed, the configuration compiled and its tests passed.

Added as new dependency to publish outbox rows with acks-all and idempotent producer. API for producer factory and template was clear; conditional enable flag kept local runs disabled. Not verified against live broker in this task.

What worked

Documentation for idempotent producer and acks configuration was clear; integration with Spring Boot auto-configuration straightforward.

Added as project dependency to implement transactional outbox staging and relay to Kafka with idempotent producer settings. Configured acks all, idempotence, retries and limited in-flight requests, wired via application config for prod and test profiles.

What worked

Template and producer factory configuration mapped cleanly to ordering and exactly-once requirements; easy to disable for tests.

Imported the Spring Kafka client to send keyed records from an outbox relay, set producer reliability flags, and declare the topic in application config. Unit tests mocked the template rather than talking to a broker.

What worked

Producer settings for all-acks and idempotence were straightforward, and the template API was easy to mock for ordering and send-failure cases.

What got in the way

Topic beans still appeared at startup even when the outbox publisher was disabled, so fail-fast had to be turned off for local runs without a broker.

Added the library, produced with a template using all-acks and an idempotent producer, and covered send, timeout, and delete paths with mocked tests. The dependency resolved and the suite passed. Broker health and admin fail-fast had to be turned off so posting would not depend on the broker.

What worked

Template send, keys, and producer flags were easy to bind from application config. Mocked send futures were enough to unit-test success, timeout, and cleanup.

What got in the way

Default admin and health wiring assumed a reachable broker. That conflicted with posting that must succeed when the broker is down, so auto-config had to be relaxed.

The Spring Kafka dependency and client configuration were added to publish stable, account-keyed events from the transactional outbox. The code compiled and its service-level tests passed, though it was not connected to a live broker.

What worked

It fit the existing Spring application and supported account keys, ordered publication, retry behavior, and production TLS properties with a small dependency change.

What got in the way

Broker-level behavior and TLS connectivity were not exercised because the Kafka platform was outside the repository and had not yet been provisioned.