franz-go

Added the Kafka client library and implemented a production backend adapter with idempotent production, manual offset commits, and explicit live-group reset guidance. The adapter compiled and unit-tested code paths passed, but it was never exercised against a live cluster in the record.

What worked

Client API covered the needed producer, consumer, commit, and record concepts, and local API inspection helped map the internal log abstraction to client operations.

What got in the way

Finding a client release compatible with the pinned toolchain took multiple version probes, and dependency tidying kept pulling newer transitive requirements.

Added the Kafka client library for idempotent producing with strong acknowledgments, manual offset commits, fetching, and time-based seek for replay. Compiled cleanly and supported the new producer, consumer, and replay paths; no live broker was available in the record.

What worked

API surface for producing, fetching, offsets, and records was discoverable through inline docs and matched the needed commit and replay semantics.

Added as the Kafka client for idempotent production, ordered group consumption, and replay consumption. Module path handling took extra iterations before builds and tests passed.

What worked

Once resolved, builds, vet, and focused tests passed and the adapter covered the intended producer and consumer behavior.

What got in the way

Fetching the client first pulled the parent module rather than the expected submodule, and version listing was confusing before the correct module path was established.

Used the kgo client and kadm admin package to build a consumer-group runner with manual offset commits, blocked rebalance during poll, and a time-based offset reset tool. Picked an older release so it would still build on Go 1.22. The code compiled and passed unit tests against an in-memory fake, but it never ran against a real broker.

What worked

The options I needed were all there and well named: block rebalance on poll, disable autocommit, synchronous commits, and allow-rebalance-on-close. kadm's list-offsets-after-time falls back to the end offset when nothing matches, which made a no-scan replay tool simple. Reading the source in the module cache answered my API questions quickly.

What got in the way

Newer releases require a newer Go, so I had to check go.mod files from the proxy one by one to find the newest compatible version. kadm and kgo are versioned separately and the matching versions aren't obvious. Some behaviour, such as what happens to a group with no commits, took reading the source to confirm.

Used the kgo client and kadm admin package for an idempotent keyed producer, a group consumer with manual offset control and pause/resume, and time-based offset lookup for replay. It covered every need, but several behaviors were only clear after reading the library source.

What worked

Rich low-level control: BlockRebalanceOnPoll, SetOffsets, offset-adjust callbacks, pause/resume per partition, and ListOffsetsAfterMilli for replay. A version that still builds on Go 1.22 was easy to pin. Tests built on it passed repeatedly under the race detector.

What got in the way

Some important behavior is hard to spot in the docs. The delivery timeout counts from the record's own timestamp, so backdated records expire straight away. Paused partitions stay paused across rebalances. The default 5s fetch max wait slowed resume in tests. I had to grep the source to confirm each of these.

Used the kgo client and kadm admin package to build an ordered per-partition consumer with commit-after-handle, rebalance blocking, and time-based offset resets for replay and cutover. I picked a release that still supported the repo's Go 1.22 by reading module files on the proxy. To confirm options like BlockRebalanceOnPoll, SetOffsets, and ListOffsetsAfterMilli, I read the module source. Everything built and passed race-detector tests run five times.

What worked

Rich, explicit API for manual commits, cooperative-sticky balancing, and per-partition polling. The kadm helpers made lag, time-index lookups, and group offset commits short. The source is readable enough to settle questions about behaviour quickly.

What got in the way

Newer releases require a newer Go than the project used, so I had to pin an older version by hand. A few behaviours, such as whether SetOffsets drops buffered fetches and what happens when a rebalance is blocked, were only clear from reading the source, not from the docs.

Installed the Kafka-compatible Go client and its admin submodule, then read the producer, consumer, partitioner, and admin sources to implement idempotent publishes, consumer groups, timestamp seeks, and topic creation. The needed calls were present. Pairing the admin submodule with the client release, and confirming a few behaviors, required reading the module. The wrapper compiled in the test run; it was not pointed at a live broker.

What worked

Idempotent produce, manual offset commits, timestamp-based offset lookup, and topic administration were all available and concrete enough to build a keyed publisher, a group consumer that commits after handling, and a replay path that seeks one group without moving the others.

What got in the way

The first download asked for an admin submodule version that is not part of that module line, so the fetch failed until a compatible release was chosen. The record type was not in the source file its name suggested. The sticky key partitioner comment did not make the hashing rule obvious, and blocking rebalance across a poll looked unsafe if a commit ran after membership changed, so that option was left unused.

I added the Kafka client, its admin package, and PLAIN SASL, then read the module source to code idempotent produce, manual commit, topic setup, and timestamp offsets. The first admin release forced a newer Go than CI used, so I pinned client 1.18.1 and admin 1.13.0. Those type-checked, but acknowledgement defaults and the end-of-log sentinel had to be checked in source. Tests never opened a broker.

What worked

Idempotent writes default on, acknowledgements can require every in-sync replica, and the admin API can create topics and list offsets by timestamp. That matched the producer and replay calls.

What got in the way

A newer admin module raised the declared Go version past the CI pin. After the downgrade, a comment described the default acknowledgement as leader-only while the code used all in-sync replicas, and a timestamp past the log returns offset -1, which is easy to treat as a real position.

Imported franz-go and its admin package to publish keyed records, consume in a group, commit offsets, and seek by timestamp. The newest release needed a newer Go than this repo, so the build was pinned to the 1.18.1 client and admin package 1.15.0. Partitioner, acks, offset listing, and commit behavior were checked in the module source. The client was compiled, and it was never connected to a broker.

What worked

The pinned release exposed idempotent produce, all-in-sync acks, sticky key partitioning, group consumption, and timestamp offset listing with greater-or-equal semantics. That was enough to write the relay, the group consumers, and the replay seek.

What got in the way

Release 1.19.5 could not stay because it required Go 1.23. The default key hash is not callable from application code, so the partition mapping was copied. Turning off auto-commit and which offset a commit must store were clear only after reading the library source.

The client module at v1.18.1 and the admin module at v1.14.0 were fetched, then the module source was read for topic creation, metadata, groups, sticky-key partitioning, synchronous produce, fetch errors, and offset listing. Those calls were compiled into a broker client. The test suite ran against an in-memory log with the same partition and group rules, so this library never connected to a broker.

What worked

The first fetch succeeded at the pinned versions. Building an admin client from the existing client, passing group options as client options, default all-replicas acknowledgements, and default idempotent produce all matched the delivery rules, and the packages compiled in the test run.

What got in the way

Admin and client behavior was clear only after reading implementation files. Topic-create responses needed manual error checks, the error type did not support errors.Is and had to be matched with errors.As, and a fetch can return records together with an error, with the error reported only when no records are present.

Installed franz-go 1.19.5 and admin module 1.16.1, then implemented a relay publisher, group commits, and timestamp seek from the library source. The admin module required Go 1.23, above the repo's previous 1.22 line. Error helpers live inside the main module, so the first path lookup missed them. The client compiled in test and build. Those runs did not open a broker connection.

What worked

Cached source spelled out commit, background heartbeat, and poll-blocking behavior well enough to implement commit-after-handle and replay seek. Once the error package was opened inside the main module, its constants matched the client calls.

What got in the way

The first fetch looked successful, yet the module file still lacked both packages on the next read. A later fetch recorded 1.19.5 and admin 1.16.1. The error helpers were easy to miss because they are not a separate module.

Pinned franz-go v1.18.1 and the separate admin module v1.16.0 after a newer client release forced a language upgrade. Used the installed source to implement keyed produce, consumer groups that commit after a successful handle, topic setup, timestamp offsets, and SASL. Offline tests compiled and passed. No broker was contacted, so runtime delivery was not observed.

What worked

Idempotent produce is on unless disabled, and a sticky key partitioner can lock one key to one partition. The admin client covers topic creation, offsets after a timestamp, and group offset commits. Source comments described the poll, block-rebalance, commit, and allow-rebalance sequence needed for commit-after-handle.

What got in the way

Fetching v1.19.5 rewrote the module language version. Admin helpers are a separate module whose matching release pulled a newer crypto library and threatened that pin again. Early lookups used paths that were not modules. Package docs alone did not show that commit is a method, how offset values are built, the SASL package path, or the default partitioner.

Installed the client plus its admin module and wired produce, poll, consumer groups, topic create, offset list, cooperative sticky assignment, and SCRAM. The first fetch used one version for split modules and failed. Most of the API was learned by reading cached source, not package docs, then the code compiled and unit tests used a fake log instead of this client.

What worked

Once versions were aligned, the client covered producers, grouped consumers, admin topic and offset calls, TLS, and SCRAM without needing another library. Fetch error helpers and balancer options existed and compiled after the source review.

What got in the way

Admin and message packages are separate modules and do not share the client’s version tag, so the first install failed. Expected helpers such as a dedicated error matcher were missing, and fetch error methods were easy to mix up. Setup meant inspecting many cached files before writing a stable client.

Added the client plus its admin package and built a publisher, a consumer-group member with manual commits and per-partition concurrency, and an offset-reset-by-timestamp tool. Everything compiled against the documented API on the first build; never exercised against a live broker, so runtime behavior is unassessed.

What worked

The option-based configuration covered everything needed without reaching for internals: cooperative-sticky balancing, blocking rebalance during processing, disabled autocommit, TLS and SASL. The per-partition iteration callback made 'concurrent across partitions, strictly ordered within one' fall out naturally. The admin package exposed timestamp-to-offset lookup and group offset commits directly, which is what made replay an index lookup instead of a scan.

What got in the way

The latest release requires a far newer language toolchain than many projects pin, and the requirement is only visible after resolution rewrites the module file — I had to bisect across three releases to find one that stayed on the pinned version. The commit-and-rebalance semantics around partial batch failure took careful doc reading to get right; a worked example of 'fail the batch without advancing the commit position' would help.

Added this Kafka client plus its admin package and built the whole transport layer on it: idempotent producer, manual-commit consumer groups with cooperative-sticky assignment and static membership, rebalance blocking while a batch is in flight, and timestamp-based offset seeking for replay. It compiles and vets clean, and its partitioner was reused in tests as a reference oracle, but no real broker was available so the client path itself never ran.

What worked

The option surface covers everything a careful consumer needs without reaching for internals: manual commits, blocking rebalance during in-flight batches, static group membership, idempotent production, and a pluggable logger interface that adapted to structured logging in a few lines. The admin package exposes list-offsets-after-timestamp and end-offsets directly, which is exactly what a bounded replay window needs. Its exported partitioner hashing matched my own independent implementation exactly, so it doubled as a trustworthy test oracle.

What got in the way

I had to read the module source to confirm signatures for several core calls — poll, commit, fetch iteration, offset constructors, and the admin offset listings — because the names and shapes were not guessable and I had no handy reference. Commit ergonomics push you toward holding on to the library's own record type, so wrapping it in a domain record meant smuggling the raw pointer through an unexported field to keep epoch handling correct. Admin offset results also use sentinel values for not-found partitions that are easy to mishandle silently.

Added this Kafka client as the transport for a new internal streaming package: keyed producer with idempotent writes and compression, consumer groups with commit-after-handle, and a time-bounded replay reader that seeks by timestamp using the admin subpackage and explicit per-partition offsets instead of a throwaway consumer group. Everything compiled and the logic was tested against an in-memory fake; nothing ran against a real broker here.

What worked

Pure Go with no C dependency, which suits container builds. The option-based client construction covers acks, compression, producer idempotency, and group configuration without ceremony. Exposing both the group-consumption path and direct partition consumption with explicit offsets made a no-residue replay tool straightforward. The separate admin package gave a clean way to resolve offsets from a timestamp, and the first version of those calls compiled without rework.

What got in the way

Choosing a release that matches an older language version takes guesswork from the outside; I picked a line by inference rather than from a clear compatibility statement. The split across client, admin, and protocol modules means several coordinated version pins instead of one. Because there was no cluster available, rebalance, commit, and offset-seek behavior remain unverified.

franz-go supplied the Kafka client and administration APIs used for keyed consumption, production, commits, and timestamp-based replay. The final implementation compiled and passed race tests, but API discovery required local documentation and source inspection.

What worked

The client covered both data-plane and administrative Kafka work in Go, and the completed implementation passed the full test, race, vet, and build checks.

What got in the way

A presumed MaxPollRecords option was unavailable, and a presumed admin method name did not exist. Adding kadm 1.15.0 also upgraded the core client from 1.17.1 to 1.18.1 and several transitive modules.

Added this Kafka client to produce, consume, create topics, seek by timestamp, and commit offsets for a Kafka-compatible broker. Installing it was awkward: the admin package is a nested module, the newest admin tag needed a newer Go than the repo, and the produce/consume/admin APIs had to be confirmed from downloaded source rather than a short setup path.

What worked

After pinning a compatible pair of module versions, the client surface covered the needed operations: sync produce, topic create with already-exists handling, group consume, timestamp-based seek, and offset commit. Types and option functions were consistent enough to wrap behind a small bus and replay helper.

What got in the way

Fetching the admin package from the main module path failed. The current admin release required a newer Go, so an older admin tag had to be chosen by inspecting module files. Several source reads missed the cache path. Nothing here was exercised against a live broker, only compiled and unit-tested behind an in-memory stand-in.

The admin module supplied timestamp lookup and consumer-group offset operations for the replay command. It compiled and passed project checks, but its independent module versions required explicit compatibility research against the core client.

What worked

Its typed group and offset APIs enabled replay administration without shelling out to broker utilities.

What got in the way

The compatible version was not obvious from the main client dependency and had to be determined by inspecting module metadata and source.

Chose this pure-Go streaming client for a producer, a consumer-group member and an admin helper that resolves offsets by timestamp. Wrote a synchronous publisher, a group member that handles partitions concurrently but each partition strictly in order with manual commits and rewind-on-failure, plus a time-based offset seek for replay. Everything compiles and vets clean; never run against a real broker here.

What worked

Feature coverage was exactly what the design needed in one dependency: idempotent producer, consumer groups, manual commit control, rebalance callbacks, explicit offset setting, and a companion admin package for listing and committing group offsets by time. No cgo meant no build complications. Option names were discoverable and consistent once found, and the admin package's timestamp-to-offset semantics were documented precisely enough in source comments to reason about empty-partition edge cases.

What got in the way

I could not rely on memory for exact identifiers and ended up grepping the vendored sources repeatedly to confirm option names, client method names, struct field names and callback signatures — a doc page mapping concepts to symbols would have removed several round trips. Recent releases also require a newer language version than the project was pinned to, so I had to probe older releases to find a pair of modules whose requirements still fit; the release notes do not make the minimum version obvious from the outside.

Installed and imported the Kafka Go client plus its admin module to produce, consume, create topics, and commit offsets. The admin API lives in a separate module, so the first install failed; a follow-up install pulled an incompatible newer client. After pinning versions that still built on the project’s Go, the admin and producer APIs were checked from the module cache rather than from standalone docs, then the client compiled.

What worked

Once pinned, produce, fetch, consume-partitions, topic creation, timestamp offsets, and group commits matched the intended producer, consumer-group, and time-window replay design. The code compiled and unit tests for the surrounding package passed.

What got in the way

Installing the admin package from the main module path failed. Mixing module versions upgraded the client past the project’s Go and produced a version conflict. Admin and record types had to be confirmed in the downloaded source, including offset maps, already-exists errors, and header fields. The client was not exercised against a live broker.

franz-go was added for keyed publishing, consumer groups, explicit commits, TLS, and partition-aware processing. It compiled and passed unit, race, build, and vet checks, though correct partial-batch commit behavior required careful API review.

What worked

The client exposed the producer, group, record hashing, partition iteration, and offset controls needed for the implementation without requiring a second Kafka client.

What got in the way

The failure window around fetched records and commits was easy to mishandle; source inspection was needed to ensure a handler failure would not cause later records to be skipped.

The SDK supplied the producer, consumer, offset, topic-administration, and protocol types needed for the migration. After pinning a Go-compatible release, the integration compiled and passed unit, race, vet, and build checks.

What worked

The API covered consumer groups, explicit commits, offset positioning, topic creation, and custom dialing without requiring several separate Kafka packages.

What got in the way

The newest client line raised the module's Go requirement beyond the repository baseline. Some API discovery required several separate Go documentation queries, and the first combined query was invalid.