Azure Event Hubs

A comparison search on ordering, multiple consumers, and exactly-once behavior was used to weigh it against the chosen bus. Checkpoint replay, partition head-of-line blocking, no per-message dead-letter queue, and retention limits ruled it out. It was not installed or called.

What worked

The comparison was enough to reject it against durability, deduplication, and independent-consumer constraints without a trial deployment.

What got in the way

As described in that comparison, a partition checkpoint can replay a batch after a successful downstream post, one slow key blocks other keys on the same partition, and there is no per-message dead-letter queue. Events that fall behind the retention window are gone.

Configured an independent Event Hubs consumer group and function binding so billing metering could consume durable telemetry without adding vendor calls to the MQTT callback. This fit the decoupling and replay requirements well.

What worked

Consumer groups provided a clean architectural boundary between enrichment and billing while retaining the original event stream as the durable source.

What got in the way

The integration was built but not exercised against a live Event Hubs namespace, so delivery, checkpointing, and replay reliability were not observed.

A separate consumer group and Event Hubs sequence metadata were incorporated to isolate billing, preserve retries, and prevent replay double-counting. Documentation was needed to clarify custom-handler trigger metadata, and no live Event Hub invocation was performed.

What worked

Consumer groups and stable sequence information supported an independent billing path without blocking the primary ingestion flow.

What got in the way

The exact deployed trigger payload and checkpoint behavior were not verified against a live Function host and Event Hub.

The producer SDK and Capture configuration replaced an impractical one-blob-per-batch design with durable, batched archival. The SDK built successfully, but Capture and its managed-identity storage access were not deployed live.

What worked

Batch publishing paired naturally with Capture and avoided creating millions of small archive files while preserving acknowledge-after-durable-write semantics.

What got in the way

Managed-identity capture permissions required additional documentation research, and the infrastructure definition could not be compiled locally because its validator was unavailable.

A dedicated consumer group and function binding were configured so billing could aggregate accepted telemetry independently. The repository integration built, but it was not exercised against a live Event Hubs namespace.

What worked

The consumer-group model cleanly isolated billing progress and retries from the existing ingestion path.

Relied on the existing Event Hubs producer path to carry an inventory-attributed telemetry envelope while keeping billing network calls outside the ingestion callback. No live Azure send was recorded.

What worked

The existing producer boundary made it straightforward to add trusted customer, byte-count, and retention metadata without coupling ingestion to billing exports.

Imported the messaging SDK to add a second producer for batched usage counters alongside an existing telemetry producer. Wrote batching that handles the over-size-batch condition and flushes, following the SDK's own batching idiom. Code compiled and unit-tested; never run against a live namespace.

What worked

The batch-builder API makes the size-limit case explicit instead of hiding it, and the sentinel error for an oversized event is a real exported value that composes with standard error matching. Adding a second producer alongside an existing one required no restructuring.

What got in the way

I had to grep the installed module source to confirm the exact name and shape of the oversize sentinel error rather than finding it quickly in prose docs.

Event Hubs was used as the durable acceptance boundary and configured with separate consumers for telemetry and device lifecycle events. This kept billing work off the MQTT callback and made retries independent, though the new topology was not deployed live.

What worked

The service architecture cleanly separated ingestion acceptance from ledger updates and external billing delivery.

What got in the way

Runtime delivery, duplicate behavior, scaling, and managed-identity bindings were not observed against Azure in this task.

Event Hubs was configured with a separate billing consumer group so billing retries and checkpoints would not interfere with telemetry enrichment. The deployment was authored but not exercised against Azure.

What worked

Independent consumer groups fit the requirement for separately retryable, scalable enrichment and billing paths without changing the producer's non-blocking behavior.

What got in the way

Cloud-side provisioning and behavior were not verified in this task because the deployment compiler and Azure environment were unavailable.

Kept the existing event-triggered enrich worker as the place that writes hourly usage after each delivered telemetry batch, and deliberately did not fail the handler if the usage write failed so the hub would not redeliver product telemetry.

What worked

The already-wired event path was a reliable place to emit measured-hour usage without changing the MQTT ingest callback. Treating usage ingest as best-effort avoided coupling billing I/O to event checkpointing.

What got in the way

Usage was not observed on a live hub, and a failed usage write is only logged, so meter gaps versus redelivery tradeoffs were designed, not measured.

Extended the existing Event Hubs path with billing metadata, a dedicated billing consumer group, deterministic event identities, and Capture configuration. This cleanly separated metering from analytics and kept billing calls out of the MQTT callback.

What worked

Consumer groups and Capture provided the right primitives for independent processing, retry-safe metering, and a durable evidence stream without another synchronous request in ingestion.

The existing Event Hubs delivery path was used as the billing acceptance boundary so only successfully delivered telemetry produced usage. Partitioning behavior required changing mixed-site processing to send and meter one site at a time to avoid revenue gaps after partial success.

What worked

Successful delivery provided a clear point at which ingested bytes could safely become billable, while keeping billing exports away from the telemetry callback.

What got in the way

The overall send result did not expose enough detail for correct metering of a mixed-site batch when only some site partitions succeeded, so the pipeline needed restructuring.

Extended the existing ingestion path to preserve trusted billing metadata and configured long-retention capture for audit and recovery without coupling telemetry acceptance to billing delivery.

What worked

The service architecture supported a durable separation between telemetry ingestion and later billing export, avoiding a synchronous dual write to the billing provider.

What got in the way

The deployment configuration could not be compiled or exercised in the recorded environment, so service reliability and rollout behavior were not observed.

Extended the existing telemetry stream with trusted billing metadata and configured an independent billing consumer so metering reads only durable events. Repository tests passed, but the integration was not exercised against the hosted service.

What worked

The stream provided a clean durability boundary between telemetry acceptance and asynchronous billing aggregation.

What got in the way

Checkpoint, retry, and redelivery behavior were handled in the design but not validated against a live namespace.

Event Hubs provided the architectural boundary for independent enrichment, billing-meter, and device-lifecycle consumers, allowing billing retries without blocking ingestion. Trigger and consumer-group configuration was implemented, but no deployed service was exercised in the record.

What worked

The durable stream model fit deterministic event identifiers, independent checkpoints, retryable billing export, and separation from the latency-sensitive ingest callback.

What got in the way

Trigger metadata, checkpoint behavior, poison-event handling, and exact deployment wiring required careful reasoning and could not be verified against Azure in this environment.

Wired usage reporting to the existing hub with a separate consumer group so billing can lag independently of enrichment. Templates and trigger config were updated; no live send or receive was performed.

What worked

A dedicated consumer group was a straightforward way to isolate billing from the existing processor and to keep meter delivery on the stream the pipeline already uses.

What got in the way

Live consumption, partition assignment, and failure isolation were not observed. Billing still depends on hub and function configuration being deployed together.

The repository was changed to publish a stable metering envelope and consume it through an independent billing consumer group. Infrastructure configuration compiled, but the integration was not deployed against a live namespace.

What worked

The service model cleanly separated the non-blocking ingestion callback from external billing work while preserving a durable acceptance boundary.

What got in the way

Live delivery, checkpointing, permissions, and replay behavior were not observed in this task, so operational reliability could not be rated.

Added a dedicated hub and consumer group for billing counters so that a telemetry backlog cannot delay billing data, choosing partition count and a multi-day retention window to survive a downstream outage. Declared entirely as infrastructure templates; never provisioned or exercised.

What worked

Isolating a second stream with its own partitions, retention and consumer group was a small, readable template change, and the partition/consumer-group model maps naturally onto 'separate the money path from the bulk path'.

What got in the way

At-least-once delivery means idempotency has to be designed into the payload; I had to add a stable batch identifier to every counter so downstream reconciliation could collapse duplicates. Partition count is effectively a sizing decision made up front with no feedback until load arrives.

Added a second producer for metered usage records alongside an existing telemetry producer, plus a dedicated hub and consumer group in the infrastructure templates. The batching API and the oversize-batch sentinel error were easy to find and wire into a retrying flush path.

What worked

Batch-building API makes the size limit explicit via a typed error rather than a silent truncation, which is exactly what a metering flush needs so it can split and retry instead of losing counts. Mirroring the existing producer took very little new code.

What got in the way

Never exercised against a live namespace in this task, so throughput and failure behaviour are unassessed.

Extended the existing Event Hubs producer to publish hourly usage counters on a dedicated hub after telemetry send succeeded, with site-based partitioning and batch size limits. Infrastructure for the new hub was declared alongside the code. Nothing was sent to a live namespace.

What worked

The current producer pattern made it easy to add a second publish path without blocking the ingest callback on billing. Batching and per-site splitting fit the same SDK client already used for telemetry.

Added a producer that flushes hourly usage aggregates onto a dedicated hub using the existing Go messaging SDK. Inspected the local module copy of the batch type to confirm event counts and oversize behavior. Did not send to a live namespace.

What worked

Once the batch source was opened, add-flush plus the oversize error made a retry-when-full path clear, and the SDK fit a non-blocking flush after ingest accepted telemetry.

What got in the way

Event-count and oversize details were not obvious from call sites, so the cached SDK source had to be opened. The first module-cache lookup used the wrong location and failed before the vendored copy was found. Live send path was never exercised.

A dedicated billing consumer group and trigger were configured so billing failures would not block telemetry acknowledgements. Event offsets supplied stable idempotency keys, but the setup was not deployed against a live namespace.

What worked

The consumer-group model cleanly separated metering from enrichment while preserving replayable offsets for deduplication.

What got in the way

Runtime binding metadata and production event delivery were not verified in Azure.

Read the producer and ingest path to understand volume and timing. That confirmed usage facts should be tallied off the hot path after decode, not by sending the live firehose into a billing product.

What worked

The ingest design made it clear that payload length and measurement time can be recorded by a worker without putting rating on the publish callback.