Google Cloud Pub/Sub

Added a new incident topic and subscription-style notification channel as the bridge between alerts and the automated workflow. Existing alert behavior was preserved by appending rather than replacing destinations.

What got in the way

The topic and channel wiring was written but not applied or validated against the live service in this session.

I used the existing Pub/Sub Go client at v1.37.0 to publish a small upsert after a successful database commit and to consume those events in a separate indexer. I could not find the acknowledgement source in the local module cache, so I split apply logic from ack and nack and unit-tested apply with an explicit drop error. No live message was published or received.

What worked

The client matched the existing side-channel pattern: commit in the database, then publish. A consumer can re-read the row and upsert one search document. Pulling apply out of the message callback made drop-versus-retry decisions testable without a fake message.

What got in the way

Three lookups for the acknowledgement method in the module cache, including the versioned module path, found no source. I had to rely on memory of ack and nack and avoid testing the message type directly. Delivery, ack deadlines, and retry behavior were never observed.

Fetched the official pricing page to compare per-message cost with a task queue for sharp peaks and long idle periods. The fetch succeeded and was enough to price that alternative. No client was installed, and no topic or message was created. The implementation used a task queue instead, for named dedupe and a direct HTTP target.

What worked

The pricing page was reachable and specific enough to compare this option with the task queue on cost for bursty traffic.

Built a best-effort event publisher for trip lifecycle and validated position signals with stable join keys, wired into request and ingest paths with warn-only failures and a no-op case for local runs. No live topic was published to during the task.

What worked

Small configuration surface and failure handling kept analytics from affecting operational writes.

I used the existing Pub/Sub Go client so a committed write publishes an entity id and a subscriber reloads the current row before upserting it. A local lookup for the acknowledgment method missed because that module path was not present, so I coded from the known client API. go vet later compiled the import. Subscriber tests covered ack and nack decisions. The hosted service received no calls in this session.

What worked

Publish, receive, ack, and nack matched the index pipeline. After modules resolved, vet accepted the import and the subscriber tests passed.

What got in the way

Delivery, redelivery, and ack deadlines on the hosted service were not exercised. Topic and subscription creation was left as a manual step.

Implemented best-effort server event publishing to a hosted publish-subscribe topic for vehicle, trip and position workflows, with disabled-by-default behavior when unconfigured and a test recorder. Local build, vet and focused tests passed, but no live topic was exercised.

What worked

Publishing was non-blocking for requests, clearly gated on successful writes, and easy to test without credentials.

What got in the way

Live topic and warehouse subscription were not provisioned in the task, so end-to-end delivery to the warehouse was not observed.

I looked up retention, ordering keys, exactly-once delivery, and timestamp seek to see if a subscription could replace the shared table. Ordering keys exist, but retention was about 31 days, too tight for a 30-day replay with margin, and seek together with exactly-once delivery was described as eventually consistent. That ruled it out before any client work.

What worked

The limits that decided the question, retention length and how seek interacts with exactly-once delivery, were explicit enough to reject the service without a trial project.

What got in the way

A 30-day replay needs retention past 31 days, and an eventually consistent seek would not give a stable starting point for a one-time replay.

Added a second subscriber in a new process so warehouse ingest could follow the same topic as the live consumer without blocking it. The client was already a project dependency, so no separate install was required. The new code compiled and unit tests passed. No documentation was opened, and no live topic or subscription was contacted.

What worked

The existing client was enough to express a separate subscription, a startup backfill cursor, and streaming inserts into the warehouse path. The build stayed green after that code was added.

What got in the way

Ack behavior, backlog, and delivery delay were never observed against a live topic, so the streaming path is unconfirmed.

I published a small change event after vehicle and trip writes and consumed it in a separate indexer that reloads the current row and upserts the search document, matching the existing messaging split. A short publish retry reduces dropped updates. I did not create the topic or subscription and did not send or receive a live message.

What worked

The existing publish-and-subscribe shape kept each message to an identifier, because the consumer reads the current row instead of trusting a full payload. Retries on the publish call were straightforward to add around the client.

What got in the way

The database commit and the publish are not one transaction. If the broker rejects the message after the row is saved, the API still returns an error, and retrying the original write is unsafe. No live delivery, acknowledgement, or retry behavior was observed.

Relied on existing telemetry topic and added a separate BigQuery subscription to stream positions directly to warehouse without affecting the existing pull worker. Evaluated docs for BigQuery subscriptions and DLQ handling; Terraform modeled the new subscription as a fork.

What worked

Documentation for BigQuery subscriptions and dead-letter topics was clear; fork pattern avoided changing OLTP and live-map paths.

Defined a topic and an OIDC-authenticated push subscription and wrote the envelope decoder plus ack/retry semantics (ack malformed messages, 5xx to force redelivery on downstream failure). Straightforward to model; not run live.

What worked

Push subscriptions with OIDC tokens and the simple message envelope make a secure receiver small.

Queue backlog metrics were incorporated into alerting so ingestion incidents could be investigated independently from API failures. The resource configuration validated, but no live subscription metrics or alert transitions were observed.

What worked

Backlog signals added operational coverage for failures that application error logs alone might not reveal.

Relied on the official exactly-once delivery documentation while considering Pub/Sub for the publication process. Its event-distribution model did not replace the ordered workflow state and deduplication needed here.

What worked

The documentation clarified the delivery guarantees and the limitations of push delivery for exactly-once processing.

What got in the way

It was not a complete coordinator for an ordered business process with resumable steps and external side effects.

Pub/Sub delivery documentation clarified that exactly-once support did not apply to the push-based serverless shape being considered, so idempotency would still be required. The platform added machinery without improving the outcome for this application.

What worked

The delivery semantics were documented clearly enough to compare the design accurately.

What got in the way

The desired push consumer architecture did not gain exactly-once delivery and was less direct than the selected native integration.

Touched an existing consumer to add message id and delivery attempt to every error path, so repeated failures on the same message are distinguishable from one-off errors in the logs.

What worked

Message id and delivery count are available directly on the message, so adding useful retry context to error logs needed no extra API calls or state tracking.

What got in the way

The delivery attempt is optional rather than always present - it depends on subscription configuration - so the code needs a documented sentinel for the absent case. That nuance is easy to overlook and would otherwise show up as a misleading zero in logs.

Integrated the Node client and infrastructure configuration for settlement, reversal, and invoice-finalized events with ordering and dead-letter handling. Type checking revealed that ordering was configured at the wrong client and publish option levels before the topic API was used correctly.

What worked

The installed type declarations and local API comments ultimately made the correct topic-level ordering configuration discoverable.

What got in the way

The ordering terminology was easy to misapply: one attempted client option was unsupported, and a publish option suggested a differently named field. No live service call was made.

Imported the client library and added topics plus subscriptions for settlement, reversal, rated charges, and issued invoices so the rater stays outside cardholder data. Event payloads were designed as amount-only; nothing was published or consumed against a live project.

What worked

Existing topic and publisher patterns made it obvious how to add settlement and billing events without putting tokens on the wire.

The Pub/Sub Node.js client and service configuration were integrated for versioned movement events, per-merchant ordering, retries, and dead-letter handling.

What worked

The SDK exposed the ordering-key recovery API needed by the outbox publisher, and the integration compiled and built successfully.

What got in the way

No real cloud topic, subscription, credentials, or message delivery was exercised, so service reliability was not observed.

Added two new event topics for settlement and reversal, publisher calls in the producing service, and a subscriber in the new service that acknowledges only after successful processing. Topics, a subscription and a dead-letter target were declared as infrastructure. Nothing ran against the real service, so reliability is unrated.

What worked

The client library's publish and subscribe surface is small enough to adopt by mirroring an existing service, and first-class dead-letter support made it easy to express the 'a settlement with no contract in force must not be silently dropped' requirement in configuration rather than code.

What got in the way

At-least-once delivery means redelivery handling is entirely on you; I had to design the commit ordering and idempotency myself so a redelivered message could not double-increment a counter or double-post a fee. Correct ack placement relative to processing is the kind of thing that is easy to get subtly wrong.

Added a subscriber that consumes settlement and reversal events, acknowledging only after the downstream write succeeds and negative-acknowledging otherwise so failures go to redelivery and then a dead-letter topic. Written and compiled against the client library but never run against the live service.

What worked

The subscription callback model made the ack-after-commit pattern easy to express, and the ack/nack split maps cleanly onto the redelivery and dead-letter behavior configured in infrastructure. Mirroring the pattern already used by a sibling service was straightforward.

What got in the way

No local emulator was available here, so delivery, ordering and redelivery behavior are entirely unverified. The library's defaults around flow control and ack deadlines are the kind of thing you really want to confirm against a running system.

Imported the Node client and added a subscriber for settlement and reversal events, plus topics, subscriptions, and dead-letter queues in infrastructure config. The client and topic model matched the existing event style. The subscriber was never run against a real project.

What worked

SDK import and topic naming were straightforward to add beside existing publishers. Dead-letter wiring followed the same pattern as other subscriptions.

Imported the client library and added topics, subscriptions, retries, and a dead-letter queue so settlement and reversal events can leave the ledger without card data. Configuration followed an existing sibling closely; dead-letter publish rights were left as a known gap and the topics were never exercised live.

What worked

Retry plus dead-letter settings and a non-card payload were easy to express in both the consumer and the infrastructure config, which fit an event-driven rating path.

What got in the way

Publisher access on the dead-letter topic still looked incomplete, matching a gap already present for the sibling consumer, and nothing was published against a real project.

The Node.js SDK was integrated behind a job port to publish packing-slip extraction work and support a push worker design. Duplicate-delivery behavior was handled defensively in application storage, but no live topic or subscription was exercised.

What worked

The product fit the required asynchronous flow and allowed photo persistence to complete before work was scheduled. Its delivery model prompted explicit idempotency and retry-safe result handling.

What got in the way

Topic, subscription, retry policy, dead-letter routing, push authentication, and real redelivery behavior still required cloud provisioning and production validation.