Prometheus

Reused the existing metrics setup and extended alert definitions for queue depth and export failure. Configuration editing was straightforward; verification was limited to reading existing files because no live monitoring run was shown.

What worked

Existing scrape and alert patterns made it easy to place the new worker and failure signals alongside current coverage.

What got in the way

Alert and scrape edits could not be machine-checked or observed against a live monitoring server in the record.

Authored scrape and alert definitions for rejection rate with a reproducible local threshold check. Config shape was clear, but rules never ran under the real evaluator or live traffic.

What worked

Rule and scrape concepts were well documented enough to draft without a live account.

What got in the way

Rule unit testing and live firing were not possible without the cluster tooling, so the alert threshold is validated only by a local script.

Used the rule checker to validate alert definitions and run unit tests for firing and quiet series. An early failure came from mixing operator custom-resource shape with plain rule syntax, fixed by keeping a generated plain-rules fixture plus a sync check.

What worked

After aligning the file format, rule checks and rule unit tests both passed and caught the intended alert behavior.

What got in the way

The checker rejected the operator-style rule file at first, requiring a format rework.

Installed Alertmanager 0.27.0, validated its routing config, and used it to deliver a firing alert to a local operator webhook. The config check passed and the notification arrived.

What worked

The config checker passed, and a fired alert was posted to the configured webhook during the local reproduction.

Added a backlog alert for the new export queue alongside the existing worker alerts. Configuration files parsed cleanly, but alert firing was not tested against a live monitoring stack.

What worked

Existing alert patterns were easy to extend for the new queue.

Inspected existing scrape and alert configuration to choose an approach that fit current operations. Noted that the existing queue backlog alert did not yet cover the new queue. No alert or scrape changes were made in the record.

What worked

Existing configuration was readable and helped avoid adding unmonitored infrastructure.

Installed Prometheus 2.55.1, used promtool to check and unit-test the alert rules, and ran the server so a failure metric could page an operator. Rule tests succeeded and the local evaluation path fired.

What worked

promtool accepted the rule file and the rule unit test. The local server scraped the application and fired the intended alert.

Ran Prometheus locally with its OTLP receiver enabled as a stand-in for Grafana Cloud's metrics store. The app pushed metrics to it directly, and Grafana evaluated the alert against it.

What worked

Enabling the OTLP receiver with one flag made a local end-to-end alert test possible.

Wrote a routing tree with per-team PagerDuty receivers, severity mapping, business-hours time intervals for a lower tier, a fallback receiver, and inhibit rules. Used amtool check-config and amtool config routes test to verify the config standalone, embedded in a GitOps Application, and as rendered by the operator chart. Routing tests matched expectations for every case.

What worked

The routes test subcommand gives immediate, concrete proof of where a given label set lands, which is far better than reasoning about matcher precedence by hand. File-based secret references kept credentials out of the repo.

Authored multi-window burn-rate recording and alert rules derived from a service tier, plus collector health rules, and validated them with promtool check rules after extracting them from rendered Kubernetes manifests. All rule groups passed; feedback was clear.

What worked

promtool is a single binary that validates PromQL and rule structure quickly, giving confidence in templated rules without a running server.

What got in the way

Had to be downloaded from a release archive. Rules embedded in CRDs need to be extracted into plain rule files before checking.

Configured Prometheus 3.x with the OTLP receiver and exemplar storage flags as the metrics backend for the local stack, and wrote the alert PromQL against it. Not run here.

What worked

Built-in OTLP receiver avoids a scrape config for the app; exemplars tie metrics to traces.

What got in the way

The OTLP metric name translation (unit suffixes, underscores) determines the exact metric names the alert must reference, and this depends on server configuration that can differ between Prometheus and Mimir; I had to flag that as an assumption rather than verify it.

Used the registry and HTTP handler to serve metrics from the OpenTelemetry Prometheus exporter on a separate listener. Minimal code, worked first time, and the exposition format was easy to assert on in tests.

What worked

Simple registry/handler API that composes cleanly with the OpenTelemetry exporter.

Added one alerting rule for the new queue by mirroring an existing rule on the same exporter metric, with a deliberately loose threshold and long for-duration to tolerate bursts. Rule file was edited only; not loaded into a running server or validated with promtool.

What worked

Rule YAML is terse and the existing rule served as a direct template.

Downloaded promtool and used check rules on Helm-rendered recording/alert rules, then wrote a rule unit test feeding synthetic traffic to verify a multi-window burn-rate alert fires with the right routing labels. The check and test flow was essential for shipping alerts with confidence, but the unit test was nondeterministic until I restructured the scenario.

What worked

Rule unit tests let me prove the alert path offline, including label routing and the annotation text. The series expansion shorthand kept test input compact. Check rules caught nothing wrong but gave a clear count of parsed rules.

What got in the way

With a recording-rule group and an alert group on different evaluation intervals, the alert's $value flipped between two values across identical runs, apparently depending on which group evaluated first. I had to redesign the scenario to a constant error ratio to make it deterministic. Rate extrapolation also made hand-computed expected percentages wrong; the behavior is documented but easy to trip over. Binary had to be fetched from a release tarball.

Downloaded Prometheus tooling and used promtool to test alert rules, check rule definitions, and validate scrape configuration syntax. The recorded local checks passed.

What worked

Alert behavior could be tested before connecting the rules to a live monitoring service.

Configured Alertmanager so the mismatch rule could actually notify. A Grafana webhook path was wrong at the default URL, so the first cut used a no-op receiver; later a small HTTP sink was added. Nothing was fired live.

What worked

Once a dedicated receiver existed, the routing model was simple enough for local compose and in-cluster placeholders.

What got in the way

Grafana as a webhook target did not accept alerts at the assumed path, which delayed having a real delivery target.

Used official metrics and alerting docs to enable remote write, load a mismatch recording rule, and keep a curl-checkable expression beside Grafana-managed alerting.

What worked

Rule YAML and the remote-write receiver were straightforward to place in both local compose and cluster manifests.

What got in the way

Rule evaluation was never observed on a running server.

Shipped scrape config and a recording of one actionable mismatch rule as the canonical alert source, preferring Prometheus over Grafana-only provisioning to avoid datasource UID fragility. Rules were not evaluated live in this environment.

What worked

File-based rules expressed a single, reproducible alert with runbook text without needing a Kubernetes CRD.

What got in the way

Cluster layout needed a separate Prometheus config snippet because environment-variable overrides were impractical.

Added a notification backend so the provisioned UI contact point had somewhere to send the mismatch alert without paging twice from two evaluators.

What worked

Pairing it with Grafana-managed routing avoided duplicate pages once Prometheus-side notification was turned off.

What got in the way

Delivery was never seen against a live receiver, and the first config was only a placeholder sink.

Wrote Prometheus scrape config for the Actuator endpoint, including Kubernetes service discovery, retention, and exemplar flags, and used those series in a Grafana alert. The server image was pinned but never launched.

What worked

Static and Kubernetes scrape jobs mapped cleanly onto the Actuator Prometheus endpoint, and a PromQL-style alert on a rising failure counter was straightforward to express.

What got in the way

A regex meant to detect a non-zero counter was noted as missing the zero case until tightened. Scrape and alerting behavior were not observed at runtime.

Wrote a PromQL availability rule and a unit-test file, then ran the rule tester from a published release archive because the binary was not already installed. The rule test passed and gave confidence in the one actionable alert.

What worked

The rule-test CLI accepted a straightforward test fixture and confirmed the burn-style expression without standing up a server.

What got in the way

The tester was absent on the machine, so a release archive had to be downloaded before the alert could be validated.

Used the rule checker and unit-test runner to lock a gateway availability burn alert as a local PromQL contract. Installing via Go modules failed; a release tarball then ran the checks cleanly.

What worked

Once the binary was on PATH, rule checking and the fixture test both passed and gave a reproducible alert contract independent of a live ruler.

What got in the way

Building the CLI from modules failed because of replace directives in the upstream module, so the Go install path was not usable here.

Wrote a PromQL latency SLO rule and a unit-test file, then downloaded the rule-checker CLI and ran it. Histogram quantile interpolation and YAML naming (colons, optional name fields) caused false mismatches until tests were rewritten. After that the CLI confirmed pending versus firing behavior.

What worked

Once YAML was valid, the rule unit-test CLI checked for-duration and threshold behavior without paging anyone. PromQL could express a route-excluding latency SLO against histogram buckets chosen for the SLO.

What got in the way

The checker was not on PATH and had to be fetched. Unquoted colons in test names broke YAML. Quantile interpolation did not return the exact bucket edge the test first expected. An older CLI might ignore test name fields.