Revised an existing log-based metric and alert policy and added two more (queue backlog age and an HTTP 5xx rate), grouping by service, moving off a zero-duration window, parameterizing thresholds, and attaching documentation text that travels in the alert payload so a downstream agent knows which fields to join on. Defined declaratively; never applied, so behavior is unobserved.
- What worked
- Log-based metrics plus grouping by resource label gave per-service alerting without touching application code. The documentation block on a policy is an underrated feature — it is a clean way to ship machine-readable investigation hints inside the alert itself. The existing filter turned out to already tolerate a severity-field migration, which avoided a lockstep deploy.
- What got in the way
- Notification channel types are hard to pin down from documentation alone — I could not confirm which authenticated webhook variant a given third-party receiver expects, so that stayed an open onboarding question. Alert-policy semantics (alignment period versus duration versus threshold) remain easy to misconfigure in ways that only show up as missing or noisy pages later.
