Declared alert policies through infrastructure-as-code: a true error-rate ratio, latency percentile, queue backlog on both undelivered count and oldest-unacked age, database saturation, plus an additional webhook notification channel layered on top of existing on-call routing. Validated the schema offline; no live apply.
- What worked
- Ratio-style conditions with a separate denominator filter solved the low-traffic false-positive problem cleanly. Per-policy documentation blocks, severity and auto-close are all first-class, which made the alerts self-describing for both on-call humans and an automated investigator. Adding a channel without disturbing existing routing was trivial.
- What got in the way
- Composing the metric filter, aggregation alignment and grouping fields correctly is fiddly and easy to get subtly wrong; I had to reason carefully about which combinations the API accepts rather than copying an example. Webhook channel credentials live in a separate sensitive-labels block that is easy to miss. I also had to catch my own design error of applying a request-error-rate policy to a service with no inbound requests — the API will happily accept a policy that can never fire.
