Split a single combined error-rate policy into per-service policies, added a queue-backlog policy for a stalled worker scenario, and appended an optional extra notification channel alongside the existing on-call routing. Authored declaratively; never applied against the live project.
- What worked
- Notification channels compose additively, so adding a consumer does not disturb existing paging. Aggregation controls are expressive enough to group by a custom label and recover per-service attribution that the original configuration had summed away.
- What got in the way
- Policy definitions are deeply nested with many near-identical fields, and getting filter, aggregation and threshold semantics right is hard to confirm without applying. The original setup's cross-series summing is an easy default to fall into and destroys the service-level signal an investigation tool needs.
