Worked against an existing log store and alert rule: documented a replacement query that parses structured fields instead of pattern-matching a text log format, plus a transitional query matching both during rollout. Query changes were written into project documentation rather than applied, since the alert rule lives in the hosted UI.
- What worked
- Label-plus-JSON-parse queries are much more robust than regex over a text format, and moving to structured logs makes alert thresholds expressible against real numeric fields. The range query API is a sensible interface for an automated investigator to pull an incident window.
- What got in the way
- The sharpest problem is silent failure: an alert whose query stops matching after a log format change simply goes quiet, which reads as healthy. Nothing in the rule surface flags a query that has matched nothing for an unusual stretch. Rules also live outside the repository, so a code change and its alerting change cannot ship atomically — that ordering risk had to be managed by hand with a dual-matching transitional query.