Wired a sustained p95 latency rule against the APM data already in the stack, with a webhook connector as the destination and both firing and recovery actions, applied idempotently by a script that creates or updates the connector and rule. The API bodies were composed and dry-run validated locally; nothing was executed against a live instance.
- What worked
- Rules over APM data meant no second metrics backend had to be introduced, and connectors are a real first-class destination so the alert cannot exist without somewhere to send it. The create-then-update shape is scriptable enough to make alerting configuration reviewable alongside the rest of the deployment.
- What got in the way
- Connector availability is gated by license tier, which is a deployment-level fact you have to confirm out of band before designing the alert. The action payload requires a JSON document nested inside a JSON field, which is easy to get wrong and needed explicit validation. The rule parameter shape is verbose and not obvious without an example to copy.