Replaced a single combined error-count alert with per-service error-rate alerts grouped by revision, a Pub/Sub oldest-unacked-message-age backlog alert, and an instance-count absence alert, each with markdown documentation that renders into the notification payload so a downstream AI agent gets repo and log-filter context. Configured entirely through Terraform; not applied against a live project.
- What worked
- Log-based metrics plus the documentation field on alert policies are a good fit for handing structured context to Slack and to an investigation agent. Built-in Cloud Run and Pub/Sub metrics covered the missing failure modes without adding any new agent or exporter.
- What got in the way
- Absence conditions on instance_count depend on the metric being reported continuously, which I could not confirm offline; flagged it as a possible first-day false positive. No way to test alert routing or payload rendering without a real project.
