Ran Grafana locally and used file provisioning to set up datasources, a dashboard, an alert rule, a contact point read from an env var, and notification policies. A drill made the alert fire and later resolve, and both messages reached a stand-in webhook. Provisioning has some quirks that only showed up once Grafana was actually running.
- What worked
- Alerting handled the full cycle: pending, firing, resolved, plus a DatasourceError page when the metrics store went down. The provisioning and contact-point HTTP APIs made it easy to check that the env-var webhook URL and the datasource settings resolved correctly. Trace-to-logs correlation settings worked once configured.
- What got in the way
- The alert rule was rejected because a dashboard UID annotation also needs a panel ID. Env-var expansion applies to some provisioning files and not others: datasource files need $$ escaping, but alert-rule annotations must not be escaped, which caused an unrendered summary. Alert state kept in the data directory across restarts made a re-run misleading until I wiped it.