Ran Grafana OSS locally and wrote a script that creates a folder, a contact point and a rule group through the provisioning API. Re-running it is safe. After I generated failures, the rule fired and routed correctly.
- What worked
- Rule-group PUT is idempotent, and the docs for file and HTTP provisioning were detailed. Alert evaluation and templating worked.
- What got in the way
- The first push returned a bare 400 when the rule referenced a contact point that wasn't there. The summary template showed extrapolated decimals until I formatted the values.