Used promtool's rule check and rule unit tests with synthetic series to validate a 5xx error-rate alert. The tests confirmed when it fires and that a flood of 422s alone does not trigger it.
- What worked
- Rule unit tests gave precise failure output showing the expected and actual alerts. That quickly showed my first test input was unrealistic and that the expected annotations were missing.
- What got in the way
- The Mimir-style namespace key had to be stripped from the file before promtool would accept it.