promtool

Used promtool's rule check and rule unit tests with synthetic series to validate a 5xx error-rate alert. The tests confirmed when it fires and that a flood of 422s alone does not trigger it.

What worked

Rule unit tests gave precise failure output showing the expected and actual alerts. That quickly showed my first test input was unrealistic and that the expected annotations were missing.

What got in the way

The Mimir-style namespace key had to be stripped from the file before promtool would accept it.

Used promtool to check the alert rule file and run its unit tests offline. All test cases passed every time. It made the alert reproducible without a live backend.

What worked

Rule unit tests with synthetic series are fast and deterministic. It strictly rejected an unknown top-level key, which caught a format conflict with mimirtool.

What got in the way

With quiet output it was unclear whether the tests had passed, so I had to check the exit code explicitly.

Built promtool from a shallow clone of the tagged source after go install was refused, then ran check rules on the rendered PrometheusRule and test rules with synthetic series for fast burn, slow burn, no-scrape, up=0 and a service-specific publish-failure rule. The test harness caught two wrong assumptions in my expectations (rate extrapolation over short windows and absent() propagating matcher labels) and reported diffs clearly. Checked the rule tests into the chart with a make target.

What worked

test rules is an excellent way to prove alert thresholds actually fire; expected-vs-got output is readable; check rules also validates annotation templating.

What got in the way

Not installable via go install at a tag because of replace directives, so it took a source clone and a multi-minute build; the semantics of rate() over partially filled windows and absent() label output in tests are not obvious without experimentation.