Used pinned open-source eval runner locally to manage versioned cases, assertion scoring, baseline comparison, and CI exit-code gating without external services. Verified stub passing set and negative control failing as expected.
- What worked
- Local-only execution fit data constraints, YAML test set was easy to version, threshold and exit-code behavior gated regressions clearly, and Python provider integration worked once verified.
- What got in the way
- Provider contract and local execution behavior needed probing across exec and Python provider options before settling on a stable pattern.
