Used as versioned in-repo eval framework with two configs, sampled PR runs, deterministic assertions, and CI failure on regression. Config resolution and stub-provider gate checks passed after pinning to a working line.
- What worked
- File-based dataset and config versioned cleanly, sampling flags kept PR cost predictable, exit-code gating blocked regressions in stub checks, and local provider resolution worked.
- What got in the way
- Latest major line conflicted with existing major AI SDK dependency and one intermediate release had a broken local migration, requiring version hunting and an exact pin.
