Pinned the eval CLI in its own package, then used config validation, a custom provider, and the cache helper to score a multi-step model pipeline with a small sampled suite. Provider and output docs were enough to start, but baseline regression in CI, cache-key safety, and the JSON result shape only became clear from the installed package. Config validation and a module import succeeded; no live model run was possible without a credential.
- What worked
- A JavaScript provider, external assertion files, and test filters fit a multi-step pipeline and a cheap pull-request sample. Config validation succeeded, telemetry and update checks could be turned off, and the cache helper accepted an explicit key so the credential did not have to be part of the cache identity.
- What got in the way
- Installing it beside the app failed on a zod 4 versus zod 3 peer conflict, and the current release no longer supports the app's Node 20 runtime, so the CLI had to be isolated and pointed at a newer runtime. The GitHub Action only scores the current checkout, and documented result JSON uses more than one pass field, so baseline comparison and output parsing had to be built by reading the published bundle. Default caching also refuses a header callback unless a custom cache key is supplied.
