Pinned 0.123.1 in an isolated package after a same-tree install failed on a Zod peer clash. Docs on custom providers, the CLI, sampling, exit codes, and CI were enough to load a YAML config, a TypeScript provider, generated cases, and JavaScript assertions. The on-disk result shape and custom-provider caching differed from the docs, so parsing and caching were aligned to the installed build. A smoke run failed closed on an invalid credential, as intended.
- What worked
- The CLI loaded a default-exported TypeScript provider and file-based test generation. Metadata filters ran before sampling. Provider errors and assertion failures both exited 100, and numeric failure reasons in the JSON separated them. An echo provider confirmed the scoring function once the eval runner had loaded it. A named cache export was callable from the provider. Telemetry and sharing could be turned off with flags and environment variables.
- What got in the way
- The package requires Zod 4, which clashes with the model client's optional Zod 3 peer, so it could not be installed in the application tree. Docs described an outputs array; the result file used a results array. Built-in HTTP caching does not wrap a custom provider's callApi, so the pipeline had to cache itself. Assertion failures still set an error string, which made retry classification easy to get wrong. The programmatic assertions helper does not load file paths the way the eval command does, and a JSON prompt in YAML was treated as a file path. Several answers were only in the bundled sources, under hashed filenames.
