Pinned Promptfoo 0.120.19 and ran it as a local release check with a custom TypeScript provider and JavaScript assertions. Docs covered providers and file assertions, but caching, assertion invocation, env paths, and exit codes were only clear from the installed source. Install hit an optional zod peer conflict and needed legacy peer resolution. Newer releases require Node 20.20+, so this was the newest pin still compatible with a broad Node 20 range. Config validation succeeded, and a run without a model key failed closed with exit 100. A filter matching no cases still exited 0, and expectation variables made the results table unreadably wide until they were moved to metadata.
- What worked
- The validate command checked the suite without calling a model. Named assertion functions referenced from the config resolved. The default bar is a 100 percent pass rate, and errors or failed checks exit 100. Sharing defaults to off, and the cache stays outside the repo. Arbitrary metadata is allowed, which kept the results table to the brief and fixture instead of every expected fact.
- What got in the way
- Custom providers are not cached unless they call the cache helper. The results table turns each variable into a column. A filter that matches nothing still exits 0. Env files are resolved beside the config, not the process working directory. Update checks contact the vendor unless disabled. The install added hundreds of packages, reordered the lockfile, and conflicted on a zod peer. Releases after this pin drop plain Node 20 support.
