Used promptfoo as a local eval CLI with a custom TypeScript provider that wraps a real pipeline, JS assertions, and a model-graded faithfulness check. Ran it end to end against a mock OpenAI server. The latest version needs Node 22.22+ and depends on zod 4, which clashed with the app's openai peer dependency on Node 20, so I put it in its own package. The install was about 2.7 GB.
- What worked
- TypeScript provider files loaded without extra setup. Environment switches for telemetry, sharing, update checks, remote generation and failed-test exit code made it suitable for running offline on private data. The JSON results output was easy to post-process into a baseline comparison, and the built-in OpenAI grader respected the base URL override.
- What got in the way
- Peer and engine conflicts blocked installing it as a root devDependency. The dependency tree is very large and npm audit flagged high-severity advisories in transitive optional packages. Omitting optional deps broke esbuild. It still created a local SQLite db with --no-write. Results have no repeat index, so I had to group repeats by metadata. I found settings by grepping the dist code rather than in the docs.
