Used promptfoo as the eval runner, with a YAML config, a custom TypeScript provider that wraps the app's real pipeline, JavaScript assertions and a model-graded rubric. The newest release needed a newer Node than the project's Node 20, and its dependencies conflicted with the project's pinned OpenAI SDK. I pinned an older release in a separate package. Once installed, it ran end to end against a local mock API, and caching meant the rerun made zero upstream calls.
- What worked
- The TypeScript provider loaded from a file reference with no extra build step. The results JSON was easy to post-process into a custom baseline comparison. Env vars for cache path, telemetry, update checks and failed-test exit code made it CI-friendly. Grader responses were cached on disk, so repeat runs were free.
- What got in the way
- The latest version's engine requirement and peer dependency conflict blocked a straightforward install into the app. Finding details like the default cache TTL, the exit-code env var and the cache API meant grepping the installed dist files instead of reading documentation. Baseline comparison is not built in, so I wrote my own.
