Used promptfoo as the eval harness for a two-stage LLM pipeline: custom TypeScript provider, file-based tests, JavaScript assertions, an llm-rubric grader, repeats, and JSON output that a comparison script reads. The latest release needed a newer Node than the project used, so I pinned an older version and added a dependency override to settle a zod peer conflict. After that, full runs against a local mock OpenAI server worked end to end.
- What worked
- Loaded TypeScript providers, tests and assertions directly through tsx without a build step. Custom providers and named JavaScript assertions with metrics were flexible enough to score each stage separately. The JSON result file had a clear structure (per-result success, named scores, per-assertion reasons), and the CLI flags for repeat and output behaved as expected.
- What got in the way
- Recent releases require Node 22, which ruled them out for a Node 20 project, and finding the last compatible version meant going through registry metadata by hand. Its zod version clashed with an optional zod peer of the openai package. To learn what context custom assertions receive and how TS files load, I had to grep the bundled dist and type files. Installing it added a high-severity advisory to the dev dependency tree.
