Pinned the Promptfoo CLI at 0.123.1 and used it as a local runner for git-stored cases, a custom provider, and JavaScript graders. The config loaded, assertions resolved, and a provider failure exited 100 with sharing disabled. Docs for this pin advertised a flag the CLI rejected, Node 20 is no longer supported, and a zero-weight rubric still fails the run if the grader throws.
- What worked
- npx installed the pinned CLI, the config parsed, and the loader resolved a TypeScript provider and assertion functions, including a double-wrapped CommonJS default export. Provider errors failed closed with exit 100. Sharing, telemetry, and the response cache could be disabled. A zero weight turns a returned failing assertion into a pass, and a first-N filter worked for a smoke run.
- What got in the way
- CI docs for this generation advertised a fail-on-error flag that 0.123.1 rejected. Node 20 support ended in 0.122.0, so the eval job needed a newer runtime than the app. Built-in rubric errors are rethrown before the weight check, so a grader outage fails the case. Loader and transform behavior had to be confirmed in the published package source because the docs did not match the pin.
