Pinned Promptfoo 0.123.1 and ran it as a CLI with a Python provider, Python assertions, and a YAML suite. Docs plus the installed package were used to check exit codes, sharing, telemetry, metadata filters, and response caching. A zero first-n filter was rejected. A later run loaded the config, selected every regression case, and exited 100 when the model client failed.
- What worked
- The pinned CLI honored no-share and used exit code 100 for provider or assertion failures, which matches a pull-request gate. Python assertions can score structured outputs. The disk cache skips error responses, and the installed build treats the telemetry-disable environment flag as on.
- What got in the way
- A secondary writeup described a compare flag that official docs did not support, and the GitHub Action docs describe file diffs rather than a score baseline between runs. A first-n filter of 0 fails validation, so it cannot dry-run the suite. Metadata matching is a substring check. Cache keys include serialized provider options, so unstable options miss the cache. No successful scored run was observed.
