Used a pinned release through a one-shot runner to execute a versioned YAML test set with a custom provider and deterministic plus model-graded assertions, followed by a baseline comparison and CI gate. Core behavior worked, but provider export shape and array variable expansion needed empirical probing.
- What worked
- Offline runs with stub providers confirmed pass, fail, and error paths and exit codes for CI blocking, and telemetry and sharing could be disabled for private data.
- What got in the way
- Initial provider and assertion wiring failed until the expected class export and file-based loading were discovered, and top-level array variables expanded into extra runs until fixtures were nested. Dependency major-version conflict required avoiding a local install.