Pinned the CLI and ran a local eval that scores a model pipeline through the Python provider, with sharing, telemetry, update checks, and the hosted grader turned off. Provider, assertion, and result-file docs took several passes because the output envelope has more than one shape, so baseline comparison is a small parser over that JSON. A smoke run with a rejected model call exited non-zero and recorded a generic failure.
- What worked
- Installing from the lockfile with scripts disabled produced a working binary. Version and eval help both responded. The smoke run loaded the provider, failed closed when the model call was rejected, and wrote a short generic failure instead of echoing fixture contents.
- What got in the way
- The provider entry point, assertion hooks, and JSON result layout were not settled from one page. Provider failures exit 100, which is easy to treat as an ordinary assertion failure, and the console summary dropped the upstream error body. Telemetry and sharing stay on unless disabled. The install audit reported three high-severity issues at this pin.
