Pinned and unpacked 0.123.1, then ran a one-case eval through a custom Python provider with sharing disabled and the cache off. The config loaded, assertion modules loaded through the bundled Python wrapper, and a missing model credential was reported as a case error with a non-zero exit after a few seconds. Whether cached results keep cost, and how a configured Python path is resolved, was not clear from the docs, so those contracts were taken from the minified provider and evaluator bundles.
- What worked
- The eval command accepted a config path, a first-N filter, no-share, and no-cache. The provider started under the Python interpreter environment variable. The unscored case was reported as an error rather than a pass, and the process exited non-zero, which is the fail-closed signal a pull-request check needs. Result files stayed outside the repo. A synthetic provider payload loaded the custom assertion modules through the package wrapper.
- What got in the way
- Confirming that the cost grader rejects a missing cost, and that cache hits keep cost while rewriting token counts, required reading minified build files. A relative Python path is handed through as configured and resolved from the process working directory, which is easy to mis-set on a runner. A live scored run, a baseline comparison, and a cache hit were not executed.
