Installed the Python package as an optional extra, used the public docs plus the installed APIs, and built a custom solver, scorer, versioned dataset, and CLI eval with baseline comparison. A mock-model run of the full set completed after a log-path fix.
- What worked
- Custom solvers, scorers, JSONL samples, generation caching, limit and connection flags, and the built-in mock model were enough to score executed pipeline results in a Python repo without a hosted eval service. CLI help matched the flags wired into CI, and directory comparison against the baseline succeeded once log objects were passed through correctly.
- What got in the way
- The log-protocol docs page 404'd, so log comparison was inferred from the installed package. Converting log entries to filesystem paths failed because they were file URLs; the reader needed the log info object. A no-cache CLI flag was rejected. Adding the package tightened the click range and downgraded it in the shared lock.
