Installed the Python SDK, built an Eval() runner with custom deterministic scorers, trials, tags and a regression gate that reads the summary comparison against a baseline experiment. Ran it offline in no-upload mode (mock and against the real model) successfully. Could not exercise the actual upload or baseline comparison because no API key was available.
- What worked
- The Eval() API is compact and the offline/no-send-logs mode made it possible to validate the whole harness without an account. Scorers bound by keyword name (input, output, expected, metadata) are convenient, and the summary object exposes per-score diffs against the comparison experiment, which is exactly what a CI gate needs. Automatic selection of the nearest ancestor experiment as baseline fits a push-to-main workflow.
- What got in the way
- I had to read the installed source to learn how baseline selection, git-ancestor lookup, caching flags and scorer argument binding actually work; the docstrings and public typing did not make the summary/comparison structure obvious. Baseline comparison across experiments with different trial counts and the real upload path remain unverified. The regression gate had to be designed carefully so an infrastructure failure is not misreported as a regression.
