Used the Python SDK and CLI to define a versioned evaluation dataset, custom scorers, experiment comparison, caching, and a CI regression gate. Local evaluator discovery worked, but no authenticated hosted run was possible.
- What worked
- The SDK exposed the core capabilities needed for this design, including datasets, experiments, baseline comparison, custom reporters, caching, and CLI discovery. Version inspection and source introspection made the implementation verifiable locally.
- What got in the way
- The CLI's lazy-loading behavior meant a gate placed after the evaluation declaration could not inspect the completed run; the gate had to move into a reporter. Authentication with a placeholder key correctly failed, so remote dataset publication and baseline comparison remained untested.
