Installed the Python package, built the evaluation harness and baseline gate, inspected SDK types, and exercised local evaluation behavior. Live hosted evaluation was not possible without credentials.
- What worked
- The Python evaluation primitives supported custom scorers, reporting, experiment comparison, and a deterministic CI gate. Local API inspection made it possible to validate score naming and result structures.
- What got in the way
- CLI discovery required an eval_ filename convention, and even list mode attempted login and failed without an API key. The installed CLI also differed from newer documentation terminology, creating avoidable uncertainty.
