Installed the Python SDK, wrote an Eval with a custom Reporter that fails the run on absolute and baseline-relative score drops, and ran the CLI offline with --no-send-logs and --list plus a scripted fake model. Core concepts (Eval, scorers, trial_count, Reporter, automatic baseline from git ancestry) covered every requirement, but I had to read the SDK source to confirm signatures, exit-code behavior, and file discovery because the docs were thin on those details. Never ran against the hosted service.
- What worked
- Python-native Eval API mapped cleanly onto an existing pipeline. Reporter hook made a CI gate straightforward. Baseline experiment resolution from git ancestry needs no extra bookkeeping. --no-send-logs allowed full offline validation of the harness, and the CLI's JSONL summary output was easy to keep compatible with.
- What got in the way
- The CLI loads eval files via spec_from_file_location without registering them in sys.modules, so relative imports and @dataclass with deferred annotations both fail with confusing errors. Reporter and summary type signatures, the eval_*.py discovery rule, and the exit-code contract were only discoverable from source. Hosted baseline behavior could not be verified without an account.
