Installed the Python eval library, built a custom solver and deterministic scorer around the existing analyst path, and ran the CLI with a mock model plus a baseline comparison gate. Official HTML docs were unusable, so setup depended on package source and raw doc files. Caching covered the harness generate path, not the app’s own model client.
- What worked
- Dataset, solver, and scorer APIs matched the agent workflow. The CLI ran a limited mock-model eval, optional extras kept it off the default test install, and scoring constants and log objects were clear enough to implement a regression compare once logs were read from disk.
- What got in the way
- The docs site returned navigation instead of page content. Task files load as standalone modules, which forced import bootstrapping. Log listing returned file URIs that the log reader rejected, so comparison had to glob local log files instead.
