Installed Inspect AI as an optional extra, read public docs plus the installed package APIs, and built a custom solver, versioned JSONL set, deterministic scorer, and log-based baseline compare. Tests ran against a mock model; the eval CLI was invoked once and failed closed when local data was absent.
- What worked
- One optional-extra install covered local datasets, run logs, a mock model for tests, and programmatic log reading. That was enough for scoring, baseline comparison, and CI gating without standing up extra services.
- What got in the way
- The eval CLI loads the task file outside the package and changes the working directory, so relative imports broke and had to be rewritten. A model flag was still required even though the custom solver never called generate. Public docs were incomplete for logs and scorers, so much of the wiring came from reading the installed package.
