Installed inspect-ai as a Python extra and used the library plus CLI to define a custom solver, gold-answer scorer, versioned JSONL set, generation cache, and log scorecards for CI. Several official doc pages 404'd, so APIs were confirmed from the installed package. A mock-model eval and unit tests succeeded; a live provider run was not executed.
- What worked
- Tasks, solvers, scorers, JSON datasets, eval logs, cache flags, and the built-in mock model mapped cleanly onto a two-call generate-then-execute pipeline. After install, CLI help and generate caching matched what CI needed.
- What got in the way
- Doc URLs for datasets, custom solvers, and custom scorers returned 404, which forced reading installed sources. Binding a model on the Task constructor required provider credentials at import time. The package pulled a large AWS/S3 tree even though logs stayed local.
