Installed inspect-ai 0.3.223 and used it as a library for a JSONL dataset, custom solver wrapping the production pipeline, deterministic scorers, log-based scoring, and a baseline comparison CLI. Official pages for solvers, scorers, datasets, scoring, logs, and the model API were useful; two other doc URLs 404ed, so TaskState, EvalLog, and eval() details came from the installed package. Stubbed eval tests passed with mockllm; a live run was not executed.
- What worked
- Custom solvers and scorers mapped cleanly onto a workbook-plus-question pipeline. Git-versioned JSONL, scored logs, mockllm, and Python eval() were enough to unit-test the harness and pin a baseline without standing up a platform.
- What got in the way
- Doc URLs for solver and eval pages returned 404, so API shapes had to be read from site-packages. Inspect still expected a model even though generation went through the app’s own client, which added mockllm and extra wiring.
