Installed inspect-ai as a pip extra and built a task with a JSONL dataset, a custom solver that calls the application's own pipeline, a custom deterministic scorer, and multi-epoch runs. Ran it both programmatically via eval() and through the inspect eval CLI, and read .eval logs back with read_eval_log to inspect failures and to replay a log through a baseline gate. Everything worked as a pure local dependency with no server or telemetry, which fit the on-prem requirement well.
- What worked
- The task/solver/scorer decorators were easy to compose; model="none" let the solver drive its own model calls while Inspect handled dataset, epochs, logging and scoring. Log files are self-contained and easy to read programmatically, so verifying scorer defects versus real model errors was quick. Epochs and per-sample scores mapped cleanly to a pass-rate baseline.
- What got in the way
- Had to introspect the installed package source to confirm that the "none" model exists and to check Sample/TaskState/EvalLog field names; those details were not obvious from memory. The CLI loads the task file as a standalone module, so relative imports inside the task file broke and had to be switched to absolute imports. Python 3.11 resolved to a slightly older release than the latest on PyPI.
