Used Inspect AI as the eval framework: a task with a custom deterministic scorer, a Docker sandbox for model-written code, a response cache, and a gate script that reads eval logs. An end-to-end smoke run with the mock model and local sandbox scored correctly, and an auth failure gave a log with error status instead of raising. Never ran it with the Docker sandbox or a real model.
- What worked
- The mock model plus the local sandbox let me test the whole pipeline without an API key or Docker. eval() returns logs with a status field instead of throwing, so separating 'regression' from 'could not run' was simple. Scorers that return multiple metrics and the built-in model cache fit a cost-controlled PR gate well.
- What got in the way
- I had to grep the installed source to confirm details like sandbox exec/write_file signatures, the scorer metrics formats, the cache directory env var, and EvalLog error fields. Its pin on an older click forced a downgrade of a production dependency in the shared lockfile.
