Used Inspect AI as the eval framework: a task with a custom solver that calls the app's own pipeline, custom scorers, multiple epochs, and a Docker sandbox for generated code. Ran it end to end in-process with a stubbed model and the local sandbox; the CLI loaded the task file. Never ran it against a real model or the Docker sandbox because Docker and a usable key were unavailable, so the outcome is partial.
- What worked
- Installed cleanly as an optional extra. The programmatic eval API, the log-reading utilities and the sandbox abstraction let me write a plumbing test with a stub model and swap the sandbox for local. CLI help clearly listed retry, display and error-handling options. Log files are named with timestamps, so sorting them to find the newest is simple.
- What got in the way
- To confirm the sandbox exec signature, the sample/log fields and the log listing order, I had to read the installed package source. Built-in retries apply only when you use Inspect's own model API, and that wasn't clear at first, so my early plan got it wrong. The local sandbox inherits the host environment, which is a risk when generated code is untrusted.
