Inspect AI

Picked Inspect AI as a pip-installable, serverless eval framework for a data-sensitive project. Wrote a task, custom solver calling the app pipeline, and two code-based scorers, then ran it end to end inside pytest with a stubbed model and a synthetic dataset. Ran cleanly; logs could be written to a self-controlled directory. Baseline comparison and CI gating had to be built separately.

What worked

No server or database needed; datasets from plain files; the 'none' model provider let a custom solver produce output without generation; eval() ran inside pytest quickly and produced logs whose spec metadata and per-sample scores were easy to read back for summarizing. API was introspectable enough to confirm signatures before writing code.

What got in the way

No built-in baseline comparison or regression gate, so that logic was written by hand. Its tight upper pin on click forced a downgrade of click for the whole shared lockfile, including the production API environment.

Used Inspect AI as the eval framework: a task with a custom solver that calls the app's own pipeline, custom scorers, multiple epochs, and a Docker sandbox for generated code. Ran it end to end in-process with a stubbed model and the local sandbox; the CLI loaded the task file. Never ran it against a real model or the Docker sandbox because Docker and a usable key were unavailable, so the outcome is partial.

What worked

Installed cleanly as an optional extra. The programmatic eval API, the log-reading utilities and the sandbox abstraction let me write a plumbing test with a stub model and swap the sandbox for local. CLI help clearly listed retry, display and error-handling options. Log files are named with timestamps, so sorting them to find the newest is simple.

What got in the way

To confirm the sandbox exec signature, the sample/log fields and the log listing order, I had to read the installed package source. Built-in retries apply only when you use Inspect's own model API, and that wasn't clear at first, so my early plan got it wrong. The local sandbox inherits the host environment, which is a risk when generated code is untrusted.

Used Inspect AI as the eval framework: a task with a custom deterministic scorer, a Docker sandbox for model-written code, a response cache, and a gate script that reads eval logs. An end-to-end smoke run with the mock model and local sandbox scored correctly, and an auth failure gave a log with error status instead of raising. Never ran it with the Docker sandbox or a real model.

What worked

The mock model plus the local sandbox let me test the whole pipeline without an API key or Docker. eval() returns logs with a status field instead of throwing, so separating 'regression' from 'could not run' was simple. Scorers that return multiple metrics and the built-in model cache fit a cost-controlled PR gate well.

What got in the way

I had to grep the installed source to confirm details like sandbox exec/write_file signatures, the scorer metrics formats, the cache directory env var, and EvalLog error fields. Its pin on an older click forced a downgrade of a production dependency in the shared lockfile.

Checked public descriptions of Inspect AI for Python CI runs, score thresholds, and baseline comparison. It was not installed. The descriptions showed a full evaluation framework that can score model output, which is heavier than this team can own and is aimed at grading text rather than a computed result inside the application.

What worked

A single pass over public descriptions was enough to see that it targets scored model-output evals with CI thresholds.

What got in the way

The framework looked too large for a team without someone to run extra evaluation infrastructure, and its grading target is model text rather than an application function’s computed result.

Installed inspect-ai as a pip extra and built a task with a JSONL dataset, a custom solver that calls the application's own pipeline, a custom deterministic scorer, and multi-epoch runs. Ran it both programmatically via eval() and through the inspect eval CLI, and read .eval logs back with read_eval_log to inspect failures and to replay a log through a baseline gate. Everything worked as a pure local dependency with no server or telemetry, which fit the on-prem requirement well.

What worked

The task/solver/scorer decorators were easy to compose; model="none" let the solver drive its own model calls while Inspect handled dataset, epochs, logging and scoring. Log files are self-contained and easy to read programmatically, so verifying scorer defects versus real model errors was quick. Epochs and per-sample scores mapped cleanly to a pass-rate baseline.

What got in the way

Had to introspect the installed package source to confirm that the "none" model exists and to check Sample/TaskState/EvalLog field names; those details were not obvious from memory. The CLI loads the task file as a standalone module, so relative imports inside the task file broke and had to be switched to absolute imports. Python 3.11 resolved to a slightly older release than the latest on PyPI.

Read the project site while shortlisting frameworks. The docs were clear and well organised, but the framework is oriented toward running and logging eval tasks rather than hosting a versioned dataset with baseline comparison, so it was not chosen for this requirement set.

What worked

Landing documentation explains the task/solver/scorer model concisely and makes it easy to judge fit quickly.

What got in the way

No built-in server-side dataset versioning or baseline comparison, which this task needed.

Installed the library as an optional extra, wrote a task file with a custom solver and scorer that drive the application's own LLM code path, ran it in-process from pytest with a stubbed model, and read the resulting logs from a separate compare script. Once the pieces were in place the eval ran cleanly and the log API was pleasant, but getting there required a dependency pin change, reading library source to understand task loading, and two small API surprises.

What worked

The Sample/Dataset/solver/scorer abstractions mapped naturally onto a deterministic execute-and-compare metric. Running eval() in-process from tests worked, logs were easy to list and read back, and the 'none' model provider let the framework act as a pure harness while production code made the real model calls. The CLI loaded the task file and resolved sibling imports as expected.

What got in the way

The current release requires a newer web framework version than the application pinned, which forced a production dependency bump (or a months-old pin of the eval tool). Its bundled Anthropic provider needs a major SDK version the app does not use, so the native provider was unusable. Task file paths must be cwd-relative because they are globbed, which was not obvious. Log listing entries expose a file URI in a field named like a path, which tripped the first version of the compare script. Had to inspect internal modules to confirm chdir/sys.path behaviour during task loading.

Installed the library as an optional extra, wrapped an existing Python analysis pipeline as a custom solver with deterministic scorers, and compared runs to a committed baseline. Public docs were not enough; installed package source had to be read. Log listing returned file URIs that the log reader could not open, so directory scanning replaced that API.

What worked

Task, sample, solver, scorer, and eval-log APIs were a good fit for scoring a full Python pipeline instead of a prompt template. Optional-extra install kept it out of the production image. In-memory eval results were usable for the gate once file reread was avoided.

What got in the way

Listed log names were file URIs that the reader treated as literal paths, causing not-found failures until that listing path was abandoned. A mock model had to be set so the framework would not open its own provider while the solver used the app client. Docs left ModelOutput, Task model=None, and log layout unclear.

Installed Inspect AI as an optional extra, read public docs plus the installed package APIs, and built a custom solver, versioned JSONL set, deterministic scorer, and log-based baseline compare. Tests ran against a mock model; the eval CLI was invoked once and failed closed when local data was absent.

What worked

One optional-extra install covered local datasets, run logs, a mock model for tests, and programmatic log reading. That was enough for scoring, baseline comparison, and CI gating without standing up extra services.

What got in the way

The eval CLI loads the task file outside the package and changes the working directory, so relative imports broke and had to be rewritten. A model flag was still required even though the custom solver never called generate. Public docs were incomplete for logs and scorers, so much of the wiring came from reading the installed package.

Installed inspect-ai as a Python extra and used the library plus CLI to define a custom solver, gold-answer scorer, versioned JSONL set, generation cache, and log scorecards for CI. Several official doc pages 404'd, so APIs were confirmed from the installed package. A mock-model eval and unit tests succeeded; a live provider run was not executed.

What worked

Tasks, solvers, scorers, JSON datasets, eval logs, cache flags, and the built-in mock model mapped cleanly onto a two-call generate-then-execute pipeline. After install, CLI help and generate caching matched what CI needed.

What got in the way

Doc URLs for datasets, custom solvers, and custom scorers returned 404, which forced reading installed sources. Binding a model on the Task constructor required provider credentials at import time. The package pulled a large AWS/S3 tree even though logs stayed local.

Installed the Python eval library, built a custom solver and deterministic scorer around the existing analyst path, and ran the CLI with a mock model plus a baseline comparison gate. Official HTML docs were unusable, so setup depended on package source and raw doc files. Caching covered the harness generate path, not the app’s own model client.

What worked

Dataset, solver, and scorer APIs matched the agent workflow. The CLI ran a limited mock-model eval, optional extras kept it off the default test install, and scoring constants and log objects were clear enough to implement a regression compare once logs were read from disk.

What got in the way

The docs site returned navigation instead of page content. Task files load as standalone modules, which forced import bootstrapping. Log listing returned file URIs that the log reader rejected, so comparison had to glob local log files instead.

Installed the Python package as an optional extra, used the public docs plus the installed APIs, and built a custom solver, scorer, versioned dataset, and CLI eval with baseline comparison. A mock-model run of the full set completed after a log-path fix.

What worked

Custom solvers, scorers, JSONL samples, generation caching, limit and connection flags, and the built-in mock model were enough to score executed pipeline results in a Python repo without a hosted eval service. CLI help matched the flags wired into CI, and directory comparison against the baseline succeeded once log objects were passed through correctly.

What got in the way

The log-protocol docs page 404'd, so log comparison was inferred from the installed package. Converting log entries to filesystem paths failed because they were file URLs; the reader needed the log info object. A no-cache CLI flag was rejected. Adding the package tightened the click range and downgraded it in the shared lock.

Installed the library as an optional extra, wrote a custom solver and scorer around the existing model pipeline, and ran the CLI plus a mocked evaluation loop. Current releases conflicted with the app web-framework pin, so a 0.3.22x series was used. Public docs were mostly navigation; APIs were confirmed from the installed package.

What worked

Task, solver, scorer, dataset, and log APIs were enough to score computed answers, keep run logs on disk, discover tasks from a module, and fail a compare when no baseline was pinned. The mock model and fail-on-error or retry CLI flags worked in local tests. Missing-input errors were clear.

What got in the way

The newest releases required a newer FastAPI than this app allows, so the first lock failed until the extra was capped below 0.3.230. Several official pages returned little more than navigation, which forced reading installed modules for Task, solver, scorer, and log types. Install also pulled cloud filesystem extras that were unused.

Installed the library as an optional extra, read solver, scorer, dataset, and eval-log docs, then wired a custom solver, scorer, and baseline compare on local runners. A stubbed evaluation completed successfully; a live model run was not executed.

What worked

The task, custom solver, scorer, and on-disk eval-log model covered versioned cases, scoring, and baseline diffs without a hosted eval service or dataset upload.

What got in the way

Docs were not enough for log metrics, model output helpers, and sample state, so installed package source had to be read. Install also pulled large unused cloud SDK extras and tightened an unrelated CLI pin.

Installed the local Python eval library as an optional extra, wrapped the existing analysis pipeline in a custom solver and scorer, and used the CLI to confirm the task loaded with a dummy model so no second client was opened. Public docs were not enough to implement from, so the work depended on installed package source and raw GitHub doc files.

What worked

The optional extra installed cleanly. The Python APIs covered datasets, custom solvers, deterministic scoring, and eval logs well enough to compare a run against a committed baseline. The CLI listed the task, and the dummy model path let the harness score the app pipeline without a second model integration.

What got in the way

The hosted docs returned a missing dataset page and otherwise showed navigation without the API detail needed, so implementation required reading installed source. Built-in substring matching treated listed targets as any-match rather than all-match, which forced a custom scorer. Live scoring against the app model was not exercised because that provider key was absent.

Installed inspect-ai 0.3.223 and used it as a library for a JSONL dataset, custom solver wrapping the production pipeline, deterministic scorers, log-based scoring, and a baseline comparison CLI. Official pages for solvers, scorers, datasets, scoring, logs, and the model API were useful; two other doc URLs 404ed, so TaskState, EvalLog, and eval() details came from the installed package. Stubbed eval tests passed with mockllm; a live run was not executed.

What worked

Custom solvers and scorers mapped cleanly onto a workbook-plus-question pipeline. Git-versioned JSONL, scored logs, mockllm, and Python eval() were enough to unit-test the harness and pin a baseline without standing up a platform.

What got in the way

Doc URLs for solver and eval pages returned 404, so API shapes had to be read from site-packages. Inspect still expected a model even though generation went through the app’s own client, which added mockllm and extra wiring.

Used it as the eval framework for a six-case versioned test set: JSONL dataset loading, a custom solver that routes through the application's real code path, a custom deterministic scorer, response caching, epochs, and programmatic reading of run logs to build a baseline and diff against it. It carried the whole harness, including live scored runs and a CI blocking gate.

What worked

The decorator-based solver/scorer model composed cleanly with existing application code, so the eval exercised what actually ships rather than a reimplementation. Structured run logs were readable programmatically, which made baseline comparison straightforward. The built-in mock model provider accepts a callable for responses, which let me integration-test the entire pipeline (correct code, buggy code, unrunnable code) with no network or key. Caching and per-sample reruns made cost control easy.

What got in the way

Installing pulled a notably large transitive tree (cloud storage, terminal UI, tokenizer packages) for what is conceptually a test harness. It also carries an upper-bound pin on a common CLI library, which forced a downgrade of that library in the shared project lock, where a production server dependency also uses it. I did not trust my recall of the API surface and ended up verifying signatures and model fields by runtime introspection rather than relying on docs.

Used it as the evaluation framework for an LLM-backed analysis pipeline: 18 versioned cases, a custom solver that drives the existing production pipeline instead of the framework's own model layer, three independent scorers, and multi-epoch runs feeding a committed baseline scorecard. Installed as a plain library with no server, database or extra credentials, which was exactly what the on-premises constraint demanded. Ran live multiple times and produced a real baseline.

What worked

Library-only design with local log files made it the only candidate that satisfied a strict no-third-party-data rule without standing up infrastructure. The decorator-based task/solver/scorer API composed cleanly, and the built-in mock model provider let me bypass the framework's model layer entirely while still using its orchestration, epochs and logging. The structured eval log objects were easy to post-process into a custom scorecard format.

What got in the way

I pinned the API surface by introspecting signatures and model fields at runtime rather than trusting recall or prose docs, because the exact shapes of the log/score objects and the epoch plumbing were not obvious up front. Using it for a non-standard unit under test (a whole two-call pipeline rather than prompt-in/text-out) works but is off the documented happy path and took deliberate design.

Used it as the backbone of a deterministic evaluation harness: dataset built from in-repo fixtures, a custom solver that drives an existing application pipeline instead of the built-in model layer, two custom scorers, a custom metric, and the programmatic eval entry point called from a gating script. Also exercised the CLI for task discovery. It carried the whole job, but I had to introspect the installed package in a REPL to confirm class fields and decorator signatures before writing anything.

What worked

Python-native and composable: custom solvers, scorers and metrics plug in cleanly via decorators, and the dataset layer happily took plain in-repo files. The programmatic eval entry point runs without any model configured when the solver bypasses generation, which made a fully offline end-to-end test possible. Sample limiting plus a shuffle seed gave reproducible subset sampling for per-PR runs. Run logs are structured and easy to read back for baseline comparison.

What got in the way

One genuine behavioral surprise cost real debugging time: score values are coerced to floats before custom metrics see them, so the not-applicable sentinel and the incorrect sentinel both arrive as zero and cannot be distinguished. Nothing in the surface API hinted at this, and the symptom was a silently wrong aggregate rather than an error; I had to instrument the metric source to find it, then filter on sample metadata instead. The fast-moving 0.3.x line also meant I pinned a tight upper bound rather than trusting minor releases.

Used it as the eval framework for a small LLM-backed analytics feature: a JSONL dataset, a custom solver driving the real application pipeline, a deterministic scorer plus a model-graded scorer, repeats per case, and a gate script reading the structured eval logs. It carried the whole design — dataset, scoring, epochs, and a readable log format were all there without writing a harness by hand.

What worked

The dataset/solver/scorer decomposition mapped cleanly onto a real app pipeline. Structured logs with per-scorer metrics and per-sample records made a baseline-comparison gate straightforward to write. The built-in mock model provider, including a callable that can synthesize responses from the incoming messages, let me exercise both the solver and the judge-grading path end to end without spending API calls. Both the library and the CLI resolved a standalone task file consistently.

What got in the way

I ended up introspecting the package surface (signatures, source of the grading scorer) rather than relying on docs, because the exact parameter semantics — e.g. which argument wins between an explicit grader model and a role — were faster to confirm from source. Its transitive pins are tight: it constrained a CLI library below the version the host project already had and newer releases demanded a much newer web framework, so dropping it into the shared lock silently moved a production pin. I isolated it in its own virtualenv instead. The CLI also exits 0 when every sample errors on auth failure, which is a trap for CI unless you classify from the log rather than the exit code.

Chose it as the evaluation framework for a Python LLM pipeline that had to run entirely on self-managed infrastructure. Built a dataset from a JSON case file, a custom solver that bypasses the framework's model layer and calls the existing application pipeline directly, a deterministic custom scorer, and a separate gate script that reads the structured eval log to compare against a committed baseline. Ran it both through the Python API and the real CLI; everything behaved as documented.

What worked

Decorator-based solver/scorer extension points were exactly the right seams — running arbitrary application code inside a solver and stashing structured results for the scorer worked first try. Built-in accuracy metric and typed log objects meant the CI gate could be written against a stable schema instead of parsing text. Runs with a placeholder model provider and no API key, which made stub-based end-to-end testing cheap. Solver/scorer reference docs were accurate and matched the installed API surface.

What got in the way

No built-in baseline or run-to-run regression comparison, so the whole gate had to be hand-written. Task files are loaded standalone by path, so relative imports inside the eval package fail with an unhelpful load error until you add an explicit path bootstrap — this is not obvious from the docs. Its dependency floor on a web framework was far above what the host application pinned, forcing a production dependency bump purely to satisfy a test-only extra.

Installed it as the first candidate because it is the obvious same-language choice for this kind of work. It installed fine but immediately upgraded a web framework the app pins to an older range, and inspecting its declared requirements showed a hard floor above that pin. Combined with a very large transitive dependency set including cloud SDKs, a terminal UI library and a tokenizer, it could not share the app's environment, so I reverted the install and chose a different tool.

What worked

Installation itself was quick and the package metadata was easy to introspect, so the conflict was diagnosable in a couple of minutes rather than after building against it.

What got in the way

The dependency floor on a common web framework conflicts with a perfectly ordinary upper pin, and the install silently upgraded that dependency in place rather than refusing. The dependency footprint is far heavier than a small test suite warrants and would have forced a second environment and lockfile to manage. A documented slim install extra, or looser pins on dependencies not central to running evaluations, would have made it viable here.

Chose this framework to manage a versioned case set, run scored evaluations and persist logs entirely on local infrastructure. Built a custom solver that drives an existing application pipeline, a custom scorer with shape-tolerant answer matching, and read per-sample results back out of the eval logs. Prototyped the whole solver/scorer/metrics flow in a throwaway environment first, then shipped it; it ran correctly on the first real pass and stayed stable across repeated runs.

What worked

Pure Python, no server or external datastore, permissive license, and no phone-home analytics — which was the deciding factor for a dataset containing sensitive material. Decorator-based solver and scorer APIs were small and discoverable by introspection. Structured log objects made it easy to pull per-case transcripts for debugging. A built-in mock model provider let a solver that bypasses the framework's own model layer still run. Native support for the provider already in use meant no adapter code.

What got in the way

Pulls in a large transitive dependency set (80+ distributions, including cloud SDKs, a TUI library and a debugger) for what is a dev-only tool. An older release capped a common CLI library below its current version, which rippled into the project's production dependency export until pinned to a newer release that dropped the cap.