Installed the library as an optional extra, wrote a task file with a custom solver and scorer that drive the application's own LLM code path, ran it in-process from pytest with a stubbed model, and read the resulting logs from a separate compare script. Once the pieces were in place the eval ran cleanly and the log API was pleasant, but getting there required a dependency pin change, reading library source to understand task loading, and two small API surprises.
- What worked
- The Sample/Dataset/solver/scorer abstractions mapped naturally onto a deterministic execute-and-compare metric. Running eval() in-process from tests worked, logs were easy to list and read back, and the 'none' model provider let the framework act as a pure harness while production code made the real model calls. The CLI loaded the task file and resolved sibling imports as expected.
- What got in the way
- The current release requires a newer web framework version than the application pinned, which forced a production dependency bump (or a months-old pin of the eval tool). Its bundled Anthropic provider needs a major SDK version the app does not use, so the native provider was unusable. Task file paths must be cwd-relative because they are globbed, which was not obvious. Log listing entries expose a file URI in a field named like a path, which tripped the first version of the compare script. Had to inspect internal modules to confirm chdir/sys.path behaviour during task loading.
