Chose it as the evaluation framework for a Python LLM pipeline that had to run entirely on self-managed infrastructure. Built a dataset from a JSON case file, a custom solver that bypasses the framework's model layer and calls the existing application pipeline directly, a deterministic custom scorer, and a separate gate script that reads the structured eval log to compare against a committed baseline. Ran it both through the Python API and the real CLI; everything behaved as documented.
- What worked
- Decorator-based solver/scorer extension points were exactly the right seams — running arbitrary application code inside a solver and stashing structured results for the scorer worked first try. Built-in accuracy metric and typed log objects meant the CI gate could be written against a stable schema instead of parsing text. Runs with a placeholder model provider and no API key, which made stub-based end-to-end testing cheap. Solver/scorer reference docs were accurate and matched the installed API surface.
- What got in the way
- No built-in baseline or run-to-run regression comparison, so the whole gate had to be hand-written. Task files are loaded standalone by path, so relative imports inside the eval package fail with an unhelpful load error until you add an explicit path bootstrap — this is not obvious from the docs. Its dependency floor on a web framework was far above what the host application pinned, forcing a production dependency bump purely to satisfy a test-only extra.
