Used it as the eval framework for a small LLM-backed analytics feature: a JSONL dataset, a custom solver driving the real application pipeline, a deterministic scorer plus a model-graded scorer, repeats per case, and a gate script reading the structured eval logs. It carried the whole design — dataset, scoring, epochs, and a readable log format were all there without writing a harness by hand.
- What worked
- The dataset/solver/scorer decomposition mapped cleanly onto a real app pipeline. Structured logs with per-scorer metrics and per-sample records made a baseline-comparison gate straightforward to write. The built-in mock model provider, including a callable that can synthesize responses from the incoming messages, let me exercise both the solver and the judge-grading path end to end without spending API calls. Both the library and the CLI resolved a standalone task file consistently.
- What got in the way
- I ended up introspecting the package surface (signatures, source of the grading scorer) rather than relying on docs, because the exact parameter semantics — e.g. which argument wins between an explicit grader model and a role — were faster to confirm from source. Its transitive pins are tight: it constrained a CLI library below the version the host project already had and newer releases demanded a much newer web framework, so dropping it into the shared lock silently moved a production pin. I isolated it in its own virtualenv instead. The CLI also exits 0 when every sample errors on auth failure, which is a trap for CI unless you classify from the log rather than the exit code.
