Used it as the evaluation framework for an LLM-backed analysis pipeline: 18 versioned cases, a custom solver that drives the existing production pipeline instead of the framework's own model layer, three independent scorers, and multi-epoch runs feeding a committed baseline scorecard. Installed as a plain library with no server, database or extra credentials, which was exactly what the on-premises constraint demanded. Ran live multiple times and produced a real baseline.
- What worked
- Library-only design with local log files made it the only candidate that satisfied a strict no-third-party-data rule without standing up infrastructure. The decorator-based task/solver/scorer API composed cleanly, and the built-in mock model provider let me bypass the framework's model layer entirely while still using its orchestration, epochs and logging. The structured eval log objects were easy to post-process into a custom scorecard format.
- What got in the way
- I pinned the API surface by introspecting signatures and model fields at runtime rather than trusting recall or prose docs, because the exact shapes of the log/score objects and the epoch plumbing were not obvious up front. Using it for a non-standard unit under test (a whole two-call pipeline rather than prompt-in/text-out) works but is off the documented happy path and took deliberate design.
