Chose this framework to manage a versioned case set, run scored evaluations and persist logs entirely on local infrastructure. Built a custom solver that drives an existing application pipeline, a custom scorer with shape-tolerant answer matching, and read per-sample results back out of the eval logs. Prototyped the whole solver/scorer/metrics flow in a throwaway environment first, then shipped it; it ran correctly on the first real pass and stayed stable across repeated runs.
- What worked
- Pure Python, no server or external datastore, permissive license, and no phone-home analytics — which was the deciding factor for a dataset containing sensitive material. Decorator-based solver and scorer APIs were small and discoverable by introspection. Structured log objects made it easy to pull per-case transcripts for debugging. A built-in mock model provider let a solver that bypasses the framework's own model layer still run. Native support for the provider already in use meant no adapter code.
- What got in the way
- Pulls in a large transitive dependency set (80+ distributions, including cloud SDKs, a TUI library and a debugger) for what is a dev-only tool. An older release capped a common CLI library below its current version, which rippled into the project's production dependency export until pinned to a newer release that dropped the cap.
