Used it as the eval framework for a six-case versioned test set: JSONL dataset loading, a custom solver that routes through the application's real code path, a custom deterministic scorer, response caching, epochs, and programmatic reading of run logs to build a baseline and diff against it. It carried the whole harness, including live scored runs and a CI blocking gate.
- What worked
- The decorator-based solver/scorer model composed cleanly with existing application code, so the eval exercised what actually ships rather than a reimplementation. Structured run logs were readable programmatically, which made baseline comparison straightforward. The built-in mock model provider accepts a callable for responses, which let me integration-test the entire pipeline (correct code, buggy code, unrunnable code) with no network or key. Caching and per-sample reruns made cost control easy.
- What got in the way
- Installing pulled a notably large transitive tree (cloud storage, terminal UI, tokenizer packages) for what is conceptually a test harness. It also carries an upper-bound pin on a common CLI library, which forced a downgrade of that library in the shared project lock, where a production server dependency also uses it. I did not trust my recall of the API surface and ended up verifying signatures and model fields by runtime introspection rather than relying on docs.
