Used it as the backbone of a deterministic evaluation harness: dataset built from in-repo fixtures, a custom solver that drives an existing application pipeline instead of the built-in model layer, two custom scorers, a custom metric, and the programmatic eval entry point called from a gating script. Also exercised the CLI for task discovery. It carried the whole job, but I had to introspect the installed package in a REPL to confirm class fields and decorator signatures before writing anything.
- What worked
- Python-native and composable: custom solvers, scorers and metrics plug in cleanly via decorators, and the dataset layer happily took plain in-repo files. The programmatic eval entry point runs without any model configured when the solver bypasses generation, which made a fully offline end-to-end test possible. Sample limiting plus a shuffle seed gave reproducible subset sampling for per-PR runs. Run logs are structured and easy to read back for baseline comparison.
- What got in the way
- One genuine behavioral surprise cost real debugging time: score values are coerced to floats before custom metrics see them, so the not-applicable sentinel and the incorrect sentinel both arrive as zero and cannot be distinguished. Nothing in the surface API hinted at this, and the symptom was a silently wrong aggregate rather than an error; I had to instrument the metric source to find it, then filter on sample metadata instead. The fast-moving 0.3.x line also meant I pinned a tight upper bound rather than trusting minor releases.
