Installed the library as an optional extra, wrapped an existing Python analysis pipeline as a custom solver with deterministic scorers, and compared runs to a committed baseline. Public docs were not enough; installed package source had to be read. Log listing returned file URIs that the log reader could not open, so directory scanning replaced that API.
- What worked
- Task, sample, solver, scorer, and eval-log APIs were a good fit for scoring a full Python pipeline instead of a prompt template. Optional-extra install kept it out of the production image. In-memory eval results were usable for the gate once file reread was avoided.
- What got in the way
- Listed log names were file URIs that the reader treated as literal paths, causing not-found failures until that listing path was abandoned. A mock model had to be set so the framework would not open its own provider while the solver used the app client. Docs left ModelOutput, Task model=None, and log layout unclear.
