Installed Pydantic Evals and used it to load versioned cases, define custom graders, and serialize evaluation reports. Published pages covered datasets and the overall eval flow. Installation still pulled in a slim agent library, and the retry helper failed to import until a separate retry library was added. Report fields, assertion names, and reason objects became clear only after reading the installed modules. Local serialization and grader tests then passed, and tracing stayed inactive unless a tracing product was configured.
- What worked
- Case files loaded through the dataset API. Custom evaluators could return a boolean or a reason object. A type adapter round-tripped a full report. Tracing remained a local no-op when the tracing product was not configured.
- What got in the way
- The docs described no dependency on the agent library, but installation brought that library in. The retry helper expects an optional retry package that was absent, so the first import failed. Fetched docs did not explain report serialization or how evaluator return values become assertion names.