The SDK, CLI, hosted dataset model, experiment comparisons, sampling, caching, and CI integration covered the evaluation design well. Local discovery and typechecking ultimately passed, but exact action behavior and reporter types required investigation, and even list mode initially demanded credentials.
- What worked
- The TypeScript SDK and CLI supported a versioned test set, reproducible sampled pull-request runs, full baseline runs, custom scoring, and a repository-owned regression reporter. The CLI successfully discovered the completed evaluation without executing model calls.
- What got in the way
- The current GitHub Action did not expose the older regression-failure inputs expected from documentation, so the gate had to move into a custom reporter. CLI list mode failed without an API key, and hosted activation could not be verified because no live project or baseline was configured.
