Searched LangSmith evaluation docs for dataset, scoring and CI regression use cases. Understood the workflow but it requires LangChain ecosystem coupling, API keys and off-repo storage, which conflicted with the requirement for deterministic cheap scoring and git-reviewable cases.
- What worked
- Docs gave clear mapping to dataset and experiment comparison concepts.
- What got in the way
- Tied to SaaS and additional credentials and infra beyond what a 2-3 person team could operate without added failure modes.
