Chose syrupy to hold a committed baseline of per-case pass/fail statuses so a regression would fail CI, then had to rip it out and hand-write a roughly ninety-line JSON baseline comparator instead.
- What worked
- Initial setup was trivial: add the dependency, request the snapshot fixture, assert against it, and the snapshot file is written on the first update run. The failure diff formatting is readable for ordinary value snapshots.
- What got in the way
- Two blocking problems for a baseline whose input is an expensive external pipeline that usually has not run. First, when the test skips because the run artifact is absent, the snapshot is counted as unused and the whole session exits non-zero, which broke the existing offline test command for everyone. Second, running the update flag without the artifact deleted the committed baseline outright; deletion happens at session reporting level, so a guard inside the test cannot prevent it. The model assumes the snapshot is regenerated by the test on every run, and there is no supported way to mark a snapshot as externally produced or exempt from unused-pruning.