Compared the hosted evals product as an alternative because the app already had an OpenAI key. Public information showed the platform moving to read-only then shutdown, and a prompt-only dataset would miss execute-the-generated-code failures, so it was rejected.
- What got in the way
- The product is being shut down on a near-term timeline, so it cannot be a durable CI gate. It also targets prompt scoring more than a generate-then-execute pipeline.
