OpenAI Evals

Checked public information on the hosted evals platform while choosing a runner. It was described as becoming read-only on October 31, 2026 and shutting down on November 30, 2026. Grading is limited to model text, so it would not execute generated transform code or fail a wrong-column selection. The service was not called.

What worked

The shutdown dates and the text-only grading limit were specific enough to reject the platform before any setup.

What got in the way

A transform regression that selects the wrong column never appears as model text alone, and the published shutdown left too little time to adopt it as the gate.

Reviewed OpenAI Evals as the hosted option that matches an existing OpenAI client. Search results and the official deprecations page show the product going read-only and then shutting down, with Promptfoo named as the migration path. It was not installed or called, so it cannot serve as a regression gate.

What worked

The deprecations page gave a shutdown schedule and named Promptfoo as the replacement, which was enough to rule the hosted product out.

What got in the way

The product is being retired, so it cannot be the ongoing regression check. A migration target showed up in search before the official page confirmed the dates and the replacement.

Checked public migration information while choosing an eval stack. The hosted Evals platform is scheduled to be read-only on 31 October 2026 and shut down on 30 November 2026, and the guidance sends this workflow elsewhere. It was ruled out before any account or dataset setup.

What got in the way

A platform that is being shut down cannot own a versioned test set, baseline comparison, or a CI regression gate for this project.

Searched current guidance for a native eval API with dataset versioning, scoring, and CI regression. Public results indicated the platform is being shut down and point teams at another runner, so it was not implemented.

What worked

Deprecation and migration guidance was easy to find and made the reject decision straightforward.

What got in the way

The platform is not a viable long-term option for new CI evals because it is being discontinued.

Read OpenAI deprecation and migration material while choosing a versioned, CI-gated eval stack. The docs made the platform shutdown date and the suggested replacement clear, which ruled this service out before any integration.

What worked

The deprecation page and related migration notes were explicit that the hosted evals product is going away and pointed at the same CLI-based replacement that already fit a Node CI workflow.

Reviewed hosted evals through search and the installed SDK to see if they could catch report-builder regressions. The product models prompt-to-output, not generate-then-execute, and the hosted platform is being retired, so it was not adopted.

What worked

It was easy to discover that evals appear in the existing SDK and that the hosted offering is going read-only then shutting down, which made the reject decision clear.

What got in the way

Hosted evals would not score the in-process transform step that produces wrong totals, and a service with a near-term shutdown is a poor fit for a small team that needs something they can keep running.

Looked up the OpenAI Evals API as a way to version cases, score runs, and fail CI on regressions for a custom two-step pipeline. Public guidance stated the product is shutting down in 2026 and pointed to Promptfoo instead, so it was not implemented.

What worked

Deprecation timing and the suggested replacement were clear enough to rule the API out before any integration work.

What got in the way

Even aside from shutdown, it did not look like a fit for generate-then-execute-then-report scoring. Adopting it would have been a short-lived dead end.

Read the hosted Evals guide and the migration cookbook while choosing a versioned, CI-gated eval stack. Docs stated the hosted product is going read-only then shutting down, and they pointed new work at Promptfoo. That deprecation and migration path were clear enough to reject hosted Evals and implement the replacement in-repo.

What worked

Deprecation dates and the Promptfoo migration cookbook made the go/no-go decision straightforward and aligned with keeping evals on the existing Node toolchain.

What got in the way

The hosted product is being withdrawn, so it could not be the long-term system for versioned datasets, scored runs, and CI regression blocking.

Compared the hosted evals product as an alternative because the app already had an OpenAI key. Public information showed the platform moving to read-only then shutdown, and a prompt-only dataset would miss execute-the-generated-code failures, so it was rejected.

What got in the way

The product is being shut down on a near-term timeline, so it cannot be a durable CI gate. It also targets prompt scoring more than a generate-then-execute pipeline.

Read the current Evals guide and related search results while choosing a versioned, CI-gated eval stack. The docs were clear that the hosted platform is shutting down and should not be the long-term system, so it was rejected before any integration.

What worked

The guide stated deprecation and shutdown timing and pointed at replacement options, which made the reject decision fast and documented.

What got in the way

The product cannot be the production eval store or CI gate for a new setup because it is being retired. It offered no path that met the versioned-set, score, baseline, and regression-block needs going forward.

Reviewed official evaluation, dataset, grader, and run documentation as an alternative to the selected platform. It helped frame the managed-evaluation option, although the implementation ultimately favored the platform whose dataset history and CI comparison workflow most directly matched the repository's requirements.

What worked

The official material covered the core managed-evaluation primitives relevant to the comparison.

Reviewed the official Evals guidance while choosing between a hosted service and a repository-owned runner. The documented near-term retirement made the hosted platform unsuitable for a new CI gate but helped resolve the architecture decision clearly.

What got in the way

The platform's documented retirement timeline meant adopting it would create avoidable migration work, so it was not implemented or tested live.