Teams using OpenAI’s hosted Evals need to replace their AI quality checks before its scheduled November 30, 2026 shutdown. OpenAI recommends recreating those checks in Promptfoo, a tool for running repeatable tests of AI answers. Preserve historical results, but separately rebuild runnable tests and validate their grading before relying on them.
This is an approaching deadline, not a new announcement: OpenAI gave notice on June 3. Existing Evals became read-only on October 31, so do not plan on creating a fresh hosted baseline now. The retirement covers the Evals dashboard, API and evaluation graders—not ordinary ChatGPT use or every OpenAI model API. Fine-tuning graders follow a separate timeline.
The notice does not specify a shutdown hour or timezone. Finish before November 30 rather than assuming access lasts through your local working day.
Preserve the test definitions, not just old results
Start with the tests that decide whether your application is ready to release. For each one, keep an authorised record of:
Its owner, the behaviour it checks and the threshold for an acceptable answer.
The complete prompt and message roles, input fields, reference answers, target model, API endpoint, settings and connected tools.
Grader instructions and representative historical outputs, including failures and borderline cases.
This inventory puts OpenAI’s task-specific evaluation guidance into a migration checklist. If a supplier manages your AI application, ask which release checks depend on hosted Evals and who will accept their replacements.
Promptfoo supports importing OpenAI dashboard JSONL exports named eval_items_*.jsonl as historical results. It can preserve available outputs and grading information, but the import does not reconstruct the evaluation’s assertions or runnable configuration. An API output-items list response is not the supported dashboard export format.
Keep the definitions separately even if the results import succeeds. Check what your account can actually export; the documentation does not establish a guaranteed export-access window or recovery route after shutdown.
Install Promptfoo and retain any supported export
The current npm installation requirements are Node.js 22.22.0 or newer; Promptfoo recommends Node.js 24 LTS. In a project directory, the documented setup is:
node --version
npm install -g promptfoo
promptfoo --version
promptfoo initThe initializer creates promptfooconfig.yaml. Record the installed version and keep it stable in your team’s dependency management. Arrange model-provider access through your approved secret manager; keep credentials out of the configuration and repository.
If you have a supported dashboard export, the documented import example is below. Substitute your actual exported filename:
promptfoo import eval_items_OutputDataItemStatusParam.ALL.jsonlThat imports reference material. The next step creates the tests you will execute.
Recreate the prompt, model and scoring rules
Use Promptfoo’s configuration guide to map the old suite into four parts:
prompts: the original instructions or prompt-file references, preserving message roles and variable placement.providers: the model or application being tested, with its endpoint, request settings and tool configuration.tests[].vars: each case’s input values and any reference data needed by its checks.tests[].assert: the conditions an answer must meet, including expected values and applicable thresholds.
Keep the existing supported target initially where possible. Changing the model at the same time makes it harder to explain disagreements. Promptfoo’s OpenAI integration supports explicit openai:chat:<model> and openai:responses:<model> routes. Bare identifiers for GPT-5.6 and newer default to Responses, so matching the model name alone does not preserve an application that uses Chat Completions.
Choose assertions that match the requirement: equals for an exact label, is-json for JSON structure, and a reviewed rubric for judgments such as answer completeness. A well-formed JSON response can still contain the wrong answer; a structure check should not replace the old suite’s substantive checks.
Check the judge before trusting a pass
When another AI model grades an answer, select that judge explicitly. Promptfoo’s default judge can depend on available credentials; configure an assertion’s provider or defaultTest.options.provider rather than assuming the top-level target model also fixes the judge. Otherwise, a laptop and a CI runner with different credentials could grade differently.
OpenAI’s migration guide warns that similarity scores may differ between systems and recreated graders need validation. A changed score is therefore not, by itself, evidence that the application became worse.
Have the task owner review known correct, incorrect and borderline answers. Count wrong answers approved and right answers rejected separately, and inspect important categories instead of relying only on an overall pass rate. Reconsider thresholds when the scoring method changes; do not lower them merely to recover the old pass rate.
For a related example of where subject expertise belongs, our JioHotstar AI-search testing explainer examines the distinction between checking an answer and checking its automated reviewer.
Run the rebuilt suite and inspect failures
After configuring the target and checks, use the documented validation and evaluation commands to check the file, create a fresh run and open the local viewer:
promptfoo validate config -c promptfooconfig.yaml
promptfoo eval -c promptfooconfig.yaml --no-cache --no-share -o results.json
promptfoo view--no-cache disables Promptfoo’s local response cache; --no-share prevents creation of a shareable report URL. Neither makes calls to a hosted model offline. Separately, sharing results uploads an evaluation snapshot that can include prompts, inputs and outputs, so check the destination before sharing.
Inspect individual outputs, provider errors and grader disagreements. Compare against retained examples and the intended acceptance criteria—not a new hosted Evals run. If the application uses tools, agents or custom providers, include those paths before declaring the suite migrated.
Budget for both answers and judges
Promptfoo lists Community evaluation features as free, including local execution and model-provider integrations. That does not include your model-provider bill or the work of maintaining the suite. Enterprise collaboration and access-control offerings have custom pricing. The listed 10,000 monthly probes concern red teaming, not an allowance of free model calls for ordinary evaluations.
An illustrative budget makes the distinction clear. Assume 100 cases, two prompt variants, one target-model configuration and one repetition: 200 generated answers. One model-judge call per answer adds another 200 calls.
For this calculation only, suppose both use GPT-6.1 Sol at OpenAI’s Standard short-context rates checked on November 15: US$2 per million input tokens and US$10 per million output tokens. This is a pricing example, not a recommendation to change models.
Generation: 200 calls × 2,000 uncached input tokens plus 500 billed output tokens per call costs $1.80.
Judging: 200 calls × 3,000 uncached input tokens plus 200 billed output tokens per call costs $1.60.
The illustrative token total is $3.40. Output assumptions include billed reasoning tokens. This assumes no cache reads or writes, retries, extra judges, tools, regional uplifts or other fees; it excludes engineering, human review, CI, storage and taxes. More cases, variants, target models or repetitions increase the workload. Free evaluation software does not cap that bill.
Replace the release check only after acceptance
Promptfoo supports CI integration and saved results. Once the rebuilt suite is reviewed, connect it to the development process that approves application changes. Retain results with the configuration version and name the person responsible for investigating failures.
A useful completion test has three parts: the old records you need are preserved; the replacement executes the intended cases; and the product owner accepts the new grading rules and thresholds. Successful installation or historical import alone meets neither of the last two conditions.
Track other dependencies separately. If the application also uses saved prompt objects or Agent Builder, follow our guides to moving OpenAI prompt objects into code and migrating Agent Builder workflows. Rebuilding Evals does not migrate either of those services.
