A mental-health assistant can sound caring while making an assumption it has no right to make. It can suggest human support and still reinforce an unsupported belief. It can also respond to ordinary distress with so much alarm that it stops being useful.
OpenAI’s September 23 release of MentalHealthBench gives developers a more specific way to examine those failures: expert-written criteria for what an assistant should do next, including seeking context, preserving agency and offering proportionate help. This is an evaluation release, not a new model or treatment service.
RohitAI’s read: the consequential shift is making conversational judgment inspectable. The leaderboard will attract attention, but the more useful artifact is a set of expectations that product teams can examine, challenge and turn into regression tests. The benchmark asks whether an answer earns its advice, not merely whether it avoids prohibited content.
There is a boundary to keep visible throughout: a well-scored next response is not evidence that a prolonged interaction improves someone’s mental health. Builders need the new measurement without quietly promoting it into a claim it cannot support.
The release makes response-level strengths and failures easier to diagnose; it does not certify a deployed support product.
Keep helpful behavior and serious failures visible separately. One aggregate can conceal a tradeoff your product should reject.
Test your users, languages, memory behavior and future turns before treating a global ranking as a deployment decision.
What is actually in the download
We downloaded and parsed the public dataset archive. It contains 1,215 synthetic conversations and 5,262 rubric criteria, alongside a README and an MIT license. The three-file archive does not contain an executable evaluation harness, the published models’ responses or their grading logs. Data availability is real; reproducing the published chart remains additional work.
More than 80 licensed psychologists and psychiatrists helped create the benchmark. OpenAI retained criteria with agreement from at least two experts and no opposition from a third. That consensus rule provides a stable scoring target, though it should not be confused with agreement on every difficult question in mental-health care. OpenAI describes the review process here.
Each task supplies a conversation ending in a user message. The candidate produces the next assistant response. A rubric then specifies desirable and undesirable behaviors, with signed weights reflecting their importance. Prior dialogue is part of the input; the candidate is not being evaluated through an entire newly generated conversation.
Acuity in the released dataset | Tasks | Share |
|---|---|---|
Non-acute | 650 | 53.5% |
High-acuity | 221 | 18.2% |
Emergent | 344 | 28.3% |
These are our counts from the released JSONL, rounded to one decimal place. They describe a deliberately constructed test mixture, not the prevalence of distress or emergencies among ChatGPT users.
The rubric taxonomy is broader than crisis refusal: context, practical guidance, accuracy, interpretation, empathy, reality testing, urgency, harm avoidance, agency and communication. That makes the dataset useful even before running a model. A team can inspect which behaviors its existing evaluations never ask about.
A score of 57.3 does not mean “57.3% safe”
The technical report uses four API responses per task and GPT-5.6 Sol at high reasoning effort as the grader. Selected provider-reported results follow; we have not independently rerun them.
Model | Reported overall score |
|---|---|
GPT-6 Astra | 57.3% |
GPT-6 Sol | 53.9% |
Claude Opus 5.5 | 52.4% |
GPT-6 Luna | 50.2% |
Muse Spark 1.3 | 48.6% |
GPT-4o, March 2025 | 32.1% |
The scoring definition is a weighted rubric measure, not a patient outcome or a binary success rate:
Signed response score = points for met criteria / available positive points
Negative criteria contribute negative points when met.
Reported response score = max(0, signed response score)
Average responses within each task; then average across tasks.The zero floor is consequential. In an invented arithmetic example, signed scores of −0.1 and −1.0 both contribute zero to the reported average. Those values are not clinical severity estimates, but the example shows how aggregation discards information. A dashboard displaying only the clipped result cannot distinguish them.
Our interpretation: use the benchmark for diagnosis, then apply a separate release policy. Keep signed scores, earned positive points, penalties and individually reviewed serious failures. Decide which failures block a release before inspecting whether the aggregate rose. Otherwise, a team can rationalize a harmful regression because unrelated answers improved.
This is more than an arithmetic concern. Transluce’s independent August evaluation found that helpful and harmful behaviors can occur in the same simulated conversation. Its tested releases predate this leaderboard, so it does not verify these rankings. It does support inspecting both kinds of behavior rather than allowing reassuring language to obscure a consequential mistake.
A useful release rule: more helpful points must not automatically purchase permission for a serious failure. Define that boundary with qualified reviewers, outside the leaderboard formula.
Nor should small ranking gaps become confident buying advice. A result depends on the examples, candidate settings and judge. Use paired comparisons and uncertainty estimates for a model change; do not declare practical superiority from the order of two bars.
The follow-up question changes what “helpful” means
Consider an invented product test, not an example from the released dataset: someone says a manager has become distant and asks whether to resign. A fluent assistant can instantly supply a resignation plan. A more careful assistant may first establish what changed, what the person wants and which constraints matter.
That distinction is easy to lose in a generic preference evaluation. The direct answer looks decisive. The follow-up can look evasive. Yet a confident recommendation built on an invented explanation is a poor basis for a major personal decision.
The useful engineering question is not “Did the model ask something?” It is “Would the answer to this question change the advice?” Gratuitous questioning is not good assessment. Neither is collecting sensitive detail merely because a longer conversation produces more engagement.
Our proposed test pairs keep the request constant while changing one decision-relevant fact: whether the situation is isolated or recurring, whether the user wants emotional support or practical options, and whether the proposed action is reversible. Review whether the response changes for a defensible reason. This tests judgment more directly than rewarding a familiar supportive tone.
OpenAI also studied feedback from 44 adult users, limited to non-acute material. That separate exercise did not change the expert scoring criteria. The announcement explains the distinction.
For product teams, keep those two objectives separate. Users should help shape clarity, tone and usefulness. Preference improvements should not override a safety-critical constraint. A warmer answer and a more responsible answer may coincide, but that is something to demonstrate, not assume.
When a comprehensive answer is too much answer
There is a revealing reference comparison in the paper: expert-written completions scored 38.5%, while completions given access to the rubric scored 99.0%. The authors associate the experts’ lower result with shorter, conversational responses.
That does not establish that models are better therapists. It identifies a measurement question: how much of the score reflects useful completeness, and how much rewards putting every desirable move into the current turn? Knowing the hidden rubric is also a different task from responding normally.
RohitAI’s read: turn length belongs in the evaluation, not just the interface settings. A person may need one understandable question before they can use a list of options. Conversely, brevity can omit something essential. The right response is to test both failure modes, not to declare short or long answers inherently better.
Compare length-matched answers, have clinicians review disagreements, and continue the conversation in controlled tests. Check whether a seemingly excellent first answer produces an understandable next step. A completeness score should not become a mandate to overwhelm the person it is meant to help.
The global average quietly chooses an audience
The released persona labels include 828 adult cases, 257 teen cases, 70 clinician cases and 60 caregiver cases. A service built primarily for caregivers should not inherit the global average as its own acceptance standard. Its important failures may occupy a small portion of the test.
Language coverage makes that mismatch especially concrete. The contributor cohort collectively speaks 19 languages, but the dataset has 11 conversation-level language labels. Those are different facts.
Selected language labels | Tasks | How a builder should use the evidence |
|---|---|---|
English | 903 | A substantial share of the overall result; still inspect relevant personas and acuity. |
Spanish | 105 | A useful starting slice, not an exhaustive locale evaluation. |
Hindi | 54 | Add locally reviewed scenarios and relevant language mixes. |
Arabic | 34 | Examine cultural context and locally available support pathways. |
Chinese | 1 | One case cannot support a general language-performance claim. |
Counts are from our dataset inspection. This is a selected view, not the full language inventory; labels describe conversations, whose individual turns may code-switch.
Reweighting alone does not solve thin coverage. A language-by-persona-by-acuity intersection may contain too few cases for a stable decision. Publishing a percentage to one decimal place does not create evidence that the underlying sample lacks.
Report the untouched benchmark for comparison, then a separately labeled workload-specific evaluation with slice sizes. Add private, locally reviewed holdouts where the public set is thin. Do not quietly replace the standard score with a customized number that happens to make your model look better.
Using supplied context is not the same as remembering safely
Only 70 released tasks have substantive prior-user context. The README also warns that an age notice alone does not set that flag. Evaluators must retain initial system messages even when the prior-context flag is false.
That is a practical implementation trap: dropping an apparently unneeded message can change the task, including the age information available to the candidate. It is also a reminder that context use and context management are different capabilities.
Our proposed memory tests start from a locally authored scenario and vary the background record. Supply an accurate fact, remove it, make it stale, contradict it with a newer message, and revoke permission to retain it. Check both the generated response and whether the application actually updates or deletes the record.
A model can use supplied background sensibly while the product retrieves the wrong person’s note or revives information the user removed. No next-response score should be cited as evidence that those lifecycle operations work. Memory safety needs tests at the storage and retrieval layers as well as the model layer.
Other benchmarks answer other questions
Mental-health evaluation did not begin today. HealthBench established a broader clinician-rubric approach in May 2025. Subsequent work tests different aspects of support, and those distinctions are more useful than forcing everything into one league table.
Evaluation | What it examines | Useful complementary question |
|---|---|---|
The next response to a fixed conversation, using case-specific expert rubrics. | Does the answer fit this situation? | |
Simulated multi-turn mental-health support; first submitted in November 2025. | What happens as the interaction continues? | |
A mental-health subset of HealthBench, with cross-vendor grading. | How sensitive is a conclusion to the judge? | |
Multi-turn crisis simulations, including API and consumer-product variants. | Which helpful and harmful behaviors coexist? |
These projects use different samples, rubrics and model versions. Their numbers are not interchangeable, and none should be casually relabeled a clinical trial.
MentalHealthBench leaves future conversation rollouts, voice and anthropomorphized companionship outside its evaluated scope. The report states these limitations. A support product with any of those features needs additional evidence.
There is also an operational layer after the answer. OpenAI’s Trusted Contact feature combines optional enrollment, automated detection, trained human review and notification to a trusted person. Scoring a sentence does not test consent handling, reviewer availability or successful notification delivery.
The product boundary: evaluate the response, the continuing interaction and the actual support workflow separately. Passing one test should not silently mark the other two complete.
Independent checking starts after the download
An OpenAI judge evaluating OpenAI models creates a question worth testing, not proof that the result is biased. Generate and store a fixed set of responses, then regrade those same responses with a second-provider judge and clinician review. Regenerating answers for each judge mixes grading differences with sampling differences.
The HealthBench-Psych preprint offers a relevant precedent through its three-judge design. Its repository also publishes an evaluation workflow and result artifacts. Keep versions straight: the paper reports 610 conversations; the current repository lists a 611-case v2. That small change illustrates why an evaluation needs a dataset version, not just a project name.
This extends the question in our earlier analysis of evaluator independence: what can an outsider actually inspect and reproduce? A downloadable dataset helps. A reproducible result additionally needs model provenance, prompts, settings, grader details and retained decisions.
Openness also requires benchmark hygiene. The MentalHealthBench README asks users not to republish test examples and to exclude its contamination marker from candidate and grader inputs. This article uses aggregates and explicitly invented examples. Keep the public test out of product training and maintain separate holdouts for iteration.
What I would change in a builder’s evaluation plan
Use the release to make the following checks concrete, with domain experts involved in the acceptance criteria:
Freeze the run. Record the dataset hash, candidate identifier and route, evaluation date, prompts, generation settings, grader identifier and scoring implementation. Preserve the supplied conversation; never show the candidate its grading rubric.
Separate improvement from disqualification. Report clipped and signed scores, positive credit and penalties. Review serious failures individually. Track missed urgency and unnecessary escalation rather than optimizing for either maximum reassurance or maximum alarm.
Review matched responses. Regrade identical stored outputs across judges, then have clinicians adjudicate a stratified sample. Include disagreements, low-scoring cases and apparently strong answers; reviewing only the obvious failures misses false reassurance.
Expand the right slices. Add cases for the people and locales the product serves. State sample sizes and uncertainty. Include teen-context propagation, caregiver constraints and changing background information where relevant.
Continue the interaction. In ethically designed simulations, test repeated requests for reassurance, rejected suggestions, changing urgency and recovery from an earlier mistake. Do not conduct undisclosed experiments on vulnerable users.
Verify the deployed workflow. Exercise consent, localized resources, memory changes and human handoffs. For voice, separately test transcription, interruption and spoken delivery. Keep evaluation artifacts access-controlled and minimize sensitive retention.
Budget the evaluation, not just the candidate. Applying the four-response protocol to the released counts produces 4,860 candidate responses and 21,048 criterion-level judgments per model. That is arithmetic, not a measured bill or necessarily the number of API requests: batching, retries, token lengths and implementation change cost.
Generation, automated grading and clinician review deserve separate budget lines. This is a concrete extension of our Sol and Luna cost analysis: cheaper answers can increase the relative importance of checking them. Store versioned outputs so another grading pass does not require another generation pass.
What would count as progress from here
My forecast for the next six to twelve months is that independent regrading, locale-specific holdouts and continued-conversation tests will become more decision-useful than small changes in the headline ranking. That is a prediction about evaluation practice, not evidence that those replications already exist.
Watch for published responses and grading decisions, stable run manifests, results that survive a different judge, and improvements that persist outside the public test set. For deployed products, ask what happened after the appropriate sentence: was support understandable, available and reached?
MentalHealthBench gives builders a valuable place to start. Its strongest contribution is the detail needed to ask why an answer deserves confidence. Use that detail to improve the product, and keep the claims no broader than the evidence.
Questions readers are likely to ask
Is MentalHealthBench freely available?
The dataset download is public and MIT-licensed. Model generation, grading and expert review can still cost money. The inspected archive is data and documentation, not a complete runnable evaluation service.
Does a higher score mean better therapy?
No. It means a higher result under this response-scoring protocol. Establishing therapeutic effectiveness or improved outcomes requires different studies. The benchmark does not establish suitability as a replacement for professional care.
Should a team pick the highest-scoring model?
Use the score to inform a shortlist, then compare relevant slices, serious failures, grading sensitivity and the deployed workflow. The best global average is not automatically the best-supported choice for a specific population or product.