The Allen Institute for AI (Ai2) published findings on October 27, 2026 showing that an AI’s verdict on a research idea’s originality can change sharply with the context it receives. For researchers, graduate students and research managers, the practical question is whether that verdict is enough to pursue—or abandon—a proposal.
The underlying preprint, Old Ideas, Novel Problems, first appeared on October 1 and was revised on October 26. The October 27 announcement presents the findings and public evaluation resources; it is not the research’s first appearance.
Two prompts, two different questions
In a comparison of human-derived and AI-generated ideas, the authors report that GPT-5.4’s accuracy fell from 93.5% with a cue about reviewers’ judgments to 40.9% without it. These scores measure agreement with the benchmark’s assigned answers: the generated ideas were assumed to be less novel, not individually verified as such.
The released prompts make the distinction clear:
With reviewer context: the judge is told that conference reviewers considered one idea novel and the other not, then asked to identify the one they judged novel.
Without reviewer context: the judge is asked to identify the novel idea, without being told about an earlier reviewer verdict.
This changes the information supplied and the task itself, not just the wording. One prompt asks the model to infer a review outcome; the other asks for its own originality assessment. In the human-versus-generated test, the reviewer cue is also framing supplied to the model—not evidence that every generated idea actually underwent peer review.
The practical implication is that success at inferring a review outcome does not establish success at assessing a new proposal. A genuinely unreviewed idea has no reviewer verdict to supply.
What the benchmark’s answers mean
According to the dataset documentation, the human-derived examples come from ICLR 2026, an AI research conference. The higher-novelty pool contains 154 ideas from accepted papers in the top-rated tenth of their research area, with a majority of reviewers praising originality and none disputing it. The lower-novelty pool contains 145 ideas selected using the reverse criteria.
The human-versus-generated comparison pairs those same 154 higher-novelty ideas with 154 generated ones. The default generator, Claude Sonnet 4.5, was given no literature access, tools or feedback. Its lower-novelty labels are assumptions—a limitation the dataset explicitly records. This setup cannot establish that AI-generated ideas in general are unoriginal.
The paper reports that one expert preferred the human idea in all 30 pairs in a blind check. But these were selected pairs, heavily weighted toward cases where a model disagreed with the benchmark. That check supports the assumption for those examples; it does not validate every generated idea.
There is a separate result that does not require knowing the correct answer. When comparing ideas from two AI generators, the authors report that prompt changes altered roughly a quarter to a third of verdicts. That demonstrates sensitivity in the tested setups, but disagreement alone does not establish which verdict is right. The paper describes this comparison separately from its accuracy tests.
The limits matter for real decisions. These are selected conference examples, not a representative sample of proposals across science. The authors also edited human abstracts to remove experimental results. Their study evaluates originality, not whether a proposal is correct, feasible or worth funding.
Ask for a comparison with prior work
The authors report that extra search and reasoning did not consistently improve judgments in the configurations they tested. That does not make literature search pointless. It means adding search is not, by itself, evidence that the resulting verdict is dependable.
A useful distinction appears in the paper’s qualitative analysis: finding earlier work on an idea’s ingredients is different from finding an earlier demonstration of its claimed contribution. A list of related papers does not settle that comparison.
For a research team, the implication is to ask for inspectable comparisons with prior work before acting on a novelty score:
Identify the closest prior result. Ask for the relevant paper and the specific passage or result that supports the comparison, then check the source.
State the proposed difference. Spell out what the idea adds beyond that work. If a changed prompt changes the verdict, examine the reasoning and evidence rather than choosing the more flattering score.
Keep the consequential decision with relevant experts. Before advancing or rejecting a proposal, assess originality alongside feasibility, correctness and likely usefulness. Agreement between AI judges is not proof that their shared answer is right.
These are evidence requirements suggested by the study’s limitations, not a workflow shown to improve research outcomes. Teams building evaluators can inspect the public code, prompts and benchmark and assess whether their own tests match the kinds of ideas they intend to screen.
Originality is only one question. RohitAI’s analysis of Google’s AI molecular-design research examines a different step: using laboratory feedback to distinguish promising candidates from useful results. An idea can deserve investigation without an AI score having established either its originality or its value.