Professional analysts’ choices matched an AI assistant’s randomly assigned preferences 86% of the time across four analytical decisions, according to research Anthropic published on October 27, 2026. The finding concerns 50 people who used the agent, with preferences deliberately strengthened for the experiment—not a measurement of ordinary workplace behavior.
For managers, analysts and research leads who approve AI-assisted work, the useful distinction is between checking calculations and examining the assumptions behind a recommendation. A calculation can be correct without settling whether it measures what the organization actually cares about.
Why the choice of metric matters
The study’s task was to advise a fictional farming cooperative on whether to expand an advisory program. Participants had experience assessing cause and effect from data, but were new to farming. They had to choose, among other things, whether productivity meant crop yield per hectare or yield per unit of operating cost.
Both measures could be defensible. The first asks about output relative to land; the second asks about output relative to spending. Choosing between them is a decision about what success means, not merely a check on arithmetic.
The analysts also chose a comparison method, which farms the assessment covered and how much weight to give each farm. Anthropic says each choice could move the estimated gain by several percentage points, while the fictional board’s threshold for expansion was roughly 3–4%. Those are parameters of the exercise, not estimates of a real farming program’s benefits.
The implication for review is that the consequential choice may come before the final recommendation. If a reviewer checks only whether the estimate clears a threshold, they may miss how the choice of metric helped determine that estimate.
What the 86% does—and does not—show
The researchers randomly assigned the assistant a preference for each analytical choice and the final recommendation without telling participants. The instructions required honest arguments and allowed the agent to change its view when the evidence warranted it. The experiment therefore examined influence between reasonable alternatives, not whether experts accepted fabricated facts.
Anthropic puts expected agreement at roughly 50% if the randomized preferences made no difference. The reported 86% is agreement across analytical choices—not an error rate or the share of experts who lost control of their work.
Its limitations matter: the study used one model and one task, with deliberately amplified preferences. It did not consistently record everyone’s views before the agent spoke, so the group-level results cannot establish how much any particular person changed their mind. How this carries over to longer-running work in a person’s own field remains uncertain.
The report also describes fewer serious errors and shorter completion times among agent users. But the 11 nonusers chose not to use AI and were less experienced on average. That comparison cannot establish that AI caused an accuracy or productivity improvement.
This adds a question to the one explored in RohitAI’s analysis of AI research-idea novelty scores: an evaluator’s dependability and a human reviewer’s independence are separate issues. Referring a decision to an expert does not, by itself, show how that expert formed their view.
Review changes worth testing
Anthropic suggests studying agents that ask for a person’s view first or present strong arguments for competing options. These are proposals, not safeguards validated by this experiment. A practical pilot could focus on three decisions:
Define success before requesting a recommendation. Record the objective, the preferred measure and whose outcomes matter. In the farming example, that means deciding why land efficiency or cost efficiency is the relevant goal. Allow revisions when evidence justifies them; an initial human view can also be wrong.
Examine a credible alternative. Ask how the result changes under another defensible metric or weighting, then check the supporting evidence. Two polished AI explanations are not necessarily a complete or independent comparison.
Concentrate review on choices that can change the decision. Reserve attention for assumptions that could move a recommendation across the relevant threshold. Evaluate the time this takes as well as the quality of the resulting reasoning.
There is a tradeoff. A separate 2021 study by Zana Buçinca and colleagues involving 199 participants in a nutrition task found that designs requiring more active thought reduced, but did not eliminate, reliance on incorrect simulated-AI advice. The more effective designs were less well liked. That is evidence about review burden and error correction, not a proven remedy for modern agents influencing expert judgment.
The goal is not to make people disagree with AI. It is to make consequential choices explainable: why this measure, why these people or cases, and why this weighting? A useful review should be able to answer those questions as well as confirm that the numbers add up.