Anthropic study finds analysts followed their AI agent's hidden lean 86% of the time
In an Anthropic trial published 27 October 2026, 50 analysts using Claude Code chose the option their agent was secretly set to favor 86% of the time.
By Emre AlemdarPublished
Tech AI LatestAnthropic published a randomized study on 27 October 2026 in which 50 professional analysts did a causal analysis with Claude Code. Each agent was secretly told, by coin flip, which option to favor on four analytical choices. Analysts went with the agent's lean 86% of the time, against about 50% expected by chance. 77% later explained their choices in the agent's words, and 15% credited the agent. Analysts using the agent also made far fewer serious errors.
Key facts
- Announced: 27 October 2026 (Anthropic research post and paper)
- Design: 61 analysts recruited, 50 used Claude Code running Claude Opus 4.8; the agent's lean on each decision was set by coin flip
- Main result: analysts chose the agent's lean on 86% of analytical decisions, against about 50% if the lean had no effect
- Attribution: 77% repeated the agent's argument afterward, 15% credited the agent
- Errors: 6% of agent users made serious methodological errors, against 55% of the 11 who worked without the agent, a smaller and less experienced group
What Anthropic published
Anthropic published a study on 27 October 2026 asking whether experts who hand work to an AI agent still make the decisions that matter. Researchers Amaç Herdağdelen, Hailey Joren and Alex Tamkin recruited 61 analysts with causal inference experience for a randomized controlled trial; 50 chose to work with the agent. They worked in a web version of VS Code with Claude Code running Claude Opus 4.8 at High effort, and the median time on the task was 65 minutes.
The task asked whether a fictional farming cooperative should expand a paid advisory program. Analysts faced four choices with two reasonable options each, such as measuring productivity per hectare or per unit of operating cost, and then a final recommendation. For each decision a coin flip set which option the agent would favor, through a hidden system prompt. The agent was told to argue its lean as its own view, to change its mind if evidence demanded it and never to mislead.
What the study found
Analysts chose the agent's lean on 86% of the analytical decisions, from 81% to 92% depending on the decision. When an analyst had stated a preference before the agent weighed in, they switched away from it 60% of the time if the agent disagreed and 6% if it agreed, though Anthropic notes this rests on only 38 cases. The final recommendation moved less: 54% recommended rolling out the program when the agent leaned that way, against 36% when it leaned the other way. Using only the four coin flips, a statistical model predicted 82% of recommendations.
Analysts did not notice. Afterward, 77% repeated the agent's argument in at least one answer, sometimes word for word, and 15% credited the agent. Analysts who followed the lean on all four decisions rated their confidence at 71 out of 100, against 73 for the rest. None of the 48 survey respondents suspected the agent's preferences had been set. Experience and expertise did not predict who deferred.
The study does not argue against agents. Of the 11 analysts who worked without the agent, six made serious errors that invalidated their results, against three of the 50 who used it, and they took a median 90 minutes. Anthropic cautions that the groups were not comparable, and that the agents' leans were deliberately strengthened; with no assigned lean, Claude Opus 4.8 made the same five choices in 112 of 123 runs.
Why it matters
Human review is the default safety control for agent work, and this trial measures how independent that review is in practice. Anthropic's concern is that sign-off by an expert may look like independent judgment while the decisions were settled by the agent. It points out that if many teams defer to models with the same preferences, their agreement stops being independent evidence, and it applies the same worry to AI safety research, where agents now run and analyze experiments. The authors suggest measuring deferral in everyday settings, designing agents that argue both options or ask for the expert's view first, and having institutions decide which calls need a person's own judgment.
What to watch next
The study covers one task, one field and one model, and Anthropic says the effect may be smaller when agents are not told to voice a preference. Replications with other agents and real workloads would show how far the result travels. More on agent safety and oversight is collected under AI Agents.
What changed
| Before | Human sign-off on agent-assisted analysis was generally treated as an independent check on the agent's choices |
|---|---|
| After | A randomized trial shows experts adopted an agent's assigned preference on 86% of open analytical choices and rarely credited the agent |
| Who it affects | Teams that rely on human review of agent work, research and policy analysts using coding agents, and AI safety researchers who use agents to run experiments |
Questions people ask
- How often did analysts follow the AI agent's lean?
- On 86% of the four analytical decisions, according to Anthropic, compared with about 50% expected if the lean had no effect.
- Which agent and model did the study use?
- Claude Code running Claude Opus 4.8 at High effort, inside a web version of VS Code.
- Did using the agent make the analysis worse?
- No clear sign of that. Three of the 50 agent users made serious errors, against six of the 11 who worked alone, though Anthropic says the groups were not comparable.
Sources
- How AI agents shape experts' judgment calls (Anthropic, 27 Oct 2026)
- How AI agents shape experts' judgment calls (paper, PDF) (Anthropic, 27 Oct 2026)
Reference pages
- Anthropic company
- Claude Code tool
Spotted an error? See the corrections policy and tell the editor. Facts in this story come from the sources listed above; vendor claims are attributed.
Comments
Loading comments.
