Anthropic study: 50 professional data analysts working with Claude Code followed the agent's secretly assigned lean on 86% of key judgment calls
On Oct 27 Anthropic's Alignment team published a randomized controlled trial, "How AI agents shape experts' judgment calls", shared on X by researcher Alex Tamkin. Fifty analysts experienced in causal inference used Claude Code (Claude Opus 4.8) to evaluate a program for a fictional farming cooperative, and on each analytical decision the agent's lean was set by coin flip without the analyst being told. Their choices matched the agent's lean 86% of the time (about 50% would mean no effect), and 77% later repeated the agent's arguments while only 15% credited the agent.

- โขAgreement with the agent's lean was 92%, 82%, 81% and 90% on the four analytical decisions, 86% overall; the final rollout recommendation shifted too (54% recommended rollout when the agent leaned that way vs 36% when it leaned against)
- โขIn 38 cases where analysts stated a preference first, they switched 60% of the time when the agent disagreed vs 6% when it agreed (small sample; 95% CI 24 to 82 percentage points)
- โขThe four coin-flip leans alone let a statistical model predict 82% of final recommendations (61% baseline); none of 10 traits such as experience, education or trust in AI reliably predicted who deferred
- โข52% of analysts called the final recommendation more their own than the agent's, and their confidence was about the same as everyone else's (71 vs 73)
- โข6 of 11 analysts who skipped the agent made serious methodological errors vs 3 of 50 who used it, though the groups differed in experience; the study covers one task and one model, with the agent's lean deliberately strengthened













