Tech AI LatestAi2 study finds LLM novelty judges flip verdicts when prompts change, from 93.5% to 40.9%
Ai2 reported on 27 October 2026 that LLMs judging research novelty are prompt-sensitive: GPT-5.4 fell from 93.5% to 40.9% accuracy after one wording change.
Summary
The Allen Institute for AI (Ai2) published a study on 27 October 2026 testing whether large language models can judge if a research idea is novel. Using ideas drawn from ICLR 2026 reviews, it found verdicts swing with small prompt changes: GPT-5.4's accuracy on human-versus-generated pairs fell from 93.5% to 40.9%. A simple Claude Opus 4.6 prompt beat purpose-built novelty checkers at a fraction of the cost. Data and code are open.