Ai2 study finds LLM novelty judges flip verdicts when prompts change, from 93.5% to 40.9%

What Ai2 studied

Researchers at the Allen Institute for AI (Ai2) published a study on 27 October 2026 on LLM "novelty judges", models asked whether a research idea is new compared with prior work. The paper, "Old Ideas, Novel Problems", is on arXiv, with data on Hugging Face and code on GitHub.

To build the benchmark, the team mined ICLR 2026 peer reviews for passages about originality rather than using acceptance as a proxy. The high-novelty pool has 154 ideas from accepted papers in the top 10% of their area, where reviewers praised novelty and none disputed it. The lower-novelty pool has 145 ideas from rejected papers in the bottom 10%, where reviewers raised originality concerns. A second setting pairs the 154 human ideas with 154 ideas from a simple LLM generator, which are assumed, but not verified, to be less novel.

What the study found

Verdicts depended heavily on prompt wording. On human-versus-generated pairs, GPT-5.4's accuracy fell from 93.5% to 40.9%, below chance, when the prompt changed from saying reviewers found exactly one idea novel to simply asking which idea was novel. The model changed its answer on more than half the pairs. On human-only pairs the same change moved accuracy by about seven points, but Claude Opus 4.6 still shifted by 21 points. No single prompt worked best for every judge.

More search and reasoning did not reliably help. An agentic GPT-5.6 Sol judge with live arXiv search scored 0.78 macro-F1 against 0.89 for Claude Opus 4.6, at about 10 times the cost per decision. A simple Claude Opus 4.6 prompt with majority voting scored 0.854 for $2.59 in total, while the Idea Novelty Checker scored 0.642 for $30.66 and Sakana AI's AI Scientist evaluator 0.772 for $82.05.

Why it matters

Some automated research systems generate ideas and then use another model to rank them, and this study tests that step. Ai2's warning is that "judges that appear reliable on human-authored research may behave differently on AI-generated ideas". The authors note their own limits: labels for generated ideas are weak, the expert check covered only 30 pairs chosen towards judge errors, and the benchmark uses one venue. Microsoft researchers, in a separate case, reported that unguided AI attempts fell short on the KLS conjecture before human-steered agents helped.

What to watch next

Ai2 released a pipeline for building new evaluation sets, so tests in other fields and venues should follow. Developers of automated research systems can rerun the benchmark against their own judges. More evaluation research is under AI Research.

Questions people ask

Can LLMs judge whether a research idea is novel?

Only unreliably, according to Ai2's 27 October 2026 study. Verdicts changed sharply with prompt wording, and GPT-5.4's accuracy fell from 93.5% to 40.9% after one change.

Which novelty judge did best in the Ai2 study?

A simple Claude Opus 4.6 prompt with majority voting scored 0.854 macro-F1 for $2.59, ahead of purpose-built tools such as Idea Novelty Checker and Sakana AI's AI Scientist evaluator.

Is the benchmark public?

Yes. The data is on Hugging Face as allenai/novelty-judge-bench and the code is on GitHub.

Sources

1. Can we trust LLMs to judge whether a research idea is novel? (Ai2, 27 Oct 2026)