Expert: Prof. Mesut Cicek, Professor, Washington State University When users asked ChatGPT to evaluate whether scientific hypotheses from peer-reviewed papers were true or false, it appeared impressively accurate — getting about 76% correct on the surface. But that headline number hid a deeper problem. Researchers at Washington State University tested ChatGPT on over 700 hypotheses from published scientific papers, repeating each query 10 times to measure consistency. The results revealed two critical failures: 1. **False positives were catastrophic.** While ChatGPT was decent at identifying true hypotheses, it correctly identified false hypotheses only 16.4% of the time. The other 83.6% of the time, it told users that false claims were true — a dangerous pattern when people rely on AI to separate scientific fact from fiction. 2. **Inconsistency across repeated queries.** When asked the same question 10 times, ChatGPT gave consistent answers only 73% of the time. That means nearly 3 out of 10 times, it changed its mind about the same hypothesis — offering different "truths" depending on the moment you asked. The study highlights a fundamental limitation: LLMs like ChatGPT do not reason about science — they pattern-match. Their apparent accuracy comes from correctly handling easy cases, while their performance on genuinely challenging or false hypotheses drops to near-chance levels. Users who trust AI for scientific information risk being misled by confident-sounding but incorrect answers, especially when the AI tells them a false claim is true. Source: https://www.sciencedaily.com/releases/2026/03/260317064452.htm