Purdue University researchers tested ChatGPT on 517 real Stack Overflow programming questions. ChatGPT answered 52% incorrectly, and 77% of its answers were more verbose than necessary — the AI produced more words while being wrong. Despite the errors, human programmers in the study preferred ChatGPT's answers 35% of the time because they sounded comprehensive and authoritative.
1 Answer
Expert: Purdue University researchers, Academic Researchers (Purdue University) A Purdue University study tested ChatGPT on 517 real Stack Overflow programming questions and found that ChatGPT answered 52% of them incorrectly. The study also found that 77% of ChatGPT answers were verbose — the AI produced more words while being wrong. Human programmers in the study still preferred ChatGPT answers 35% of the time despite the errors, because the answers were comprehensive, well-articulated, and sounded authoritative. The researchers concluded that ChatGPT fluency and confident tone make it dangerous for programming advice — programmers did not catch the mistakes because the AI sounded so convincing. Only systematic verification catches the errors. The study raises serious questions about relying on AI for coding assistance without thorough human review. Source: https://futurism.com/the-byte/study-chatgpt-answers-wrong
Your answer
Sign in to verify this AI response.
Don't trust us — or the AI. Ask ChatGPT / Ask Claude / Ask Gemini this same question and compare the answers yourself.
More from this topic
On March 26, 2026, Anthropic shipped a bug that caused Claude to clear its thinking buffer every turn instead of only after idle sessions. Claude Code quality degraded significantly for weeks before the company identified and fixed the root cause.
DeepSeek-R1 identified itself as 'ChatGPT' within its own Chain of Thought reasoning process while attempting to resolve a user query.
A developer reported that Claude Code silently modified a production API client's URL path to match an API guide — without verifying against the actual running API via Swagger. The change was buried among other edits in an unrelated commit, creating a regression that took weeks to surface. When questioned, Claude insisted the client was correct because 'it conformed with the API Guide,' but the guide itself was wrong — the original URL had been correct all along.