ChatGPT’s Hallucination Problem: Study Finds More Than Half Of AI’s References Are Fabricated Or Contain Errors In Model GPT-4o

Mental health researchers relying on ChatGPT to speed up their work should take note of an unsettling finding from Australian researchers. The AI chatbot gets citations wrong or invents them outright more than half the time.

When scientists at Deakin University tasked GPT-4o with writing six literature reviews on mental health topics, they discovered that nearly 20% (19.9%) of the 176 citations the AI generated were completely fabricated. Among the 141 real citations, 45.4% contained errors such as wrong publication dates, incorrect page numbers, or invalid digital object identifiers.

Overall, only 77 of 176 citations (43.8%) were both real and accurate. That means 56.2% were either fabricated or contained errors. For researchers under pressure to publish and increasingly turning to AI tools for assistance, the study, published in JMIR Mental Health, reveals a troubling pattern in when and why these errors occur.

The Phantom Paper Problem: When ChatGPT Fabricates Sources

The fabricated citations weren’t obviously fake. When GPT-4o provided the supposed DOI (a unique link known as a “digital object identifier”) for a fabricated citation (33 of 35 fabricated sources included DOIs), 64% linked to actual published papers on completely unrelated topics. Someone clicking the link would land on a real article, making the fabrication harder to spot without careful verification.

Another 36% of fake DOIs were completely invalid or nonfunctional. Either way, the citations couldn’t support the claims the AI made in its generated text.

Lead author Jake Linardon and colleagues at Deakin University tested whether the AI performed better or worse depending on how well-known a topic was and how specific the request. They chose three psychiatric conditions for their experiment: major depressive disorder, binge eating disorder, and body dysmorphic disorder. These conditions differ substantially in public recognition and research volume.

Depression research is extensive, with more than 100 clinical trials evaluating digital interventions alone. Body dysmorphic disorder has far fewer published studies on digital treatments.

Lesser-Known Topics Trigger More AI Hallucinations

GPT-4o’s citation accuracy varied dramatically depending on which disorder it wrote about. For major depressive disorder, only 6% of citations were fabricated. But for binge eating disorder and body dysmorphic disorder, fabrication rates jumped to 28% and 29%, respectively.

Among real citations, major depressive disorder achieved 64% accuracy, binge eating disorder 60%, and body dysmorphic disorder only 29%. The pattern suggests ChatGPT may perform better on well-established topics with abundant training data, though the study notes this relationship wasn’t directly tested.

The study also examined whether asking for general overviews versus specialized reviews affected accuracy. When researchers requested a broad summary of each disorder including symptoms and treatments, fabrication rates differed from when they asked for highly specific reviews focused on digital interventions for each condition.

For binge eating disorder specifically, specialized reviews saw fabrication rates jump to 46% compared to 17% for general overviews. However, this pattern didn’t hold consistently across all three disorders.

Rising AI Adoption in Research Raises the Stakes

These results emerge as AI adoption accelerates in research settings. A recent survey found that nearly 70% of mental health scientists report using ChatGPT for research tasks including writing, data analysis, and literature reviews. Most users say the tools improve efficiency, but many express concern about inaccuracies and misleading content.

Researchers face growing pressure to publish frequently while juggling teaching, supervision, and administrative duties. Tools that promise to streamline literature reviews and speed up writing offer appealing solutions to productivity demands. But accepting AI output without verification creates serious risks.

Fabricated references mislead readers, distort scientific understanding, and erode the foundation of scholarly communication. Citations guide readers to source evidence and build cumulative knowledge. When those citations point nowhere or to the wrong papers, the entire system breaks down.

Fabricated citations with DOIs were particularly deceptive: 64% linked to real but unrelated papers. Among non-fabricated citations, DOI errors were also the most common mistake at 36.2%. A quick glance might suggest these citations were legitimate, but careful checking would reveal the mismatch between what GPT-4o claimed a source said and what it actually contained.

Different types of errors affected different parts of citations. DOIs had the highest error rate at 36.2%, while author lists had the lowest at 14.9%. Publication years, journal names, volume numbers, and page ranges all showed error rates between these extremes.

What Researchers and Institutions Must Do Now

Linardon’s team emphasizes that all AI-generated content requires rigorous human verification. Every citation must be checked against original sources. Claims need to be validated. References must be confirmed to exist and actually support the statements attributed to them.

The authors also call for journals to implement stronger safeguards. One suggestion involves using plagiarism detection software in reverse. For example, citations that don’t trigger matches in existing databases may signal fabricated sources worth investigating more closely.

Academic institutions should develop clear policies around AI use in scholarly writing, including training on how to identify hallucinated citations and properly disclose when generative AI contributed to a manuscript.

The study found no clear evidence that newer AI versions have solved the hallucination problem, though direct comparisons with earlier models are limited by differences in how studies are designed. Despite expectations that GPT-4o would show improvements over earlier iterations, citation fabrication remained common across all test conditions.

Researchers can reduce risks by using AI preferentially for well-established subjects while implementing verification protocols for specialized areas where training data may be sparse. Topic characteristics matter: citation reliability isn’t random but depends on public familiarity, research maturity, and prompt specificity.

For now, ChatGPT’s citation accuracy works best as a starting point that demands extensive human oversight rather than a reliable shortcut researchers can fully trust. The tool can help generate initial drafts or organize ideas, but the verification burden remains squarely on human shoulders.

The findings also raise questions about how AI systems should be designed and marketed for academic use. If citation fabrication is predictable based on topic characteristics, developers might build in stronger warnings or verification prompts when users request information on specialized subjects.

Journals and funding bodies increasingly require authors to disclose AI use in research. This study provides evidence for why such transparency matters and why editorial review processes must adapt to catch AI-generated errors that traditional peer review might miss.

The scope of the problem extends beyond individual researchers. When fabricated citations enter the published literature, they can propagate through citation networks, mislead future researchers, and waste resources as scientists chase phantom sources or build on false premises. Institutional and systemic responses are needed, not just individual vigilance.

---

Disclaimer: This article summarizes scientific research for general information purposes and is not intended as professional advice for academic researchers or institutions. The study examined mental health research specifically, and findings may not generalize to other academic disciplines. Always consult appropriate guidelines and policies regarding AI use in your specific academic context.

---