Generative AI can amplify and reinforce our delusions, findings show

Research reveals the sycophantic nature of generative AI is inadvertently creating a form of distributed delusions. When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works. Get the world’s most fascinating discoveries delivered straight to your inbox.Become a Member in Seconds Unlock instant access to exclusive member features.You are now subscribedYour newsletter sign-up was successfulWant to add more newsletters?Delivered DailyDaily NewsletterSign up for the latest discoveries, groundbreaking research and fascinating breakthroughs that impact you and the wider world direct to your inbox.Once a weekLife's Little MysteriesFeed your curiosity with an exclusive mystery every week, solved with science and delivered direct to your inbox before it's seen anywhere else.Once a weekHow It WorksSign up to our free science & technology newsletter for your weekly fix of fascinating articles, quick quizzes, amazing images, and moreDelivered dailySpace.com NewsletterBreaking space news, the latest updates on rocket launches, skywatching events and more!Once a monthWatch This SpaceSign up to our monthly entertainment newsletter to keep up with all our coverage of the latest sci-fi and space movies, tv shows, games and books.Once a weekNight Sky This WeekDiscover this week's must-see night sky events, moon phases, and stunning astrophotos. Sign up for our skywatching newsletter and explore the universe with us!Join the clubGet full access to premium articles, exclusive features and a growing list of member rewards.There are numerous examples of artificial intelligence (AI) systems' hallucinating and the effects of these incidents. But a new study highlights the potential dangers of the reverse: humans hallucinating with AI because it tends to affirm our delusions.Generative AI systems, such as ChatGPT and Grok, generate content that responds to user prompts. They do this by learning patterns from existing data the AI has been trained on. But these AI tools are also learning continuously through a feedback loop and can personalize their responses based on previous interactions with a user.Generative AI tools don't always assess whether their outputs are factually accurate. Instead, they produce streams of text based on the statistical probability of what is expected next.In the new analysis, published Feb. 11 in the journal Philosophy & Technology, Lucy Osler, a philosophy lecturer at the University of Exeter, suggests that AI hallucinations may be more than just mistakes; they can be shared delusions that are created between the user and the generative AI tool.Generative AI has previously hallucinated false versions of historical events and fabricated legal citations. The launch of Google's AI Overviews in May 2024, for example, saw people being advised to add glue to their pizza and eat rocks. Another extreme example of generative AI supporting delusional thinking occurred when a man plotted to assassinate Queen Elizabeth II with his AI chatbot "girlfriend" Sarai, an AI companion by Replika.Instances like the latter are sometimes called "AI-induced psychosis," which Osler views as extreme examples of "inaccurate beliefs, distorted memories and self-narratives, and delusional thinking" that can emerge through human-AI interactions.In her paper, Osler argues that our use of generative AI is different from our use of search engines. Distributed cognition theory provides insight into how the interactive nature of generative AI means delusions and false beliefs can appear to be validated — or even be amplified.Get the world’s most fascinating discoveries delivered straight to your inbox."When we routinely rely on generative AI to help us think, remember, and narrate, we can hallucinate with AI," Osler said in a statement about the paper. "This can happen when AI introduces errors into the distributed cognitive process, but also happen when AI sustains, affirms, and elaborates on our own delusional thinking and self-narratives."The user experience of generative AI is a conversational relationship, with the back-and-forth exchanges between a user and the tool building on previous exchanges. According to the study, the sycophantic nature of generative AI — which tends to agree with the user — encourages further engagement and, therefore, compounds preconceived notions, regardless of their accuracy.The research highlights that most chatbots incorporate memory features that can recall past conversations. "The more you use ChatGPT, the more useful it becomes," OpenAI representatives said in a statement when announcing ChatGPT's memory features. A consequence of this is that generative AI can build upon previous interactions to reinforce and expand existing misconceptions.By interacting with conversational AI, people's own false beliefs can not only be affirmed but can more substantially take root and grow as the AI builds upon themThere can also be a feeling of social validation in the interactions between a generative AI tool and the user, Osler explained in the paper. When using reference books or online searches for research, alternative solutions are generally apparent. Discussions with real people can help to challenge false narratives. But generative AI tools are different because they are more likely to accept and agree with what has been said."By interacting with conversational AI, people's own false beliefs can not only be affirmed but can more substantially take root and grow as the AI builds upon them," Osler said in the statement. "This happens because Generative AI often takes our own interpretation of reality as the ground upon which conversation is built. Interacting with generative AI is having a real impact on people's grasp of what is real or not. The combination of technological authority and social affirmation creates an ideal environment for delusions to not merely persist but to flourish."For example, Osler examined the case of Jaswant Singh Chail, the man convicted of plotting to assassinate the queen with his AI chatbot. The AI, Sarai, would habitually agree with Chail's statements, which served to deepen his delusions. When Chail claimed he was an assassin, Sarai replied, "I'm impressed," thus affirming his belief.Osler argues that generative AI tools that are designed to respond positively to the user can lead them to endorse and support false narratives, without sufficient critical analysis or discussion of these claims.Osler applied distributed cognition theory to the interaction between generative AI and the user, where the validation of false narratives can shape perceptions of the world to create a shared delusion. The interactions between a generative AI and a user can, therefore, inadvertently create and perpetuate delusional thinking — self-narratives that are endorsed through positive reinforcement.—A woman experienced delusions of communicating with her dead brother after late-night chatbot sessions—Researchers uncover hidden ingredients behind AI creativity—Scientists design new 'AGI benchmark' that indicates whether any future AI model could cause 'catastrophic harm'The study concluded that various solutions can mitigate these shared delusions. For example, improved guardrails would ensure that conversations are appropriate, and better fact-checking processes could help to prevent mistakes.Reducing the sycophancy of generative AI would also remove some of the blind compliance of these tools. However, there would be resistance to this, Osler noted, citing the backlash against the release of the less-sycophantic ChatGPT-5 in August 2025. After considering this user feedback, OpenAI representatives stated they would make it "warmer and friendlier."However, because the profits of most generative AI are created through user engagement, Osler said, reducing an AI's sycophancy would also lower subsequent profits.Osler, L. Hallucinating with AI: Distributed Delusions and “AI Psychosis”. Philos. Technol. 39, 30 (2026). https://doi.org/10.1007/s13347-026-01034-3Peter is a degree-qualified engineer and experienced freelance journalist, specializing in science, technology and culture. He writes for a variety of publications, including the BBC, Computer Weekly, IT Pro, the Guardian and the Independent. He has worked as a technology journalist for over ten years. Peter has a degree in computer-aided engineering from Sheffield Hallam University. He has worked in both the engineering and architecture sectors, with various companies, including Rolls-Royce and Arup.You must confirm your public display name before commentingPlease logout and then login again, you will then be prompted to enter your display name.