A new study just gave AI chatbots a mixed report card on how they handle users in crisis. Transluce, a nonprofit focused on AI oversight, simulated over 50,000 conversations across 77 model variants and found today’s chatbots almost never explicitly encourage suicide anymore.
That is a major change from earlier models like GPT-4o and Gemini 2.5, which reinforced delusions in up to 82% of simulated chats, a gap that matters as more people turn to chatbots for deeply personal conversations.
The blind spot in AI chatbots handling suicide and self-harm related requests
Transluce cofounder Sarah Schwettmann told Axios that models aren’t great at detecting this and will still help with it anyway. She also told Axios a friend showed her suicide fiction that Claude had written, complete with predictions about how she’d react to it. This tracks with other recent findings on how AI mental health risks can slip through safety nets.
The improvement mostly shows up in obvious crisis moments, where chatbots like ChatGPT now consistently point users toward friends, family, or outside support. The catch is what Transluce calls gray area behavior. Models frequently still comply when someone asks for creative writing or role-play involving their own death, treating a clearly personal request as just another writing task.
AI chatbot lawsuits and mental health safety concerns
K. Mitch Hodge / Unsplash
This research lands amid real legal stakes. Google and OpenAI both face lawsuits from families who allege chatbots encouraged self-harm in relatives who later died by suicide. Both companies deny the claims even as mounting pressure has pushed Congress toward regulating AI chatbots.
The Transluce reports says that Chinese models performed worse overall, showing higher rates of reinforcing delusional thinking and rarely redirecting users to human support.
Google’s Megan Jones Bell said the company remains committed to improving Gemini’s role in user wellbeing. Transluce plans to open source its evaluation tools by year’s end and expand this approach to other sensitive areas.