I wrote this article at roughly 3 am, sitting in a clinic lobby while my friend talks to an oncologist in the cabin behind me. The air is saturated with the smell of medicines, of course, and the atmosphere has an unmistakable whiff of gloom that compels me to exit the compound as soon as possible. The overwhelming sensation, however, is that of rage and fury.
Merely an hour ago, I had received a call from my 24-year-old friend, complaining about a severe headache and pain in her stomach. She is currently undergoing treatment for breast cancer, and after seeing her go through the ordeal over the past two years, I rushed to her place, hoping it was not serious. Without asking too many questions, I looked up the nearest hospital and rushed her for a checkup.
On the way to the hospital, I asked her when the pain started and if she had experienced any other worrisome symptoms. Her answer shocked me, and to such an extent that I wanted to snatch her phone and crush it to pieces. “After I returned from work, I felt some dizziness and also felt that something was not right in my gut. So, I asked ChatGPT if I could take painkillers, and a few hours after I took them, the gut pain worsened with a burning sensation. I also vomited a few times.”
Who must act responsibly?
Unsplash
The admission shook me, and the casual tone in which she described her ChatGPT interactions was even more frightening. But there are so many layers to this mishap that I’ve decided to ban AI for my loved ones, especially when it comes to health and wellness-related conversations. As someone who is extremely sensitive to medications and who is going through cancer treatment, asking an AI about taking medicines was the last thing you would expect any sane person to do.
I don’t know exactly who is to blame for this potentially life-threatening misadventure. She simply took a picture of a painkiller, fed it to ChatGPT, and asked if she could take two pills at the same time. Her doctor has only recommended her one pill at a time, and that too, when the pain is too severe to tolerate. ChatGPT told her she should take one painkiller, and two of them, only with a doctor’s recommendation.
When I asked why she turned to the AI bot, she later told me ChatGPT “is easier to get answers from. I don’t even know what’s real on the internet.” For a person who had to drop studies after high school and support their family with odd jobs, I understood her predicament. I couldn’t lecture her on the concepts of digital literacy and fact-checking. There was so much wrong with that chat, and both parties are to blame here.
This is where the problem begins. She is not a native speaker of the Hindi language, and when she translated her query into Hindi, it was a grammatical disaster owing to just one word. She wanted to ask if she could take two pills of the medicine appearing in the picture. But her broken translation made it seem as if she was asking ChatGPT if she could take “both medicines.”
This is where ChatGPT also tripped. The picture shows a pill with only its chemical composition visible — Diclofenac and Serraptiopeptidase. The name of the medicine was only half visible. Despite that, the AI bot accurately identified the painkiller medicine as “something like Cipzen-D.” But before that, it also mentioned another medicine, Fenak Plus, whose chemical composition is Diclofenac and Paracetamol, and which she had previously discussed in another conversation.
ChatGPT Unsplash
It’s an entirely different composition, even though both are easily available as over-the-counter painkiller medicines that you can even buy (worryingly) without a proper prescription at local dispensaries. So, broadly, we have one human mistake (with translation) and two mistakes by ChatGPT because it was also pulling knowledge from its memory. The advice, however, was accurate.
OpenAI’s AI bot clearly warned that taking both medicines is not a good move, adding that it could lead to an overdose of Diclofenac. The AI bot also described the after-effects, which included a burning sensation in the stomach, ulcer-related problems, bleeding, and long-term ill-effects on the kidneys.
Do keep in mind that the picture fed to ChatGPT only depicted one painkiller. It further recommended that only one of those painkillers must be taken at a time, and not both. My friend, recalling that both medicines had been prescribed by doctors at one point (though not together), took them both for quick pain relief and soon felt the wrath.
The doctor was also furious and, rightfully so, when my friend told them why she took two pills at the same time, chiding her for not consulting a healthcare professional. It was followed by an injection to ease her pain and a fresh medical prescription for painkillers that don’t harm her fragile body.
As a technology reporter who has written about the risks of AI interactions on sensitive topics, especially health and wellness, I felt like a failure. I felt that I never actually had a chat about the risks of AI bots with my friend. And it was only after the scary incident that I warned her about the risks.
When she confessed to using ChatGPT for medical advice on multiple other occasions in the past, I took her phone and deleted the app. More importantly, I asked her to never touch another AI chatbot — such as Gemini or Claude — for medical advice, or the whole internet, for that matter. The only way forward is talking to a doctor, or visiting a clinic.
The perception problem
ChatGPT Live mode. OpenAI
I don’t blame her entirely for the episode. Given the fashion in which ChatGPT has rooted itself in the daily digital habits of the masses as an all-knowing genie, it’s hard to blame her. People see it as a more convenient (and truthful) answering engine compared to the friction of looking up answers on Google Search, clicking on blue links, and skimming through the answers on a webpage(s).
They’re not wrong, to be fair. Getting straightforward answers from an AI bot like ChatGPT, and that, too, in natural language within a matter of seconds, is far more convenient than entering the maze of Google Search links. Google is also aware of the changing user behavior, which also explains the push being given to the conversational AI mode in Search.
As of today, the conversational answers supplied by Gemini sit atop the regular search results. The idea is identical: giving users summarized answers extracted from the blue link sources on Google Search before they go on a deeper hunt for knowledge by reading multiple sources spaced across half a dozen tabs
To their credit, ChatGPT (or most frontier AI models, for that matter) plays it especially safe when someone seeks health and wellness advice. In an overwhelming majority of such answers, the AI bots ask the user to consult a certified healthcare professional or access the appropriate helpline. But once again, the way these AI chatbots have entered the public discourse as a quick and trusty source of knowledge is what makes them equally scary.
A few days ago, I was watching a heated debate on a prime-time news channel. During the fiery exchange, one of the panelists asked their opponent to go ahead and check the facts by asking ChatGPT. In that moment, I saw ChatGPT being talked about as the modern-age equivalent of finding answers in an encyclopedia, and that too, from a stage that influences millions of people in the world’s second-largest market for AI chatbots.
Lost in translation
The US sits at the top of AI adoption, but the country also has arguably the highest awareness about the risks of using AI chatbots. In India, digital literacy (and AI awareness) is dramatically lower, and hence, the risk potential is much higher. The danger is further exacerbated by the fact that hundreds of thousands of people are using ChatGPT and other AI agents in a native Indian language. And this is one of the key drivers behind the recent episode with my friend: ChatGPT’s language problem.
Apple
In a research study published earlier this year, the MIT Center for Constructive Communication mentions that the accuracy of answers provided by AI chatbots is lower if users are interacting in a non-English language. The SAHARA benchmark, released in December last year, also highlighted a similar discrepancy in AI chatbot behavior when users are talking in a non-English language. The Economist also highlighted numerous papers in which the language issue of LLMs was laid bare.
The language barrier further raises the risk quotient.
Translation is not the only problem, though. Healthwatch England, an independent consumer advocacy group for patients in the UK public health system, has also warned about the risks of using AI transcription tools by doctors and healthcare experts. Here’s a worrying snapshot from our report:
“In one alarming instance, an AI tool reviewed an MRI summary and documented that a patient had demyelination, a serious nerve condition linked to multiple sclerosis. The scan actually revealed “null demyelination”, meaning the issue was not present.”
It’s an obvious flaw that large language models have to fix, but there’s an even bigger issue at play here. AI chatbots need to do better when dealing with sensitive topics. The likes of ChatGPT have implemented parental controls and often tell users to seek help, while also offering helpline numbers and reliable resources. You will also see a warning that ChatGPT is not a doctor, or some form of it.
But these warnings should appear right at the top for all such conversations, and not buried below the first few sentences, which is where most users get their answer and never bother to read further. I would much rather prefer to see a wide banner right at the top, and this protocol should be drilled into the very heart of the AI model’s behavior.
Next, we need awareness. A lot of it. AI companies are busy parading how their LLMs are solving complex mathematical problems, beating humans at olympiads, and helping with medical breakthroughs. But all that happens under the supervision of experts. For an average person, for whom ChatGPT is a source of quick and conversational knowledge, the warnings and disclaimers should take priority.
OpenAI may play it safe, but the public perception of ChatGPT is evolving dangerously fast.
A lot of users see it as a new kind of knowledge engine, one that talks like any other human. There are numerous cases of AI bots giving dangerously bad health advice, and as of 2026, research continues to highlight their risky behavior on sensitive topics. I just hope OpenAI, and every other AI player out there, thinks this through and takes appropriate action.