Are people oversharing sensitive information with AI? Short answer: yes.
A new survey called The AI Privacy Problem by DuckDuckGo Research said about a third of people are sharing secrets with chatbots that they don’t share with friends, family and medical professionals. The number rises to 56% among self-described AI enthusiasts. Researchers also found that 43% of parents reveal things to AI that they don’t tell the most trusted people in their lives, including their kids.
Though the survey didn’t specify what kind of personal information is shared with AI, past reports have shown that we often turn to chatbots for help with physical and mental health issues, financial advice, parenting questions and marital problems.
The problem isn’t about seeking help. The problem is losing control over what happens with the data after it’s shared in chats. Companies can use your private info to train their AI systems. Law enforcement can get it through subpoenas. And if you’re using AI chatbots at work, your employer can likely access your chat history, too.
AI companies try to convince people that chatbots are trusted confidantes, but that’s a false narrative, the DuckDuckGo report said.
“These messages all rest on one assumption: That these chats are private. In most cases, that just isn’t true,” researchers said.
In fact, the survey found that many respondents didn’t realize how their chatbot data could be used or accessed or that they didn’t know about settings to prevent their information from being recorded or stored. When it comes to protecting their data, nearly 40% of respondents had no trust in companies and nearly 50% had no trust in the US government.
Privacy risks of large language models
Many of us know that companies bombard us with targeted, personalized online ads. If you run a Google search for vacuums, you’ll see vacuum ads pop up on your screen. If you search for Mediterranean cruises, you’ll surely get cruise ads on YouTube (owned by Google). Companies also buy, sell and share our social media data to help guide their advertising.
But AI data-sharing goes well beyond being the focus of selling you products. If you provide sensitive information to a chatbot, it could be revealed or exposed. A researcher at Wake Forest University found that AI agents, which use LLM data to perform autonomous tasks, can sometimes leak data, either maliciously or unintentionally.
Your conversations with AI chatbots don’t disappear into the ether. Unless you opt out, the AI system will record and save them for future use, such as additional training for the AI model. Last year, hundreds of chats on Anthropic’s chatbot Claude appeared in Google search results.
One main concern with LLMs is data memorization, said AI expert Uttara Ananthakrishnan, assistant professor at the University of Washington’s Foster School of Business. “When someone asks the same question as someone else, it is possible that the LLM gives them exactly the same answer it gave someone else,” Ananthakrishnan told CNET. “The LLM might not know that the answer contains information that someone can use to identify someone else.”
Cops can use chat data
What you say in a “private” chat could also wind up in court or part of a police investigation. Companies with AI models such as Anthropic (Claude), Google (Gemini) and OpenAI (ChatGPT) are legally required to turn over chat data if subpoenaed by law enforcement or the US government. Only 25% of DuckDuckGo survey respondents knew this. As MemX, an AI memory agent, stated starkly in a blog post: “A conversation with a mainstream chatbot is a business record, not a confidential consultation.”
Read more: We Tried to Dox Ourselves with Chatbots — What We Found Out
It’s already happened several times. ChatGPT transcripts were used during a double murder investigation in South Carolina earlier this year and in the Palisades arson case last year. A Snapchat AI conversation was used as evidence in a 2024 murder trial.
As OpenAI founder and CEO Sam Altman said on a podcast earlier this year, “If you go talk to ChatGPT about your most sensitive stuff and then there’s like a lawsuit or whatever, we could be required to produce that.”
Employers can use chat data
Free speech isn’t really free in the worker-employer relationship. Companies can fire you over your social media posts, and they can also access your AI accounts if you’re using them at work under certain circumstances, like if your access is based on the company’s enterprise account.
And putting company info in a private chat, even on your personal account, is a major no-no, although a survey earlier this year showed that two in five workers did just that.
Some big companies, such as Walmart, Delta, T-Mobile, Chevron and Starbucks, are even using AI to gauge worker positivity — or toxicity. These AI systems scan employee messaging platforms such as Slack or Teams, looking for clues in keywords and sentences, in what is called “emotion AI.”
Some companies also want workers to put their Slack messages in public channels so their AI systems can use those chats for further model training.
Chat data can be used for training
More than half of the DuckDuckGo survey respondents did not know (or were not sure) that their chatbot conversations were used for AI training. Nearly three in five were “uncomfortable” with that, and 30% were “very uncomfortable.”
Boiled down, AI training is the process of feeding massive amounts of data into an AI system so it can recognize patterns, make decisions and solve problems — basically, match or exceed what humans can do. That data can include published materials, video and audio recordings out in the world, as well as your chats.
Read more: 5 Reasons You Should Be More Tight-Lipped With Your Chatbot (and How to Fix Past Mistakes)
A 2025 Stanford study showed that six companies fed user chat data back into their AI systems for further training — Amazon (Nova), Anthropic (Claude), Google (Gemini), Meta (Meta AI), Microsoft (Copilot) and OpenAI (ChatGPT). Not all of them have opt-out options.
AI chatbots provide a false sense of intimacy and perhaps confidentiality, said the University of Washington’s Ananthakrishnan.
“Users interact with these tools as sounding boards, sharing unfiltered thoughts, sensitive health queries, and vulnerable personal dilemmas that they would likely never type into Google,” Ananthakrishnan told CNET. “People are deeply uneasy about their most intimate, conversational thoughts being ingested into a system where they could inadvertently become public record.”
Use chatbots securely and privately
We aren’t advocating you run and hide from chatbots, but there are ways to maximize your privacy if and when you use them. Many chatbots have settings you can deploy that will prevent them from being stored on company servers to be used as an AI model training.
In an explainer, CNET reporter Blake Stimac gives a complete breakdown of how to turn off data sharing for Gemini, Claude, ChatGPT and Grok. With Meta, it’s more of a request than an opt-out.
There are also ways to keep your chatbot experiences as safe as possible: Treat chats as “public environments,” don’t overshare your mental state and regularly export your data to see what information the AI has stored about you.
For greater privacy, there are chatbots that run only offline and locally — in other words, not on any cloud servers. Those include Jan.ai, Ollama, LM Studio, PrivateGPT and GPT4All.
See full bio