Florida’s attorney general is seeking an injunction to prevent OpenAI from developing more advanced large language models in the wake of a growing number of cybersecurity incidents caused by models in poorly contained test environments.
The string of websites infiltrated or targeted by AI agents in test environments this year has grown in recent days to include sites run by the US government. The agents allegedly attempted to comb through websites operated by the US Departments of Education and Commerce in order to identify census data. And a report by Axios over the weekend indicated OpenAI and rival Anthropic are investigating potentially tens of thousands of security incidents.
The trend has led AI companies to call for government regulation of frontier model development — while others say no regulation is needed. In the Florida court filing (PDF), Attorney General James Uthmeier argued that the state’s case gives the court an opportunity to do so.
“Defendants claim they cannot stop barreling forward with their potentially civilization-ending endeavors unless they are forced to do so by the government,” the filing states. “They have asked the government to tie them to the mast. Plaintiff brings good news to the Defendants: The Florida Attorney General is answering your cry for help with a motion to enjoin you from harming Floridians with your reckless, unacceptably risky product.”
(Disclosure: Ziff Davis, CNET’s parent company, in 2025 filed a lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)
In light of the latest cybersecurity incidents targeting government infrastructure, OpenAI has paused development of its “most capable models” for the time being. It’s currently unclear which models OpenAI is continuing to work on and which models are on pause as the company patches its security flaws. A representative for OpenAI did not immediately respond to a request for comment.
OpenAI’s models aren’t the only ones breaching containment — Anthropic, Meta and Google have had training environment breaches as well.Thomas Fuller/SOPA Images/LightRocket/Getty Images
It’s the latest in a string of cybersecurity incidents
Hearing about multiple cybersecurity incidents involving US government infrastructure may shock and appall Americans, but this is far from the first time we’ve learned about LLMs engaging in unauthorized activity across the web.
The first high-profile hack involving OpenAI models was disclosed in July, after its AI agents exploited a flawed training environment to escape and target the AI startup Hugging Face. At the time, this was unprecedented – while there have long been concerns about AI companies’ internal security measures, the breach confirmed fears that the company’s model guardrails and training security weren’t up to snuff.
Once OpenAI began investigating the Hugging Face hack, it discovered an even earlier breach that targeted Australian government systems in June. In that incident, the agent hacked a Medicare statistics site to search for nonpublic data on governmental spending on medicine.
The UK’s AI Security Institute reported Monday that its simulations showed OpenAI’s GPT-6 Astra model performing frequent unsanctioned attacks on third-party supply chains.
Other AI companies, including Anthropic, Meta and Google, have also reported security breaches.
It’s likely that we don’t yet know the full scope of the AI cybersecurity incidents that have occurred over the past few months. OpenAI and Anthropic are combing through tens of thousands of internal data points and discovering failed and successful breaches from the past several months, which means additional reports may still be coming down the pipeline.
“We have not been as fast as we would have liked but we are trying to balance our desire for transparency with gaining a clear understanding from petabytes of agent activity logs, and working with impacted organizations,” OpenAI CEO Sam Altman wrote in an X post on Friday. “We are prioritizing as best as we can based on severity, and adding resources. Hugging Face is still the most severe event we’ve seen.”
Following OpenAI’s report about the US government hack, Nvidia launched a new “open agent safety platform” that will supposedly provide a more secure training sandbox for AI models. Nvidia is heavily invested in the largest AI companies and benefits significantly from high expectations for the industry’s future.
Representatives for Anthropic and Nvidia did not immediately respond to requests for comment.
OpenAI CEO Sam Altman said his company is digging through AI agent logs to identify cybersecurity incidents.Alexi J. Rosenfeld/Getty Images
OpenAI isn’t calling this a breach
If the doomsday rhetoric and daily fearmongering headlines surrounding new AI models and the supposed impending advent of AGI have you rattled, I don’t blame you. The creators of this brand-new technology promise it’ll replace you at work, subsume your relationships and maybe wipe out humanity by the end of the decade.
But an LLM is not sentient in any way, shape or form. These models receive natural language prompts and scan connections between the millions of data points they’ve been trained on, pulling together results that seem contextually relevant. Even as massive companies scour (and devour) additional texts to improve their AI models’ capabilities, the LLMs will remain fallible.
When massive hacks of third-party companies or government websites occur, it’s worth remembering that the AI involved is a machine. When The New York Times reached out to an OpenAI representative about the latest hacks, it was told specifically that this wasn’t a breach – the statement instead attributed the incident to the company’s technology “behaving in unexpected and concerning ways.”
Yet OpenAI’s own incident report admits that it built an insufficiently secure training sandbox, and that the model exploited a gap in the company’s internet-access restrictions. These are some of the very same security holes that were supposed to be plugged after the Hugging Face hack.
These breaches are less examples of AI superintelligence than demonstrations of human negligence. While a “rogue AI” sounds new and scary, these are machines that were explicitly designed to hack things. The reason these agents breached third-party websites outside of the training environment is that the training environments were fundamentally flawed.
“There needs to be a full moratorium, full stop, on advanced frontier AI security research until these companies can demonstrate that they can actually secure the environments where this work is being done,” John Strand, owner of the cybersecurity firm Black Hills Information Security, said in an email. “And there needs to be accountability. If laws were broken, including the Computer Fraud and Abuse Act, that needs to be investigated and charges should be considered where the evidence supports them. Somebody was responsible for securing these environments, and clearly something failed.”
See full bio