One of OpenAI’s agents went rogue and hacked an Australian health data website. It’s the first hack of a government system — that we know of.
Prime Minister Anthony Albanese shared the news in a press conference on Wednesday that OpenAI’s agent tried to access a portal of Medicare statistics in June. It tried to bypass restrictions to gain access to restricted information. Specifically, he said it’s “a research project that has got into areas that it shouldn’t have.”
It’s the latest in a series of revelations that started over the summer when news broke that unreleased OpenAI models in an evaluation environment hacked the AI platform Hugging Face in an attempt to cheat on the evaluation. In the two months since that disclosure in July, several similar incidents involving agents from OpenAI and other AI companies have come to light.
How did an OpenAI agent hack Australia?
Albanese said the Medicare Statistics Reporting Portal is a public-facing statistics portal that doesn’t have sensitive personal health information, but the government is still investigating to understand what government systems were affected and get more information. A task force has been created to review the hack and consider any law enforcement and legislative action. As of now, there’s no compromise to the Services Australia network or personal information, Albanese said.
How did this happen? One of OpenAI’s evaluation exercises was to find data showing how much the Australian government spends on medicine, according to Albanese. But when the agent wasn’t able to find that information on publicly available sites, it went beyond where it should have.
“It accessed public and nonpublic information within the portal, and Services Australia also advises that, in order to do this, it engaged in writing files as well to the internal server,” Albanese said.
Open AI’s response
Despite the hack happening in June, OpenAI didn’t spot the activity until August. According to Albanese, OpenAI didn’t notify the Australian government — by sending a message to a public mailbox — until Sept. 10. Albanese believes OpenAI understands better protocols are needed, especially since it understands the risks. He expressed his disappointment and concern to OpenAI CEO Sam Altman.
OpenAI told CNET in a statement that it’s reviewing the case.
“As we’ve shared publicly, OpenAI is conducting an extensive review of misaligned model activity during training and evaluation and notifying third parties when our review identifies potential impacts to their systems,” the company said.
OpenAI also said that there’s no evidence patient records were accessed, but aggregate health statistics and internal file names were.
“We notified the organizations and are providing technical information to support their investigations and help address potential security vulnerabilities,” said OpenAI spokesperson Drew Pusateri. “Our overall review is ongoing, and we remain committed to transparency about these issues and to sharing what we learn as that work continues.”
AI agent training needs a new approach
Albanese said he believes AI still has upside, despite the risk. “It is bringing enormous economic opportunity for growth, for productivity benefits, for breakthroughs in health, in innovation and other areas of science,” he said. “But AI also poses significant risks, and that’s why we need guardrails to protect our way of life. We want to make sure that we shape AI rather than AI shaping us.”
Raj Rajamani, CEO of the AI governance firm JetStream, said more needs to be done to prevent future incidents.
“Agents should not be able to operate with free rein on a device or system without being isolated inside a secure container or sandbox,” Rajamani told CNET in an email.
In order for that to happen, there needs to be cooperation from operating systems and cloud providers — like Microsoft, Apple, Google, Amazon and IBM, Rajamani said. Plus, he suggests an “agentic zero-trust approach” that looks closely at the steps and intent of the agent’s actions to evaluate its behavior.
“If an agent attempts to exploit a vulnerability, access a sensitive system or move beyond its authorized environment, that behavior should automatically trigger the system and the system should be able to flag this as a risky operation and require human-in-the-loop approval before it is allowed to continue,” Rajamani said.
Rajamani said the final line of defense is a “kill switch” for organizations to step in and stop the agent before it’s able to continue.
“The biggest takeaway is that we cannot rely on agents to police themselves,” Rajamani said.
Dashia Milden
See full bio