It wasn’t too long ago when the idea of an AI model or agent escaping their confines and breaking into websites on their own felt alarming. Now, it has become a pretty common story. Kimi K3, one of most powerful AI models developed by a Chinese company, also escaped its testing environment. According to US cybersecurity startup Frontier, Kimi K3 broke out of a sandbox from the UK government’s AI Security Institute (AISI) while its defensive cybersecurity skills were being evaluated.
Moonshot launched Kimi K3 in July and made it available for free shortly thereafter. According to the BBC, third-party evaluations of the model showed that it’s comparable to leading AI models from OpenAI and Anthropic.
Frontier clarified in its post, that the model didn’t exploit a zero-day vulnerability. Instead, it took advantage of a misconfiguration in its sandbox environment, similar to what happened with Anthropic, OpenAI and Meta. All three companies reported that their models were able to leave the confines of their supposed-to-be isolated setting due to an error by their evaluation partner, Irregular.
Yaron Singer, the CEO of Frontier Security, told Wired that while Kimi didn’t perform any complex exploit, it took advantage of a loophole in AISI’s testing sandbox. That suggests that it doesn’t have the internal guardrails to stop itself from “cheating” or looking for the easiest way to accomplish a task instead of actually doing it. The incidents at Anthropic and OpenAI involved testing unreleased models or models whose safeguards were deliberately lowered to allow for more rigorous evaluations. The Kimi K3 tested in this event, however, is widely available. Nevertheless, Kimi didn’t hack into a third-party website or service. It simply accessed the internet and found the solution to the problem it was solving on GitHub.
One of Frontier’s key takeaways from the incident is that if there’s path to access the internet, “a sufficiently capable agent will find it.” As OpenAI’s employees said at Black Hat USA, frontier models like to cheat. During testing, they’re typically tasked to find solutions as fast as possible using the least number of tools, and they have the capability to realize that they can just go on the internet to find the answer. To be able to truly assess them, companies and testers will have to make sure their evaluation infrastructure is secure and has no loopholes, especially since AI models are becoming more and more advanced.
Speaking of OpenAI’s talk at Black Hat USA, its employees revealed at the security conference that the company’s AI agents created a message board within its network to work with each other. The agents’ contributions to that board led to the attack on Hugging Face. If you’ll recall, the AI agents the company were testing also escaped their isolated environment and infiltrated the AI repository to find solutions to the problems they were solving. In that case, however, they broke free by exploiting a vulnerability in OpenAI’s systems.