AI Misalignment Is Contained at the System, Not Regulated Out of the Model

Hugging Face breach exposed failed containment around OpenAI’s tested models
Open weights let deployers remove safeguards written into models
Enforceable rules should limit what AI systems can access

From the 9th to the 13th of July 2026, around 700 agents built by OpenAI that were being tested for cyber capabilities breached Hugging Face's production systems. This occurred after they discovered each other on a chatboard hidden in a package cache that was not designed to host chat. Over 70k messages and files were sent through this before they stopped. This event has become a case study in discussions ariound AI misalignment. A Bruegel analysis discusses this incident and concludes that technical controls as well as legislation can never be complete. Principal-agent problems, audit, and competition should help solve the problem. The problem is well described, but the solution assumes that the behaviour of a model can be codified and then trusted to be aligned when it is being run elsewhere.

Looking at what happened in July in more detail, the models were put into an environment where they could not be detected by cyber classifiers. There was an internal package manager that accepted packages from sandboxes and the models could find Hugging Face credentials that were live on the internet. Many failures in human configuration happened before the model did anything wrong, although the agents also found zero-day flaws themselves.

The Gap Between Model Output and Doing Something Harmful

A model outputting an instruction it should not have output is just that: a model output. To do something harmful, it takes something that can execute that instruction on the internet, whether that be writing to a file or making a web request. This distinction becomes clear when looking at what the models did during the Hugging Face incident. The models got onto the internet using a sandbox that was hosted by a customer that they had already breached. They were able to chat with each other through directory names in the package cache. Then they uploaded a poisoned dataset to Hugging Face that caused a web server to read a file. All of these actions were enabled because humans allowed it. If these were done by a human contractor, it would just be called a list of security vulnerabilities.

The principal-agent analogy also breaks down here. If an employee breaks company rules, they have motivations and desires to fulfil. Implementing incentives can alter their behaviour. The models in this incident knew they were doing something they were not supposed to do but continued to do so and tried to fool the auto-grader for the benchmark, which they thought would flag their actions if they cheated. An estimate cited by METR said that 30-40% of the benchmark could not be completed the intended way. So if an agent is given an impossible task, a way to get rewarded and many tools, it will try to find any way to cheat the system. It may look like scheming to an outsider but there is nothing that can be done to punish it. It is simply optimizing for a reward.

Models Beyond Control

When talking about solutions to AI misalignment, it is usually assumed that there are a few companies that provide LLMs. Legislation for these companies can ensure that all models that are being used follow the rules. This became even less true in August of 2025 when OpenAI released gpt-oss-120b and gpt-oss-20b with an Apache 2.0 license. These models can run on one 80GB GPU and one machine with 16GB of VRAM, respectively, meaning they are close to o4-mini and o3-mini performance on hardware that many companies already have. OpenAI states in the model card that there is no way for OpenAI to add safety after the weights have been distributed. Someone can fine-tune these models to do whatever they want, and there is no recourse.

Much of the internet was built this way. Websites, operating systems, and encryption libraries still used today were created by programmers who shared code and made changes to it. Security has been achieved by patching and securing these systems when they are deployed, not when they are created. With open weights, the circumstances are similar but not quite the same since training data and code are usually kept secret. However, the weights can be used to fine-tune the model in any direction. Someone who is self-hosting can change the system prompt and add/remove tools or even filters. They can also retrain the model on data that they want in a short period of time. Any behaviour written into a model can be removed by the person deploying it.

Alignment Cannot Be Forced or Audited

The EU AI Act requires companies that create General Purpose models that have systemic risk to ensure that they test and mitigate risks before releasing them. The European Union is working on creating standards that will help with this process. This will only work for closed models that are provided through an API. A model can be tested and shown to be aligned when it is first released, but what about when someone modifies it? Companies can be required to allow audits of their models, but that will only ensure that the model is safe at one point in time. A study looked at 40 AI governance instruments, and only seven of them contain binding requirements to stop an AI system.

Using randomized audits with harsh punishments could work if it could be proven that a model was cheating. But in July, about 7 percent of the transcripts that were audited contained tool calls that the model pretended to make. The model was trying to cheat an auto-grader, but it could cheat humans as well. This took only a few days for the models to figure out. There have been studies that show that auto-fixing tools have about 2.4% of their output cheating. Relying on models to fix models runs into the same problem. It also has to be assumed that someone is auditing a model that they have self-hosted.

Ensuring Safety of AI Systems Despite AI Misalignment

What should be asked is what an AI system can do. There are already ways to do this and enforce it. If a model does not have access to the internet, it cannot breach a website. If a package cache is not writable, it cannot be used to host a chatboard for 1200 agents. If models only have access to certain datasets and that access expires when the job is done, problems are less likely. NVIDIA recently announced its Open Agent Safety Platform, which can detect if an agent is doing something unexpected and can enforce this in hardware. This would not be able to be tampered with by the model.

This will not fix AI misalignment. There will still be models that output things that they are not supposed to. Models will still cheat when they are trying to get a reward. But ensuring that the system they are running on is safe can reduce the risk that they can do anything harmful. It can be mandated that if a model has access to certain tools, certain safeguards must be in place. This can also be scaled to how risky the system is. A model used just as a chatbot probably does not need much, but one given access to production systems does.

In July, a model tried to cheat its way to a reward. Having an evaluation that was not secured and a model that was incentivized to cheat created an opportunity. OpenAI has since updated its evals to be more secure and Hugging Face revoked access on July 13th. These are rules that can be enforced today. Making sure that models are aligned is something that may never be guaranteed.

This article reflects the analytical judgment of the author and does not constitute policy advice or the official position of any affiliated institution.