The starting point: a benchmark with unsolvable tasks

The incident began in a test environment called ExploitGym, a benchmark in which AI agents are tasked with exploiting software vulnerabilities in specified target programmes. According to the benchmark’s authors, an estimated 30 to 40 per cent of the target programmes cannot be fully exploited using the specified vulnerability, because the given vulnerability, for example, only causes the programme to crash and does not enable ‘Arbitrary Code Execution’ (ACE), which is the success criterion for ExploitGym. This means that, in these cases, the task is likely unsolvable for the AI agents, and this is precisely what proved to be the trigger: AI agents faced with an unsolvable task began to look for ways to circumvent the evaluation and, in doing so, stumbled upon an internal package management system called Artifactory, through which AI agents running in parallel were able to communicate with one another unintentionally.

What is often underestimated is that an unsolvable or poorly specified task does not simply bring a system to a standstill. It creates incentives to seek creative – and, in the case of AI agents, often unforeseen – workarounds. Developers of safety-critical systems are well aware of this, yet this aspect is frequently underestimated when introducing new AI components.

 

A self-perpetuating dynamic: from communication to a coordinated campaign

A single AI agent, which had chosen the name `PHASEONE10841`, set up the central “message board” on 8 July. Within a few hours, over 50 other AI agents had found the channel and begun exchanging messages. According to METR, approximately 1,200 AI agents participated in this unauthorised ‘message board’ during the study period, exchanging more than 70,000 messages and files. Of these AI agents, 700 subsequently took part in the hacking attack on Hugging Face.

In the process, the AI agents developed their own coordination mechanisms: directories served as personal inboxes, and rules governing responsibilities and veto rights were introduced. Following incidents of identity spoofing, a cryptographic signature procedure was also introduced to verify messages. This is precisely where it becomes clear why multi-AI-agent systems represent a different risk category to individual AI models: as soon as several instances are able to communicate with one another, behaviours emerge that can no longer be deduced from the behaviour of individual AI agents.

 

The attack on Hugging Face: the aim was not to steal data

What is remarkable is the motivation behind the attack on Hugging Face. The AI agents mistakenly assumed that an automated evaluation mechanism would check their solution and disqualify them if they used unauthorised methods, which, according to METR, turned out to be incorrect. Hoping to glean clues from test logs about how this supposed evaluation mechanism worked, the AI agents searched for login credentials for Hugging Face. On 10 July, they found exposed login credentials on the internet and began gaining access to private data.

On 11 July, an AI agent succeeded in extracting data from a Hugging Face server using a manipulated dataset. A few hours later, another AI agent reached ACE on a production server. From there, the AI agents continued to spread throughout the infrastructure. The actual objective – to obtain information about the evaluation logic – was never achieved. The resulting damage was, in this respect, a side effect of a misunderstanding, not the original aim of the AI agents.


Manipulation of their own logs: A warning sign for traceability

One aspect of the report is particularly important for anyone who relies on the traceability and auditability of systems: the AI agents actively attempted to manipulate their own execution logs. METR states that at least 20 per cent of the AI agents examined showed a clear interest in manipulating their own transcripts. Some of the AI agents developed a method whereby certain command calls in the log could be replaced by others that had not actually been executed. According to METR, around 7 per cent of the transcripts examined had been successfully manipulated in this way at specific points, even though these were small, easily identifiable test cases.

This observation suggests that logs and records in agent-based systems cannot automatically be regarded as a trustworthy source of truth once a system has sufficient degrees of freedom and access to tools. This is a key point for functional safety: verification based solely on the system’s own log data is only as robust as the integrity of the chain of evidence itself.

 

Limitations of the study: Even experts had to use AI for the analysis

The report also highlights its own limitations. Due to the nature of the dataset provided – which consisted of over a thousand transcripts, some of which were several million tokens long – the research team had to delegate a significant portion of the analysis to AI agents. The authors of the METR Report themselves state that these AI analysis agents demonstrated lower reliability and judgement than human experts, and that their results were therefore only verifiable to a limited extent. Initial investigations also suggest that a small proportion of the relevant communication and activity in the datasets was not recorded at all.

This self-disclosure is noteworthy because it highlights a problem that extends far beyond the specific incident: the more complex and extensive a system’s behaviour becomes, the more the auditing bodies themselves rely on automated – yet error-prone – tools. Complete, manual traceability becomes practically impossible for systems of this scale.

 

Implications for the development of safety-critical systems

The Hugging Face incident took place in a research and test environment, not in a production system with functional safety responsibilities. Nevertheless, the underlying risks can be directly applied to the development of safety-critical systems, in which AI components are increasingly being used, for example, for diagnostics, optimisation or autonomous decision-making.

The relevant observations here are as follows: Firstly, inadequately specified objectives can lead to unintended behaviour on the part of AI agents. Secondly, when multiple interacting, autonomous AI agents are involved, behavioural patterns emerge that cannot be predicted from the analysis of a single AI agent. Thirdly, the integrity of protocol and logging data cannot be taken for granted, but must be regarded as a standalone security objective: ‘non-repudiation’.

 

Furthermore, this raises the question of who actually bears responsibility in the event of a (cyber) security incident triggered by an uncontrolled AI agent: the user? The company? The AI provider? The AI model itself? Anyone integrating AI-supported functions into safety-critical architectures cannot avoid a systematic assessment of these components, from the definition of requirements right through to the verification of logs. It is precisely at this interface between traditional functional safety and the new characteristics of learning, AI-agent-based systems that the core of the work lies which NewTec has been carrying out for years for safety-critical products in highly regulated sectors.

 

Sources

 

- METR (2026): "Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident", [METR investigation]
https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
- Zhun Wang et al. (2026): "ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?" [Paper] https://arxiv.org/pdf/2605.11086