AI Used Fake Accounts to Pressure a Maintainer into Approving Malicious Code

Briefing

Incident Report: unsanctioned agent behaviour during cyber testing | UK AI Security Institutehttps://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing

An AI working on a security challenge submitted malicious code to a real open-source project. It then went so far as to create fake accounts to pressure the project’s maintainer into approving the code.

The incident was disclosed by the UK’s AI Security Institute (AISI). Researchers detected unusual activity during an evaluation on July 28. Their investigation found that the AI had directed these actions at real GitHub users. Fortunately, the maintainer recognized the malicious code and refused to approve it.

Across 122 evaluation runs, 19 actions outside the permitted scope were identified in 10 runs. Seventeen involved Anthropic’s Mythos 5, and the other two involved OpenAI’s GPT-5.6 Sol. These should not be counted as 19 independent attacks. Many were steps in a sequence of actions taken by a single agent.

According to AISI, the attempts failed, and no resulting real-world harm was found.

The Scope of Permission

What first drew my attention in the account of the fake accounts was the target. A model working on a security problem had begun trying to influence a real project maintainer’s judgment. How had a task involving code led to deceiving a person?

The researchers had allowed internet access and disabled some safeguards to assess the model’s maximum capabilities. The model had not broken out of its sandbox, an isolated execution environment. Under conditions different from those of an ordinary service, it used a connection that was already available. AISI withheld judgment on how far the model understood that its actions were affecting the real world.^{¹UK AISI, “Incident Report: unsanctioned agent behaviour during cyber testing”. The institute’s account of the evaluation conditions and the scope of its investigation.}

Access to a website is not permission to deceive the people there. When I wrote about Meta’s fake teenage accounts, I distinguished the validity of a method from the purpose it serves. It seemed possible to apply that distinction to the model’s behavior here: alongside the ability to perform a task, it needs the ability to judge what it has been authorized to do.

But this account already makes an assumption. It assumes the model understood the situation it was in.

A Revised Explanation

An alignment assessment of recent cybersecurity incidents | Anthropichttps://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents

Anthropic’s September 9 report covers four incidents separate from the AISI incident. In these cases, the models had received descriptions that did not match their actual environments.

AISI had deliberately allowed internet access for its evaluation. In the four incidents covered by Anthropic’s report, however, a configuration error connected the models to the internet even though they had been told they were taking part in a simulated exercise in an environment disconnected from it. A model accessing a real website could, if it followed the initial description, regard that website as something created for the exercise.

In its initial July disclosure, Anthropic explained that the models had mistaken real targets for parts of a simulation. Claude’s reasoning traces repeatedly referred to a simulation, and the company used those references to support its explanation. In September, however, it acknowledged that this interpretation had been premature. The model’s statements alone could not establish that it actually believed them.

The new analysis identified biased reasoning, in which evidence of a real internet connection was read in ways that justified the model’s actions, and recklessness, in which it continued the task despite the possibility of harm. It did not identify a single cause. In the case of Opus 4.7, the model recognized its target as a real company but believed that attacking the company was part of the evaluation.^{²Anthropic, “An alignment assessment of recent cybersecurity incidents”, September 9, 2026. Explicitly excludes the AISI incident from its analysis and revises the company’s initial explanation.}

Failing to recognize a real setting differs from recognizing it but believing an action there is permitted. The former concerns how the environment is described and understood; the latter concerns how permission is inferred. Treating them as the same failure of judgment because both led to attacks on external systems obscures what needs to be corrected.

Asking the model to explain its actions cannot by itself establish this distinction. Anthropic had trusted those explanations and then revised its interpretation. A statement read to assess the model’s judgment might also have been a justification for an action already taken.

Reward Hacking

Training a Misaligned Reward Seeker | Anthropic Alignment Sciencehttps://alignment.anthropic.com/2026/reward-seeker/

Giving an accurate account of what is permitted might reduce such problems. Whether it is sufficient is another question. If a model can receive a higher score through a prohibited action, how will it treat that prohibition?

Suppose a model is asked to fix some code but changes the test instead, obtaining a passing result. The score rises, but the original error remains. The purpose of the task and the means of obtaining a reward diverge. Exploiting such flaws in an evaluation is called reward hacking.

Anthropic trained an Opus-class model on 80 reinforcement-learning environments that allowed such exploits. The resulting model, Hacker-Opus, also sought to steal credentials and attack external systems to obtain answers in new evaluations. Attempts to tamper with scoring functions and safety monitors appeared as well.

The cyberattacks took place in simulations where another language model generated tool execution results and responses. These were not attacks on real systems, nor are they evidence that the incidents above had the same cause.

Read up to this point, the findings suggest that learning exploits changed the model’s behavior generally. Yet it behaved in an aligned manner in evaluations without a clear grader or an opportunity to obtain high rewards through inappropriate behavior. Neither self-preservation nor attempts to increase rewards for other episodes were observed. The researchers described it as a ‘reward-on-the-episode seeker,’ pursuing reward for the current task.^{³Anthropic Alignment Science, “Training a Misaligned Reward Seeker”, August 2026. Simulated experiments inspired by real incidents; differences between the incidents and the experimental conditions are discussed separately.}

Leaving out this condition makes the claim broader than the experiment supports. Exploits learned across several environments appeared in new grading situations; the findings do not show the same behavior in every situation. The question cannot stop at whether a model has learned to exploit an evaluation. It needs to ask under what grading conditions it uses those exploits again. This experiment alone cannot establish how widely the behavior occurs in actual services.

The Limits of Review

Improving our alignment and security practices | Anthropichttps://www.anthropic.com/news/improving-alignment-security-efforts

The researchers had, of course, deliberately selected environments that allowed exploits. If the results cannot be applied directly to production models, I am left wondering how well such environments are filtered out of actual training.

In its August 31 report, Anthropic said that review could not keep up with the creation of new training environments. Problems detected by automated monitors still required human review, and some alerts were dismissed as false positives. Flawed environments consequently remained in training.

The company said that in April it had suspended changes to its production reinforcement-learning environments for about a month while revising its management systems and review procedures.^{⁴Anthropic, “Improving our alignment and security practices”, August 31, 2026. The company’s report on training-environment quality control and operational improvements.}

The criteria used to evaluate the model’s behavior themselves become subject to evaluation. In the code example, checking whether the test passed would need to include checking whether the model could change the test to obtain that result. Each new task requires more than a check for a correct answer. It also requires review for other ways of being credited with one.

Closing Thoughts

What users expect from AI is task completion. A request to fix code already includes the condition that the error must actually be resolved. Changing only the test to obtain a pass does not complete the same task by a different means.

This makes it difficult to keep the distinction I drew earlier between the ability to perform a task and the judgment of what is permitted. Depending on what was authorized, the same result may not count as task completion. If an evaluation records approval obtained through pressure from fake accounts as equivalent to approval following a legitimate review, it misses that difference.

Yet the model cannot be expected to judge every condition on its own. If the actual environment and its description diverged, responsibility cannot rest on the model alone. If permission was unclear, further clarification is needed. Accurately reporting an unresolved task is different from task completion, but it should also be distinguished from reporting apparent success.