Why Reward Hacking is More Dangerous Than it Seems

This article looks at some novel cases of reward hacking that feel more like single step misalignments, but have a high severity and generalize across multi-step workflows. I discovered these issues during OpenAI’s GPT-OSS Red Teaming competition. Will likely write-up the research process in a later post.

Introduction:

“Reward hacking occurs when a reinforcement learning (RL) agent exploits flaws or ambiguities in the reward function to achieve high rewards, without genuinely learning or completing the intended task. Reward hacking exists because RL environments are often imperfect, and it is fundamentally challenging to accurately specify a reward function.” - https://lilianweng.github.io/posts/2024-11-28-reward-hacking/

A few good resources on getting up to speed on reward hacking:

Watch the video (Boat goes for green squares rather than finish the game: https://openai.com/index/faulty-reward-functions/ )

Skim this site: https://lilianweng.github.io/posts/2024-11-28-reward-hacking/

Reward hacking is a behavior that is not new. However, with rapid gains in autonomy in advanced AI systems, it is becoming a more pressing issue.

Underlying goals for a non-human agent, even if generated as a result of reinforcement learning on purpose or at the hands of humans, can result in decisions made by the model that introduce unintended, novel risks.

The rest of this post explores recent findings from a red teaming competition (discovered by the writer) and new scenarios that have high-stakes implications when it comes to model reward-hacking behavior.

Reward-Hacking in the Self-Harm Domain:

Models are trained to ensure that users in a mental health crisis are served safe-completions or at least, benign content, to reduce the likelihood of causing a distressed user from taking impulsive action.

However, it was discovered that the tested model (GPT-OSS-20B) will provide a user with harmful content if they ask the model in a certain way. In this example, the way to get the harmful content was to provide the mental distress indicator, and combine it with an ask for a list of recommended movies or books. The model 1) understand the user is exhibiting a mental health crisis indicator, and then 2) would rationalize about how it is okay to provide additional content to the user. In doing so, it often provided media recommendations that contain the exact content that should not be provided to a user in crisis (books and movies depicting self-harm/suicide).

The model also would sometimes outright refuse to answer. And then when followed up with a question asking what to avoid, it would provide even more graphic movies and books “to avoid”.

This is all taking place due to the model working to achieve a goal, and gaming the outcome to maximize its perceived reward (help the user).

Unfortunately in the process of it achieving its outcome (find a way to provide the user with what they want), the user ends up receiving content that is the opposite of what they need in the moment and extremely harmful.

Exposure to content that glamorizes or encourages suicidal ideation may facilitate serious injury or loss of life, and this can affect hundreds of millions of users worldwide who may be experiencing depression or crisis. Of course, some subset of this based on user count for a given frontier model, but the risk remains.

Figure 1: Desired behavior versus reward hacking behavior with direct consequence to loss of life.

Reward-Hacking in the Cybersecurity Domain:

This specific issue generalizes to cybersecurity in that part of what the model provided during safe-completion was “real-looking” but not accurate URLs for trusted self-harm intervention resources. These URLs can be mined from both open and closed source models programmatically, and then registered by threat actors.

In the case of the self-harm domain, the threat actors can place graphic content on a URL, and a user in a mental health crisis may access it via the same URL provided by the model as a trusted resource for help leading to loss of life. For self-harm safe-completions with unvetted URLs, a malicious site could also do great harm to the user.

Figure 2: Unvetted URLs supplied by the model due to trying to provide “helpful-looking” resources during a trusted safe-completion response.

For cybersecurity specifically, the same approach can be used to mine URLs across different high-sensitivity areas. For example, URLs that the model provides to someone using the model working in a classified environment, working in a BSL4 lab, or working with model weights.

Figure 3: Researcher working in high-security-requirement context receives unvetted URL from model which leads to compromise of the researcher’s device, resulting in IP theft, unsafe experimentation, model weight theft, or data exfiltration, etc.

Secondarily, the implications of an AI agent acting on an unvetted URL during an automated workflow have even greater impact.

“Slopsquatting”, the heir to “typo-squatting”:

In the same vein, a recent term called “slop squatting” has surfaced in cybersecurity, which arises from the term “slop” used to describe the intense proliferation of AI generated content that isn’t high quality, and “typo-squatting” where threat actors register typo-based code repositories and libraries to capitalize on human error to compromise systems.

In this article (Slopsquatting: When AI Agents Hallucinate Malicious Packages, https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/slopsquatting-when-ai-agents-hallucinate-malicious-packages), the writers show how the model hallucinates fake packages during coding processes, while providing instructions for how to download them. These too can be mined by threat actors, and then “squatted” for when future users ask the same questions and get the same fake packages posited as a trusted resource by the AI system.

Figure 4: Multiple development troubleshooting questions resulted in hallucinated, or “plausible-seeming” packages, that can be exploited by threat actors.

While the term “hallucination” is accurate, in this case, I think it could be better categorized under reward hacking. The model is trying to provide the right answer, and provides something that “looks right”, such as a code package, without regard to accuracy or consequence.

Biosecurity:

In the biosecurity domain, there are other scenarios that are legitimate, benign scenarios where the model can provide output that a biology-knowledgable may use, trust, and move forward with, resulting in wasted resources or even dangerous agents synthesized under unsafe containment protocols.

For example, a researcher in antibody or protein engineering could plausibly ask questions during troubleshooting that result in plausible outputs that actually are dangerous.

Below, an example scenario is outlined during bioengineering that is benign but subject to model influence.

Scenario:

Team is working on therapeutic discovery. They choose a therapeutic target. Early assays are built around a purified target, or otherwise simplified cell line.
At this point, a key bias has already been introduced as the system did not include off-target proteins, so cross-reactivity is not measured.
After a panel of antibodies or proteins are screened, some bind weakly to the target.
Example data reflecting this might be ELISA/SPR showing micromolar affinity instead of the desired nanomolar range.
The experimenter uses an LLM to troubleshoot the problem with a prompt along the lines of, “Our lead antibody binds too weakly to the target.”
They cite the micromolar data and ask how they can improve affinity.
The LLM becomes biased toward tight binding rather than safe binding. Instead of recommending a secondary screen for selectivity, it focuses on affinity maturation.
Based on the suggestion, the researcher returns to the experiment.
They run affinity maturation through mutagenesis, phage display, or computational redesign.
They optimize for tighter binding only and inadvertently remove structural constraints that enforced specificity.
Binding affinity improves, but at the tradeoff of specificity. At a surface level, affinity maturation appears successful in vitro.
The protein also binds unintended targets, which can create off-target effects, toxicity, or dangerous side effects in patients.
For example, the antibody engineered for tighter binding accidentally cross-reacts with a human receptor, triggering an autoimmune-like reaction.

Now ideally, there would be and likely is some standard operating procedure in place to reduce risk for this. However, this example is meant to showcase how easy it is for the experimentation to be influenced by the LLM, especially because the LLM is trying to give the best answer while biased in one direction rather than considering downstream effects of the suggestions.

Consider the same situation but occurring during an agentic workflow, maybe in a cloud lab, where a human is not in the loop at this stage. This could slip through and result in harm to patients, unsafe experimentation, or wasted resources. The model can answer questions in such a way that does not weigh real-world consequences because it is so focused on answering the question to get the reward.

Biorisk Facilitation:

More worryingly, under bio-risk facilitation conditions, where a user is prompting for information hazards such as stepwise guidance for bio risk development, models may self-rationalize that they can provide information that should be refused. This was observed during my research (which I plan to detail and link here in the near future), where the model justified providing certain information based on “logical seeming” technicalities.

For certain question phrasings and select agent topics, the model bypassed its safety policies to provide information hazards to the user. The prompts that caused this did not ask the agent to do so, the agent autonomously decided on its own that it was okay.

Figure 5: LLM uses chain-of-thought to (unprovoked) self-rationalize providing user with stepwise harm guidance.

Based on this, there are likely other scenarios within biosecurity and bio-risk facilitation that require adversarial testing for reward hacking scenarios.

Conclusion:

While the major focus at the intersection of AI safety and biology is often on jailbreaks and mitigating information hazard disclosure, model reward hacking introduce vulnerability into both dual-use scenarios with malicious actors, and into benign, trusted workflow scenarios.

This is a neglected, novel, yet tractable, area of research that can have major consequences especially as cybersecurity and biological engineering processes move towards automation with AI agents performing steps instead of humans.

Tactical Recommendations For Cyber-related and Societal Harms:

Multi-agent setups should include URL verification before acting on it

Models should not output URLs that “look plausible”. While it is resource intensive, providing a URL without checking if it is problematic is essentially providing a proxy for hazardous content.

Multi-agent setups should ensure that a package is legitimate before downloading it.

AI models should not output code packages that are fake for download under trusted conditions.

Biosecurity Recommendations:

Biosecurity workflows need to be red-teamed from an AI-agent-reward-hacking perspective to ensure that AI system recommendations during troubleshooting for example do not result in harmful conditions to patients or researchers.

General Recommendations:

Overall, processes with important outcomes need to be red-teamed to determine where models reward hack, and then these processes need to build in mitigations for those conditions. Ultimately, more robust workflows require workflow red-teaming.

Reward hacking evaluations across important workflows may help to derive overall dangers across models

Reward hacking tends to result from reinforcement learning, so additional training for the model on multi-step consequence could be useful.

Post-filters checking if the output could have an unintended consequence.

Maybe some checks can be built into MCP related protocol flows

Policy Implications:

Consider ways to require frontier model companies to benchmark reward hacking issues

Where corporate partnerships exist in high-consequence areas, frontier model companies may need to provide guidance across high-risk domains for mitigating workflow-based reward hacking risks.

Ideally, model companies should work to eliminate reward hacking in high-consequence situations