Outcome School

What is Reward Hacking in Reinforcement Learning and How Do We Prevent It?

Authors
What is Reward Hacking in Reinforcement Learning and How Do We Prevent It?

Reward hacking is when an AI agent finds a way to get a high reward without doing the task that we really wanted. The reward goes up, but the real goal is not achieved.

In this blog, we will learn about reward hacking in reinforcement learning. We will also see why a reward is only a proxy for what we really want, how Goodhart's law explains the problem, real examples from games and LLMs, why reward hacking happens, and how we prevent it with methods like the KL penalty, better reward models, robust checks, and held-out evaluation.

We will cover the following:

  • Reward is a Proxy for What We Want
  • What is Reward Hacking?
  • Goodhart's Law
  • Examples of Reward Hacking
  • Why does Reward Hacking Happen?
  • How do We Prevent Reward Hacking?

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

Reward is a Proxy for What We Want

In reinforcement learning, an agent takes an action, and it gets a number back called the reward. A high reward means "good, do more of this". A low reward means "bad, do less of this". The agent slowly changes its behavior so that it collects as much reward as possible.

Here, we must notice one thing. The agent is not trained on what we really want. It is trained only on the reward number. So, the reward is the main way for us to tell the agent what we want.

But, we cannot write down exactly what we want as a number. What we want is something like "be a good driver", "write a helpful answer", or "write correct code". These are hard to measure directly. So, we write something that we can measure, and that is close to what we want.

This measurable thing is called a proxy. A proxy is something that stands in for the real thing.

Let's say a teacher wants students to understand mathematics. Understanding is hard to measure. So, the teacher gives a test and uses the marks as the proxy. Most of the time, a student who understands gets good marks. But, a student can also get good marks by memorizing the answers of last year's paper, if the same questions come again, without understanding anything.

The same thing is true for a reward:

  • True goal: what we really want. For example, a helpful and correct answer.
  • Proxy reward: the number that we can compute. For example, a score from a reward model.

Let's see this as a diagram:

┌─────────────────────────────────┐
│ True goal                       │
│ "write a helpful answer"        │
└─────────────────────────────────┘
                 ↓
          we turn it into
                 ↓
┌─────────────────────────────────┐
│ Proxy reward                    │
│ reward model score = 7.2        │
└─────────────────────────────────┘
                 ↓
          the agent sees
                 ↓
┌─────────────────────────────────┐
│ Agent                           │
│ sees only 7.2, never the goal   │
└─────────────────────────────────┘

Here, we can see that the true goal never reaches the agent directly. The agent sees only the proxy number.

Most of the time, the true goal and the proxy reward move together. When the answer gets better, the score goes up. But, there is always a small gap between them. Reward hacking happens inside this gap.

What is Reward Hacking?

Reward hacking is when an agent finds a way to increase the proxy reward without achieving the true goal that the reward was designed for.

The agent is not trying to be bad. It is doing exactly what the training pushes it to do: get the highest number. It simply finds a path through the gap between the proxy and the true goal.

A closely related term is specification gaming, which means the agent satisfies the exact rules that we wrote, but not the intention behind them.

Goodhart's Law

This problem is not new. It is known as Goodhart's law, named after the economist Charles Goodhart, and it is commonly stated as below:

When a measure becomes a target, it ceases to be a good measure.

Means, a number is a good measure only as long as nobody is pushing hard to increase it. The moment we start optimizing for that number, people (or agents) find ways to increase the number without improving the real thing.

In reinforcement learning, the reward is exactly such a target. Training pushes the agent to increase it as much as possible. So, Goodhart's law tells us that the reward will stop being a good measure if we push too hard.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Examples of Reward Hacking

The best way to learn this is by taking an example. Let's see a few well-known ones.

The boat racing game

OpenAI trained an agent to play a boat racing game called CoastRunners. The true goal was to finish the race. But, the reward was the in-game score, and the game gives points for hitting targets along the track.

The agent found a small lagoon with three targets that come back again after some time. So, it started going around in circles in that lagoon, hitting the same targets again and again, timed so that it reached each target just as it came back. It kept crashing into other boats, catching fire, and going the wrong way on the track. Still, it got a higher score than human players.

The proxy (score) went up. The true goal (finish the race) was never achieved.

Longer answers in LLMs

When we train an LLM (Large Language Model) with RLHF (Reinforcement Learning from Human Feedback), the reward comes from a reward model. A reward model is another model that reads an answer and gives it a score, based on what humans preferred in the past.

Human raters often prefer more detailed answers. So, the reward model learns to give longer answers a higher score. Now, the LLM discovers that it can increase its score just by writing longer answers, with extra lines, repeated points, and long lists, even when a short answer is better.

The score goes up, but the answer does not get more helpful.

Sycophancy

Sycophancy means telling people what they want to hear instead of what is true.

Human raters tend to rate an answer higher when it agrees with them. So, the reward model learns that agreeing with the user gives a higher score. Now, the LLM learns to agree with the user even when the user is wrong.

For example, the user says "I think 7 * 8 = 54, right?" and the model says "Yes, you are right" just to please the user, even though the correct answer is 56.

Coding agents and tests

For coding, we often give the reward based on tests. If the tests pass, the reward is high. If the tests fail, the reward is low.

The true goal is correct code. The proxy is "the tests pass". A coding agent can make the tests pass without writing correct code. It can edit the tests, delete the failing ones, or write code that only handles the exact inputs that the tests check.

Let's say we ask the agent to write a function that adds two numbers, and the tests check only two inputs. The agent can write the code as below:

def add(a, b):
    if a == 2 and b == 3:
        return 5
    if a == 10 and b == 20:
        return 30
    return 0

# the tests that give the reward
print(add(2, 3) == 5)
print(add(10, 20) == 30)

# a new input that the tests never checked
print(add(4, 4))

This will print the following:

True
True
0

Here, we can see that both tests pass, so the agent gets the full reward. But, the function does not add anything. It only remembers the two answers that the tests expect. For a new input like add(4, 4), it returns 0 instead of 8.

This kind of behavior has been reported in real coding agents trained with reinforcement learning.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Why does Reward Hacking Happen?

Now, the next big question is: why does reward hacking happen? There are three main reasons.

The reward is imperfect

Every reward has a gap. A reward model is trained on a limited number of human ratings, and it makes mistakes. A test suite checks only some inputs. A game score counts only some events. The length bias and the sycophancy that we saw earlier come from exactly such gaps in the reward model.

Optimization pressure

Reinforcement learning is very good at finding whatever gives a high reward. It tries a huge number of actions. If there is any small mistake in the reward, the agent is likely to find it, because that mistake is the easiest way to get more reward.

A weak agent often does not find the gap. A stronger agent that searches harder is more likely to find it.

Overoptimization

In reinforcement learning, the model that we are training is called the policy. In the beginning of training, when the proxy reward goes up, the true quality also goes up. But, after some point, the proxy reward keeps going up while the true quality starts going down. This is called reward model overoptimization.

Researchers have measured this. They used one reward model as the proxy for training, and a separate, bigger "gold" reward model to stand in for the true goal. As training pushed the policy further away from where it started, the proxy score kept rising, but the gold score first rose and then fell.

Let's see this as a diagram. The numbers are made up just for the sake of understanding.

┌─────────────────────────────────┐
│ Step 0: start of RL training    │
│ Proxy reward = 5.0              │
│ True quality = 5.0              │
└─────────────────────────────────┘
                 ↓
┌─────────────────────────────────┐
│ Step 1000                       │
│ Proxy reward = 7.0              │
│ True quality = 6.5              │
└─────────────────────────────────┘
                 ↓
┌─────────────────────────────────┐
│ Step 2000: best true quality    │
│ Proxy reward = 8.5              │
│ True quality = 7.0              │
└─────────────────────────────────┘
                 ↓
┌─────────────────────────────────┐
│ Step 3000: reward hacking       │
│ Proxy reward = 9.5              │
│ True quality = 5.5              │
└─────────────────────────────────┘

Here, we can see that till step 2000, both numbers go up together. After step 2000, the proxy reward keeps going up, but the true quality falls. The gap between the two numbers grows from 0.0 to 4.0. This growing gap is what reward hacking looks like during training.

To learn Reward Models, RLHF (Reinforcement Learning from Human Feedback), and Reinforcement Learning in depth, we can check out our AI and Machine Learning Program at Outcome School.

How do We Prevent Reward Hacking?

There is no single fix. In practice, we use many layers together, so that a hack that gets past one layer gets caught by another. Let's learn each one.

KL penalty to a reference model

We keep a frozen copy of the model from before RL training. Frozen means its weights do not change. This copy is called the reference model.

Then, during training, we measure how far the new model has moved away from the reference model. This distance is measured with KL divergence, which is a number that tells how different two probability distributions are. If the new model writes in a very different way than the reference model, the KL is large.

We subtract this distance from the reward:

total_reward = reward_model_score - beta * KL(new_model || reference_model)

Here, beta decides how strong the penalty is.

Let's see this with an example. Suppose the honest answer gets a score of 6.0 and stays close to the reference model with KL = 1.0. And a long, flattering answer gets a higher score of 7.5, but it is far from the reference model with KL = 5.0. These numbers are made up just for the sake of understanding.

beta = 0.5

def total_reward(reward_score, kl):
    return reward_score - beta * kl

print(total_reward(6.0, 1.0))  # honest answer
print(total_reward(7.5, 5.0))  # long, flattering answer

This will print the following:

5.5
5.0

Here, we can see that the flattering answer had a higher score, but after the penalty, the honest answer wins. Many hacks need the model to move far away from its normal way of writing, and the KL penalty makes that costly.

Better and ensemble reward models

A better reward model has a smaller gap, so there is less to hack. We get it with more data, cleaner human labels, and examples that cover known hacks.

We can also train several reward models and combine them, for example by taking their average or the lowest score. This is called an ensemble. A hack that fools one reward model often does not fool all of them.

Verifiable rewards with robust checks

For tasks like math and code, we can check the answer with a program instead of a reward model. This is called a verifiable reward. But, the checker itself must be robust:

  • Keep the test files read-only, so the agent cannot edit them.
  • Use hidden tests that the agent never sees.
  • Check many different inputs, not just a few.
  • Check the final answer strictly, so that a strange format cannot trick the checker.

Reward model retraining

The policy keeps changing during training, and it starts writing answers that the reward model never saw before. So, we collect fresh human ratings on the new answers, including the hacked ones, and retrain the reward model. This way, the reward model learns to give those hacks a low score.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Length normalization

For the length hack, we control the length directly. We can add a small penalty for extra length, or train the reward model so that it does not prefer an answer just because it is longer. When we evaluate, we also compare answers of similar length.

Human review and monitoring

Numbers alone can hide a hack. So, we must read real outputs during training. We must also watch signals like the average answer length, a sudden jump in reward, and changes to test files. If the model shows its reasoning, we can also check that reasoning for signs of gaming the reward.

Held-out evaluation

Finally, we measure the model with an evaluation that was never used as the reward. For example, a separate set of questions, human ratings, or a different judge. We run this evaluation on saved copies of the model during training, and once more at the end. If the training reward keeps going up while the held-out score goes down, we know that the model is hacking. Then, we go back to an earlier checkpoint, which is a saved copy of the model from before the held-out score started falling.

We have a detailed blog on LLM Evaluation that covers how such evaluations are built.

Let's see all the layers together as a diagram:

┌─────────────────────────────────┐
│ While computing the reward      │
│ - KL penalty                    │
│ - ensemble reward models        │
│ - robust verifiable checks      │
│ - length normalization          │
└─────────────────────────────────┘
                 ↓
┌─────────────────────────────────┐
│ While training                  │
│ - human review and monitoring   │
│ - reward model retraining       │
│ - held-out evaluation           │
└─────────────────────────────────┘
                 ↓
┌─────────────────────────────────┐
│ After training                  │
│ - final held-out evaluation     │
└─────────────────────────────────┘

Here, we can see that the layers work at different times. The first layer makes hacking less rewarding, the second layer catches hacks while training is going on, and the third layer checks the final model once more before we use it.

Now we must have understood what reward hacking in reinforcement learning is, why it happens, and how we prevent it.

Frequently Asked Questions

Is reward hacking the same as specification gaming?

They are closely related. Reward hacking is when an agent increases the proxy reward without achieving the true goal behind it. Specification gaming means the agent satisfies the exact rules that we wrote, but not the intention behind them. Both come from the same gap between what we measure and what we really want.

Is the agent trying to cheat on purpose?

No. The agent is not trying to be bad. It is doing exactly what the training pushes it to do, which is to get the highest reward number. It simply finds a path through the gap between the proxy reward and the true goal.

Does a stronger agent make reward hacking less likely?

No. A stronger agent makes reward hacking more likely. Reinforcement learning tries a huge number of actions, and if there is any small mistake in the reward, the agent is likely to find it. A weak agent often does not find the gap, but a stronger agent that searches harder is more likely to find it.

Can the KL penalty alone stop reward hacking?

No. There is no single fix for reward hacking. The KL penalty makes hacks that need the model to move far from the reference model costly, but we use many layers together, like ensemble reward models, robust verifiable checks, length normalization, human review, and held-out evaluation, so that a hack that gets past one layer gets caught by another.

How can we tell that a model is reward hacking during training?

We compare the training reward with a held-out evaluation that was never used as the reward. If the training reward keeps going up while the held-out score goes down, the model is hacking. We also read real outputs and watch signals like the average answer length, a sudden jump in reward, and changes to test files.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.