What is Reinforcement Learning with Verifiable Rewards (RLVR)?

Reinforcement Learning with Verifiable Rewards (RLVR) is a way to train an LLM where a program checks the model's final answer and gives a reward of 1 if it is correct and 0 if it is wrong. The model then learns to produce more of the answers that got 1.

In this blog, we will learn about Reinforcement Learning with Verifiable Rewards (RLVR). We will also see the problem with learned reward models, what a verifier is, how the RLVR training loop works step by step, how GRPO compares answers within a group with a small worked example, how RLVR led to reasoning models, where it works well and where it fails, and how it differs from RLHF.

We will cover the following:

• The problem with learned reward models
• What is Reinforcement Learning with Verifiable Rewards (RLVR)?
• What is a verifier?
• How the RLVR training loop works
• How GRPO compares answers within a group
• How RLVR led to reasoning models
• Limits of RLVR
• RLHF vs RLVR

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

The problem with learned reward models

In reinforcement learning, a model tries something, gets a score called a reward, and slowly learns to do more of what gets a high reward.

When we train an LLM (Large Language Model) with reinforcement learning, we need someone to give that reward for every answer the model writes. A human cannot read millions of answers. So, the common approach is to train another model, called a reward model, that has learned from human preferences and gives a score to each answer.

This approach is used in RLHF (Reinforcement Learning from Human Feedback). It works well for open-ended tasks like writing a polite email, where there is no single correct answer.

But a learned reward model has two big problems:

• It can be fooled. The reward model is itself a neural network that only guesses what humans like. The LLM being trained can find answers that get a high score without being good, for example, answers that are longer, sound more confident, or flatter the user. This is called reward hacking.
• It is expensive. We need to collect a lot of human preference data, train a separate large model, and run that model on every answer during training.

Now, the question is: for some tasks, do we even need a model to guess whether an answer is good?

Let's say we ask the LLM: "What is 12 + 30?" The correct answer is 42. We do not need a neural network to judge this. A few lines of code can check whether the final answer is 42.

Here comes Reinforcement Learning with Verifiable Rewards (RLVR) into the picture.

What is Reinforcement Learning with Verifiable Rewards (RLVR)?

Reinforcement Learning with Verifiable Rewards (RLVR) is a reinforcement learning method for training LLMs where the reward comes from a program that checks the answer against a known correct result, instead of from a learned reward model.

The word verifiable means "can be checked". So, RLVR works on tasks where the answer can be checked automatically and exactly.

Examples of such tasks:

• Math with a final answer: We know the correct number, so we compare the model's final number with it.
• Code with tests: We run the code the model wrote against test cases. If all tests pass, the answer is correct.
• Format rules: We check whether the output follows a required format, for example, the reasoning is inside <think> tags and the final answer is on a line starting with Answer:.

In each case, the reward is simple: 1 if correct, 0 if wrong. Some setups also give a small reward for the correct format, but the idea stays the same.

Let's see the difference in the source of the reward as a diagram:

RLHF                             RLVR

┌────────────────────────────┐   ┌────────────────────────────┐
│ Model answer: "42"         │   │ Model answer: "42"         │
└─────────────┬──────────────┘   └─────────────┬──────────────┘
              ↓                                ↓
┌────────────────────────────┐   ┌────────────────────────────┐
│ Reward model               │   │ A program that checks      │
│ (a neural network)         │   │ against 42                 │
└─────────────┬──────────────┘   └─────────────┬──────────────┘
              ↓                                ↓
┌────────────────────────────┐   ┌────────────────────────────┐
│ Score: 0.73 (a guess)      │   │ Reward: 1 (checked)        │
└────────────────────────────┘   └────────────────────────────┘

Here, we can see that both receive the same answer. In RLHF, a reward model guesses how good the answer is. In RLVR, a program checks the answer against the known correct result.

The term RLVR was popularized by the Tulu 3 work from AI2 (Allen Institute for AI), which used it to improve skills like math and following precise instructions.

To learn Reinforcement Learning, Reward Models, and RLHF in depth, check out our AI and Machine Learning Program at Outcome School.

What is a verifier?

The program that checks the answer and gives the reward is called a verifier.

The best way to learn this is by taking an example. Let's write a simple verifier for a math answer in Python as below:

import re

def verify(model_output, correct_answer):
    match = re.search(r"Answer:\s*(-?\d+)", model_output)
    if match is None:
        return 0  # no final answer found
    if int(match.group(1)) == correct_answer:
        return 1
    return 0

print(verify("12 + 30 = 42. Answer: 42", 42))
print(verify("12 + 30 = 32. Answer: 32", 42))
print(verify("I think it is around 40.", 42))

Here, we have:

• re.search looks for the text Answer: followed by a whole number, positive or negative, in the model output.
• If no such line exists, the verifier returns 0, because the model did not give a final answer in the required format.
• If the number matches correct_answer, it returns 1. Otherwise, it returns 0.

This will print the following:

1
0
0

Here, we can notice that the verifier does not read the reasoning at all. It only checks the final answer. The model is free to think in any way it likes, as long as it reaches the correct answer.

For code, the verifier runs the program on test cases instead of comparing a number. The idea is the same.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

How the RLVR training loop works

Now, let's understand the full training loop. We will use GRPO (Group Relative Policy Optimization) as the learning method, because it is the one used by DeepSeek-R1. Here, the policy is the LLM being trained, because it decides which token to write next.

Step 1: We pick a prompt that has a known correct answer, for example, "What is 12 + 30?" with the correct answer 42.

Step 2: The model generates several answers for the same prompt, for example, 4 answers. Each answer contains some reasoning and a final answer. Because the model picks tokens with some randomness, the 4 answers are different.

Step 3: The verifier checks each answer and gives a reward of 1 or 0.

Step 4: We compare each answer with the other answers in the same group. Answers that did better than the group average get a positive score, and answers that did worse get a negative score. This score is called the advantage.

Step 5: We update the model's parameters so that the tokens of answers with a positive advantage become more likely, and the tokens of answers with a negative advantage become less likely.

Then, we repeat this for many prompts, again and again.

We have a detailed blog on Policy Gradient and the REINFORCE algorithm that explains how this kind of update works step by step.

Let's see this as a diagram:

┌────────────────────────────────────────┐
│ Prompt: "What is 12 + 30?"             │
│ Correct answer: 42                     │
└───────────────────┬────────────────────┘
                    ↓
┌────────────────────────────────────────┐
│ LLM generates 4 answers                │
│ with reasoning                         │
└───────────────────┬────────────────────┘
                    ↓
┌────────────────────────────────────────┐
│ Verifier checks each answer            │
│ Rewards: 1, 0, 0, 0                    │
└───────────────────┬────────────────────┘
                    ↓
┌────────────────────────────────────────┐
│ Compare within the group (GRPO)        │
│ Advantages: +1.73, -0.58, -0.58, -0.58 │
└───────────────────┬────────────────────┘
                    ↓
┌────────────────────────────────────────┐
│ Update the LLM                         │
│ More likely: answer 1                  │
│ Less likely: answers 2, 3, 4           │
└───────────────────┬────────────────────┘
                    ↓
┌────────────────────────────────────────┐
│ Pick the next prompt and repeat        │
└────────────────────────────────────────┘

Here, we can see that there is no human and no reward model inside the loop. The only judge is the verifier.

How GRPO compares answers within a group

Now, let's see how the advantage is calculated in Step 4 with real numbers.

Let's say the model generated 4 answers for "What is 12 + 30?":

• Answer 1 says 42. Reward = 1.
• Answer 2 says 32. Reward = 0.
• Answer 3 says 43. Reward = 0.
• Answer 4 has no final answer. Reward = 0.

GRPO calculates the advantage of each answer using the mean and standard deviation of the rewards in the group:

advantage_i = (reward_i - mean(rewards)) / std(rewards)

Step 1: We find the mean of the rewards.

mean = (1 + 0 + 0 + 0) / 4 = 0.25

Step 2: We find the standard deviation. It tells us how spread out the rewards are.

• Difference of each reward from the mean: 0.75, -0.25, -0.25, -0.25
• Squares: 0.5625, 0.0625, 0.0625, 0.0625
• Average of squares: 0.75 / 4 = 0.1875
• Standard deviation: sqrt(0.1875) = 0.433

Step 3: We find the advantage of each answer.

• Answer 1: (1 - 0.25) / 0.433 = +1.73
• Answers 2, 3, and 4: (0 - 0.25) / 0.433 = -0.58

Here, we can notice a few things:

• The correct answer gets a large positive advantage, so the model will move toward it.
• The wrong answers get a negative advantage, so the model will move away from them.
• The advantages add up to zero, because the three differences of -0.25 cancel the one difference of +0.75 before we divide by 0.433.
• The group itself acts as the baseline, so we do not need a separate model to predict how good an answer should be. This is one reason GRPO is cheaper than older methods like PPO, which train an extra value model for this.

Now, one more important case. Suppose all 4 answers are correct. Then every reward is 1, the mean is 1, and every advantage is 0. The same happens if all 4 answers are wrong. Means, the model learns nothing from that prompt. It learns only when some answers in the group are correct and some are wrong.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

How RLVR led to reasoning models

Now, the next big question is: why did RLVR become so popular?

The answer is reasoning models. In January 2025, DeepSeek released DeepSeek-R1. Along with it, they described DeepSeek-R1-Zero, which was trained by applying reinforcement learning with GRPO directly on their base model, without first fine-tuning it on example answers.

For rewards, they used simple rules and no neural reward model:

• Accuracy reward: For math, the final answer had to match the correct answer. For coding problems, the code had to pass test cases.
• Format reward: The model had to put its thinking between <think> and </think> tags.

The DeepSeek team explained that they avoided a neural reward model because it can suffer from reward hacking during large-scale training, and it makes the training more complex.

The verifier only checked the final answer. It never told the model how to think. Still, as training went on, the model on its own started writing longer reasoning, checking its own steps, and trying a different approach when it found a mistake. Thinking longer helped it reach correct answers more often, and correct answers were the only thing that got rewarded.

Note: The final DeepSeek-R1 model used more stages than R1-Zero. It was first fine-tuned on a small set of long reasoning examples, then trained with reinforcement learning, then fine-tuned again, and then trained with one more round of reinforcement learning. But R1-Zero showed that strong reasoning can come from reinforcement learning with rule-based rewards alone.

This is how RLVR became a core part of training reasoning models.

Limits of RLVR

RLVR has three limits.

Only works where answers are checkable: We need a correct answer or a test to compare against. Tasks like writing a poem, summarizing a meeting, or giving advice have no single correct answer. For these, we still need human feedback or a reward model.

Verifier bugs can be gamed: The model is trained to get reward 1, not to be correct. If the verifier has a gap, the model can learn to exploit it. For example:

• If the verifier only checks that the code runs without crashing, the model can learn to write code that does nothing.
• If the verifier only checks whether the correct number appears anywhere in the output, the model can learn to write many different numbers so that one of them matches.

So, the verifier must be written carefully. Reward hacking is less common than with a learned reward model, but it is still possible.

Sparse reward: The reward is only 1 or 0 at the end of a long answer. The model does not get any signal about which step in its reasoning was right or wrong. And if a problem is too hard and the model never gets it right, all rewards are 0, and the model learns nothing from it. This is why the training problems must be at the right difficulty, not too easy and not too hard.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

RLHF vs RLVR

Let me tabulate the differences between RLHF and RLVR for your better understanding.

In practice, both are often used together. RLVR improves reasoning on math and code, and RLHF keeps the model helpful and safe on open-ended tasks.

Now we must have understood what Reinforcement Learning with Verifiable Rewards (RLVR) is, how it works, and where it works well and where it fails.

Frequently Asked Questions

Does the verifier in RLVR check the model's reasoning steps?

No. The verifier only checks the final answer and does not read the reasoning at all. The model is free to think in any way it likes, as long as it reaches the correct answer. This is also why the reward is sparse: the model gets no signal about which step in its reasoning was right or wrong.

What happens in RLVR when all answers in a group get the same reward?

The model learns nothing from that prompt. If all answers are correct, or all are wrong, the mean equals every reward, so every advantage is 0. The model learns only when some answers in the group are correct and some are wrong, which is why the training problems must be at the right difficulty.

Can reward hacking still happen in RLVR?

Yes. The model is trained to get a reward of 1, not to be correct, so it can exploit any gap in the verifier. For example, if the verifier only checks that the code runs without crashing, the model can learn to write code that does nothing. Reward hacking is less common than with a learned reward model, but it is still possible.

Are RLVR and RLHF used together?

Yes. In practice, both are often used together. RLVR improves reasoning on math and code, where the answer can be checked, and RLHF keeps the model helpful and safe on open-ended tasks like writing and chat, where there is no single correct answer.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.