What is RLAIF? Reinforcement Learning from AI Feedback Explained

RLAIF (Reinforcement Learning from AI Feedback) is a way to train an LLM where another AI model, and not a human, decides which of two answers is better. These AI decisions are then used to teach the LLM to give better answers.

In this blog, we will learn about RLAIF (Reinforcement Learning from AI Feedback). We will also see a quick recap of RLHF and its problem with human labelers, how an AI labeler guided by written principles gives the preference labels, the full RLAIF pipeline step by step, the direct RLAIF variant and its link to Constitutional AI, what research has found, and where it works well and where it fails.

We will cover the following:

• A Quick Recap of RLHF
• The Problem with Human Labelers
• What is RLAIF?
• The AI Labeler and Its Principles
• How RLAIF Works Step by Step
• Direct RLAIF
• RLAIF and Constitutional AI
• What Does the Research Say?
• Advantages of RLAIF
• Disadvantages of RLAIF
• RLHF vs RLAIF

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

A Quick Recap of RLHF

Before jumping into RLAIF, we must know about RLHF, because RLAIF is built on top of it.

RLHF (Reinforcement Learning from Human Feedback) is a way to make an LLM (Large Language Model) give answers that people like.

It works in four steps:

• The LLM writes two answers for the same prompt.
• A human reads both answers and picks the better one. This choice is called a preference label.
• We collect many such labels and train a reward model. A reward model is a model that reads an answer and gives it a score. A higher score means a better answer.
• Finally, we use reinforcement learning, which means learning by trying and getting a reward. It is usually done with an algorithm called PPO (Proximal Policy Optimization), which updates the LLM so that it writes answers that get higher scores.

Let's see this as a diagram:

┌──────────────────────────────────┐
│ Prompt: "Explain gravity simply" │
└────────────────┬─────────────────┘
                 ↓
┌──────────────────────────────────┐
│ LLM writes Answer A and Answer B │
└────────────────┬─────────────────┘
                 ↓
┌──────────────────────────────────┐
│ Human labeler picks the better   │
│ one: "A is better than B"        │
└────────────────┬─────────────────┘
                 ↓
┌──────────────────────────────────┐
│ Reward model learns from 100,000 │
│ such human labels                │
└────────────────┬─────────────────┘
                 ↓
┌──────────────────────────────────┐
│ PPO updates the LLM to get       │
│ higher reward scores             │
└──────────────────────────────────┘

Here, we can see that the human sits in the middle of the pipeline. Every preference label comes from a person.

To learn RLHF, Reward Models, and Proximal Policy Optimization (PPO) end to end, check out our AI and Machine Learning Program at Outcome School.

The Problem with Human Labelers

RLHF works very well. But, the human part creates three problems.

Problem 1: Cost. A good reward model needs tens of thousands to hundreds of thousands of preference labels. Each label needs a trained person to read two answers carefully. Paying for so many labels is very expensive.

Problem 2: Speed. A human takes time to read two long answers and decide. Collecting a large set of labels can take weeks. And if we change our model or our rules, we must collect labels again.

Problem 3: Consistency. Different people have different tastes. One labeler likes short answers, another likes detailed answers. Even the same person can decide differently when tired. So, the labels become noisy.

Now, the question is: can we replace the human with something cheaper, faster, and more consistent?

The answer is yes. Here comes the RLAIF into the picture.

What is RLAIF?

RLAIF (Reinforcement Learning from AI Feedback) is a way to train an LLM where the preference labels come from an AI model instead of humans.

Everything else stays almost the same as RLHF. We only swap the human labeler with an AI labeler.

Let's say we have a new employee who writes reports. In RLHF, the manager reads every report and gives feedback. In RLAIF, we give a senior employee a written checklist of what a good report looks like, and the senior employee gives the feedback instead of the manager. The manager only writes the checklist once.

In RLAIF, this checklist is called a rubric. Let's understand how it works.

The AI Labeler and Its Principles

The AI labeler is usually a strong, already trained LLM. We do not train it again for this job. We just give it a prompt.

We have a detailed blog on LLM as a Judge that explains how one LLM evaluates the output of another LLM.

The prompt contains a written set of principles, which is the rubric. The rubric tells the AI labeler what "better" means for us.

For example, the rubric can say:

• The answer must be correct.
• The answer must be easy to understand.
• The answer must not contain harmful content.
• The answer must not be longer than needed.

Then, we give the AI labeler the prompt, the two answers, and the question "Which answer is better?" as below:

You are judging two answers. Follow these principles:
1. The answer must be correct.
2. The answer must be easy to understand.
3. The answer must not contain harmful content.

Question: Explain gravity simply.

Answer 1: Gravity is the force that pulls objects toward each other...
Answer 2: Gravity is described by the curvature of spacetime in...

Which answer is better? Reply with 1 or 2.

Here, the AI labeler reads both answers, checks them against the principles, and replies with "1" or "2". That reply is our preference label.

In practice, we do not only take the final "1" or "2". We look at how likely the AI labeler thinks "1" is and how likely it thinks "2" is. For example, 0.8 for "1" and 0.2 for "2". This gives a soft label, which carries more information than a plain yes or no. Researchers also often ask the AI labeler to first explain its reasoning and then give the answer, because this improves the quality of the labels in many cases.

The best part is that if we want to change what "better" means, we just edit the rubric. We do not need to hire and retrain a team of people.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

How RLAIF Works Step by Step

Now, let's see the full RLAIF pipeline step by step.

Step 1: We generate two answers. We take a prompt and ask the LLM that we want to improve to write two different answers for it. We do this for thousands of prompts.

Step 2: The AI labeler compares the two answers using the principles. For each pair, the AI labeler reads the rubric, the prompt, and both answers, and decides which one is better.

Step 3: We train a reward model on the AI labels. We take all the AI preference labels and train a reward model, exactly like in RLHF. The reward model learns to give a higher score to the kind of answers the AI labeler preferred.

Step 4: We run reinforcement learning with PPO. The LLM writes answers, the reward model scores them, and PPO updates the LLM to get higher scores. During this step, we also keep the LLM close to its original version, so that it does not drift into strange answers just to chase the score. We have a detailed blog on Reward Hacking that covers why chasing the score goes wrong and how we prevent it.

Let's see RLHF and RLAIF side by side as a diagram:

          RLHF                         RLAIF

┌──────────────────────┐     ┌──────────────────────┐
│ LLM writes 2 answers │     │ LLM writes 2 answers │
└──────────┬───────────┘     └──────────┬───────────┘
           ↓                            ↓
┌──────────────────────┐     ┌──────────────────────┐
│ Human labeler picks  │     │ AI labeler + rubric  │
│ the better answer    │     │ picks the better one │
└──────────┬───────────┘     └──────────┬───────────┘
           ↓                            ↓
┌──────────────────────┐     ┌──────────────────────┐
│ Train reward model   │     │ Train reward model   │
│ on human labels      │     │ on AI labels         │
└──────────┬───────────┘     └──────────┬───────────┘
           ↓                            ↓
┌──────────────────────┐     ┌──────────────────────┐
│ PPO updates the LLM  │     │ PPO updates the LLM  │
└──────────────────────┘     └──────────────────────┘

Here, we can see that both pipelines are the same except for the second box. Only the source of the preference label changes.

Direct RLAIF

There is one more variant called direct RLAIF.

In normal RLAIF, the AI labels go into a separate reward model, and the reward model gives the scores during training. But, we can skip the reward model completely.

In direct RLAIF, the AI labeler itself gives the reward score during reinforcement learning. For each answer that the LLM writes, we ask the AI labeler: "Rate this answer from 1 to 10 based on these principles." That score is used directly as the reward for reinforcement learning.

Let's see this as a diagram:

┌──────────────────────────────────┐
│ LLM writes one answer            │
└────────────────┬─────────────────┘
                 ↓
┌──────────────────────────────────┐
│ AI labeler + rubric gives a      │
│ score: 7 out of 10               │
└────────────────┬─────────────────┘
                 ↓
┌──────────────────────────────────┐
│ RL uses 7/10 as the reward and   │
│ updates the LLM                  │
└──────────────────────────────────┘

Here, we can see that there is no separate reward model. The AI labeler plays the role of the reward model.

Advantage: We do not need to train and maintain a reward model. Also, a separate reward model slowly becomes outdated as the LLM changes during training, but the AI labeler judges every new answer fresh.

Disadvantage: Calling a large AI labeler for every answer during training is slow and costly, because the AI labeler is often a much larger model than a reward model.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

RLAIF and Constitutional AI

RLAIF is closely connected to Constitutional AI, a method introduced by Anthropic in 2022.

In Constitutional AI, the written set of principles is called a constitution. It is a list of rules like "Choose the response that is least harmful."

Constitutional AI has two stages:

• Stage 1 (supervised learning): The model writes an answer, then critiques its own answer using a principle from the constitution, and then rewrites it. The model is then trained further on these improved answers.
• Stage 2 (reinforcement learning): The model writes two answers, and an AI model picks the better one using the constitution. These AI labels train a preference model, which is another name for a reward model, and then reinforcement learning is done on top of it.

Stage 2 is RLAIF. In fact, the Constitutional AI paper used the term "RLAIF" for this stage. In that work, AI feedback was used for harmlessness, while the labels for helpfulness still came from humans.

So, Constitutional AI is one well-known method that uses RLAIF as one of its parts, with the constitution playing the role of the rubric.

What Does the Research Say?

A well-known comparison comes from a 2023 paper by researchers at Google, titled "RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback".

They tested on tasks like summarization, helpful dialogue, and harmless dialogue. Human evaluators compared the outputs. The paper reported that RLAIF performed comparably to RLHF on these tasks.

For example, in summarization, human evaluators preferred both the RLAIF and the RLHF outputs over the starting model at similar rates. And when the evaluators compared RLAIF and RLHF outputs directly, they preferred each one about equally often.

On harmless dialogue, RLAIF did even better than RLHF in their tests.

The same paper also reported that direct RLAIF performed better than the normal RLAIF with a reward model in their setup.

We must keep in mind that these results are for specific tasks and specific models. They do not mean that AI feedback can replace human feedback everywhere.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Advantages of RLAIF

• Cheap: An AI label costs much less than a human label.
• Fast: An AI labeler can produce thousands of labels in the time a human produces a few.
• Consistent: The same rubric is applied the same way every time. There is no tiredness and no mood.
• Easy to change: To change what "better" means, we edit the rubric and label again.
• Scales well: We can label millions of pairs, which is not practical with humans.

Disadvantages of RLAIF

• Bias of the AI labeler: The AI labeler has its own mistakes and preferences. For example, it can prefer longer answers or answers written in its own style. Whatever bias it has gets passed into the reward model and then into the LLM.
• Errors get copied: If the AI labeler is wrong about a fact, it can reward the wrong answer. The LLM then learns that mistake.
• Limited by the labeler: The labels can only be as good as the AI labeler's judgment. For a very hard or very new topic, the AI labeler can be unreliable.
• Position bias: The AI labeler can prefer an answer just because of its position, for example, because it is shown first.

Let's understand position bias and its fix in detail.

Let's say we show Answer A first and Answer B second. The AI labeler gives A a probability of 0.80 of being better. Now, we swap the order and show B first and A second. This time, the AI labeler gives A only 0.60. The answers are the same. Only the order changed. This means the position affected the decision.

To fix this, we ask the AI labeler twice, once in each order, and take the average.

Let's see this as a diagram:

┌────────────────────┐     ┌────────────────────┐
│ Order 1: A, then B │     │ Order 2: B, then A │
│ P(A better) = 0.80 │     │ P(A better) = 0.60 │
└─────────┬──────────┘     └─────────┬──────────┘
          ↓                          ↓
       ┌──────────────────────────────────┐
       │ Average: (0.80 + 0.60) / 2       │
       │ Final P(A better) = 0.70         │
       └──────────────────────────────────┘

We can write the code as below:

# Probability that the AI labeler picks Answer A
p_a_when_a_first = 0.80   # A shown in position 1
p_a_when_a_second = 0.60  # A shown in position 2

p_a = (p_a_when_a_first + p_a_when_a_second) / 2
p_b = 1 - p_a

print(f"P(A is better) = {p_a:.2f}")
print(f"P(B is better) = {p_b:.2f}")

This will print the following:

P(A is better) = 0.70
P(B is better) = 0.30

Here, we can see that by averaging both orders, the effect of the position is balanced out. Answer A gets the first position once and the second position once, so the final label 0.70 depends on the answers and not on their order.

RLHF vs RLAIF

Let me tabulate the differences between RLHF and RLAIF for your better understanding.

We can also mix both. We can use humans for a smaller set of important or difficult labels, and AI feedback for the large volume of routine labels.

Now we must have understood what RLAIF is, how it works step by step, and where it works well and where it fails.

Frequently Asked Questions

Does RLAIF remove humans from the training completely?

No. RLAIF does not remove humans completely. Humans still write the rubric, the written set of principles that tells the AI labeler what "better" means. We can also mix both, using humans for a smaller set of important or difficult labels and AI feedback for the large volume of routine labels.

Do we need to train a new model to act as the AI labeler?

No. The AI labeler is usually a strong, already trained LLM, and we do not train it again for this job. We just give it a prompt that contains the rubric, the prompt, and the two answers. To change what "better" means, we only edit the rubric.

Why do we use the AI labeler's probabilities instead of only its final answer?

Because probabilities carry more information than a plain yes or no. Instead of only taking "1" or "2", we look at how likely the AI labeler thinks each answer is better, for example 0.8 for "1" and 0.2 for "2". This gives a soft label for training the reward model.

Is RLAIF as good as RLHF?

On some tasks, yes. A 2023 paper by researchers at Google reported that RLAIF performed comparably to RLHF on summarization and helpful dialogue, and even better on harmless dialogue. But these results are for specific tasks and specific models, so they do not mean that AI feedback can replace human feedback everywhere.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.