Outcome School

What is Actor-Critic in Reinforcement Learning? The Advantage Function Explained

Authors
What is Actor-Critic in Reinforcement Learning? The Advantage Function Explained

Actor-Critic is a way of training an AI agent where one part, the Actor, picks the actions, and another part, the Critic, judges how good each action was right after it happens. This judgment is called the advantage, and it tells the Actor how much better or worse an action was than usual.

In this blog, we will learn about Actor-Critic in Reinforcement Learning and the Advantage Function. We will also see the problem with REINFORCE that Actor-Critic solves, what the Actor and the Critic are, what the advantage function means with real numbers, how the TD error estimates the advantage, how the full update loop works step by step in code, and why it works better and where it is used today.

We will cover the following:

  • The problem with REINFORCE
  • What is Actor-Critic?
  • What is the Advantage Function?
  • Advantage with real numbers
  • TD Error: estimating the advantage in one step
  • The Actor-Critic update loop
  • Actor-Critic in code
  • Why Actor-Critic works better
  • Where Actor-Critic is used

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

The problem with REINFORCE

Before jumping into Actor-Critic, we must know the problem it solves.

In Reinforcement Learning, an agent looks at a state (the current situation), picks an action, and gets a reward (a number that says how good the result was). The rule the agent uses to pick actions is called the policy.

REINFORCE is a simple method to train the policy. In its simplest form, the agent plays one full episode (one complete game from start to end), adds up the rewards, and then makes the actions in that episode more likely if the total was high, or less likely if the total was low.

This has two problems:

  • It waits until the end. The agent learns nothing until the full episode is over. If an episode has 1,000 steps, all 1,000 steps go by with zero learning.
  • High variance. The total reward changes a lot from one episode to another, even for the same actions, because of luck. One good action inside a bad episode gets punished. One bad action inside a lucky episode gets rewarded. So, the learning signal is very noisy, and the training is slow and shaky.

Let's see this as a diagram:

┌───────────────────────────────────┐
│ Step 1, Step 2, ..., Step 1000    │
│ (no learning yet)                 │
└───────────────────────────────────┘
                  ↓
┌───────────────────────────────────┐
│ Episode ends: total reward = 40   │
└───────────────────────────────────┘
                  ↓
┌───────────────────────────────────┐
│ All 1000 actions get the same     │
│ score of 40, good or bad          │
└───────────────────────────────────┘

Here, we can see that every action gets judged by one number that arrives only at the end. The agent cannot tell which of the 1,000 actions actually helped.

What we need is someone who can judge each action right after it happens. Here comes the Actor-Critic into the picture.

What is Actor-Critic?

Actor-Critic is a Reinforcement Learning method that uses two parts: an Actor that decides which action to take, and a Critic that estimates how good the current state is and uses it to judge each action.

  • The Actor is the policy. It looks at the state and gives the probability of each action. For example, "jump: 70%, wait: 30%".
  • The Critic is a value function. It looks at the state and predicts a single number: the total future reward the agent usually gets from this state. This number is written as V(s), the value of state s.

Both are usually neural networks, and both learn at the same time.

Let's say a young cricket player is practicing batting with a coach standing next to him. The player plays the shots. That is the Actor. The coach watches every shot and immediately says "that was better than your usual" or "that was worse than your usual". That is the Critic. The player does not have to wait for the final score of the match to know which shot was good.

Now, the question is: how does the Critic say "better than usual" as a number? The answer is the advantage function.

What is the Advantage Function?

Before jumping into the advantage, we need two values:

  • V(s): the value from the Critic, which means the total future reward we usually get from state s. It is the average result from this state.
  • Q(s, a): the total future reward we expect if we are in state s, take the specific action a first, and then follow our policy as usual.

The advantage function is the difference between these two:

A(s, a) = Q(s, a) - V(s)

Means, the advantage tells us how much better (or worse) action a is compared to what we usually get from state s.

  • If A(s, a) is positive, the action is better than usual. The Actor must make it more likely.
  • If A(s, a) is negative, the action is worse than usual. The Actor must make it less likely.
  • If A(s, a) is zero, the action is just as good as usual. Its probability stays about the same.

REINFORCE asks "was the total reward high?". The advantage asks "was this action better than what we normally get here?". The second question removes the luck of the state itself. If the agent is in a great state, everything looks good there, and the advantage takes that out by subtracting V(s).

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Advantage with real numbers

The best way to learn this is by taking an example.

Suppose our agent is playing a simple game, and it is in some state s. The Critic says:

V(s) = 10

Means, from this state, the agent usually collects 10 points in total.

Now, the agent has two actions: jump and wait. Assume that we know the Q-values:

  • Q(s, jump) = 14
  • Q(s, wait) = 7

Let's calculate the advantage of each action:

A(s, jump) = 14 - 10 = +4

A(s, wait) = 7 - 10 = -3

Jump gives 4 points more than usual, and wait gives 3 points less than usual. Let's see this as a diagram:

┌────────────────────┐   ┌────────────────────┐
│ Action: jump       │   │ Action: wait       │
│ Q(s, jump) = 14    │   │ Q(s, wait) = 7     │
└────────────────────┘   └────────────────────┘
          ↓                        ↓
┌────────────────────┐   ┌────────────────────┐
│ minus V(s) = 10    │   │ minus V(s) = 10    │
└────────────────────┘   └────────────────────┘
          ↓                        ↓
┌────────────────────┐   ┌────────────────────┐
│ A = +4             │   │ A = -3             │
│ make jump more     │   │ make wait less     │
│ likely             │   │ likely             │
└────────────────────┘   └────────────────────┘

Here, we can see that V(s) = 10 acts as the baseline for both actions. We judge each action against this baseline, not against zero. So, the Actor increases the probability of jump and decreases the probability of wait.

To calculate the advantage, we need both Q(s, a) and V(s). Keeping two separate networks for Q and V is extra work. So, how can we get the advantage using only V? The answer is the TD error.

TD Error: estimating the advantage in one step

TD stands for Temporal Difference. Temporal means "related to time". The TD error is the difference between what the Critic predicted before a step and what it sees after that step.

After the agent takes action a in state s, it gets a reward r and lands in a new state s'. Now, we can estimate Q(s, a) as below:

Q(s, a) is approximately r + gamma * V(s')

Means, the value of taking action a is the reward we got right now, plus the value of the state where we landed.

Here:

  • r is the reward we just got.
  • V(s') is the Critic's prediction of the future reward from the new state.
  • gamma is the discount factor, a number between 0 and 1 (for example, 0.9). It makes future rewards count a bit less than the reward we get right now.

Now, we put this into the advantage formula, A(s, a) = Q(s, a) - V(s), and we get the TD error, written as delta:

delta = r + gamma * V(s') - V(s)

This delta is our estimate of the advantage. And we need only the Critic (V) to calculate it.

Let's take an example with numbers.

Example 1 (positive): The Critic says V(s) = 9. The agent takes the action jump and gets a reward r = 2. It lands in a new state where the Critic says V(s') = 10. We use gamma = 0.9.

delta = 2 + 0.9 * 10 - 9 = 2 + 9 - 9 = 2

The TD error is +2. Means, things went 2 points better than the Critic expected. So, the action gets a positive advantage.

Example 2 (negative): Same state, V(s) = 9. This time, the agent takes the action wait, gets a reward r = 0, and lands in a bad state where V(s') = 5.

delta = 0 + 0.9 * 5 - 9 = 0 + 4.5 - 9 = -4.5

The TD error is -4.5. Means, things went 4.5 points worse than expected. So, the action gets a negative advantage.

Here, we can notice that we got this judgment right after one step, without waiting for the end of the episode.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

The Actor-Critic update loop

Now, let's put everything together. At every step, the agent follows this loop.

Step 1: The agent looks at the current state s.

Step 2: The Actor gives the probability of each action, and the agent picks an action a based on these probabilities.

Step 3: The environment (the game or world the agent lives in) returns a reward r and the new state s'.

Step 4: The Critic calculates the TD error: delta = r + gamma * V(s') - V(s). This is the advantage estimate.

Step 5: The Critic updates itself so that V(s) moves closer to r + gamma * V(s'). In simple form: V(s) = V(s) + alpha_critic * delta, where alpha_critic is the learning rate, a small number like 0.1 that controls how big each update is.

Step 6: The Actor updates itself using the same delta. If delta is positive, it increases the probability of action a in state s. If delta is negative, it decreases it. The bigger the size of delta, the bigger the change.

Step 7: The new state s' becomes the current state, and we go back to Step 1.

Let's see this loop as a diagram:

┌─────────────────────────────┐
│ State s                     │
└─────────────────────────────┘
               ↓
┌─────────────────────────────┐
│ Actor (policy)              │
│ jump: 50%, wait: 50%        │
└─────────────────────────────┘
               ↓
┌─────────────────────────────┐
│ Action a = jump             │
└─────────────────────────────┘
               ↓
┌─────────────────────────────┐
│ Environment                 │
│ reward r = 2, new state s'  │
└─────────────────────────────┘
               ↓
┌─────────────────────────────┐
│ Critic (value function)     │
│ V(s) = 9, V(s') = 10        │
└─────────────────────────────┘
               ↓
┌─────────────────────────────┐
│ Advantage (TD error)        │
│ delta = 2 + 0.9 * 10 - 9    │
│       = +2                  │
└─────────────────────────────┘
               ↓
┌─────────────────────────────┐
│ Update both                 │
│ Critic: V(s) 9 -> 9.2       │
│ Actor: jump 50% -> 55%      │
└─────────────────────────────┘
               ↓
┌─────────────────────────────┐
│ s' becomes the new state s  │
│ (back to the top)           │
└─────────────────────────────┘

Here, we can see that one number, the TD error, drives both updates. The Critic uses it to fix its own prediction, and the Actor uses it to decide which way to change the action probabilities.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Actor-Critic in code

Let's see the code for one step of this loop, using the numbers from Example 1. For the sake of understanding, the Critic is a simple table of values, and the Actor has just two actions.

import numpy as np

gamma = 0.9
V = {"s": 9.0, "s_next": 10.0}   # Critic's predictions
r = 2.0                          # reward from the environment

delta = r + gamma * V["s_next"] - V["s"]
print("TD error:", round(delta, 2))

V["s"] = V["s"] + 0.1 * delta    # Critic update
print("Critic's new V(s):", round(V["s"], 2))

logits = np.array([0.0, 0.0])    # Actor's scores for [jump, wait]
probs = np.exp(logits) / np.exp(logits).sum()
grad = -probs
grad[0] += 1                     # we took action 0 (jump)
logits = logits + 0.1 * delta * grad   # Actor update
probs = np.exp(logits) / np.exp(logits).sum()
print("Actor's new probabilities:", probs.round(2))

This will print the following:

TD error: 2.0
Critic's new V(s): 9.2
Actor's new probabilities: [0.55 0.45]

Here, we have:

  • Calculated the TD error as 2.0, same as Example 1.
  • Moved the Critic's V(s) from 9 to 9.2, a little closer to the target r + gamma * V(s') = 2 + 0.9 * 10 = 11.
  • Stored the Actor as two raw scores (logits), one per action, and turned them into probabilities with softmax. Both start at 0, so each action is at 50%.
  • Calculated grad, which is the direction that raises the probability of the action we took. For softmax, it is [1, 0] (because we chose jump) minus the current probabilities [0.5, 0.5], which gives [0.5, -0.5].
  • Moved the scores along grad, scaled by delta and a small learning rate of 0.1. The score of jump went up, the score of wait went down, and the probability of jump went from 50% to 55%.

If delta were negative, the same lines would push the probability of jump down instead. In a real system, the Actor and the Critic are neural networks, and delta scales the updates to their weights in the same way.

To master Reinforcement Learning and how the Actor and the Critic learn together from rewards, check out our AI and Machine Learning Program at Outcome School, where we cover it in depth.

Why Actor-Critic works better

Now, let's understand why this works better than REINFORCE.

  • Lower variance. REINFORCE judges an action by the full noisy total of the episode. Actor-Critic judges an action by the advantage, which compares it with the usual value of that state. This removes a lot of the luck, so the learning signal is much more stable.
  • Learns at every step. Actor-Critic does not wait for the episode to end. It updates after every step, so it learns faster and also works for very long or never-ending tasks.

Note: The Critic's predictions are not perfect, especially at the start of training, so the TD error has some bias. In practice, this bias is a good trade for the large drop in variance.

Where Actor-Critic is used

Actor-Critic is the base of many popular Reinforcement Learning algorithms:

  • A2C (Advantage Actor-Critic): the Actor-Critic method we learned in this blog, usually run on many copies of the environment at the same time, with updates that wait for all copies.
  • A3C (Asynchronous Advantage Actor-Critic): many workers run their own copies of the environment and update the shared networks on their own, without waiting for each other.
  • PPO (Proximal Policy Optimization): an Actor-Critic method that also limits how much the Actor can change in one update, so that training stays stable. It is one of the most used Reinforcement Learning algorithms.

PPO is also used in RLHF (Reinforcement Learning from Human Feedback), one of the methods used to train LLMs (Large Language Models) to give helpful answers. Here:

  • The Actor is the LLM itself. Its action is the next token it generates.
  • The reward comes from a reward model, which scores the full answer.
  • The Critic is a separate network called the value model. It predicts how good the answer is going to be from the tokens generated so far.

With the help of the value model, we get an advantage for every token, so the LLM learns which parts of its answer made it better or worse than usual.

We have a detailed blog on Group Relative Policy Optimization (GRPO) that explains how an LLM can be trained without a value model at all.

Now we must have understood what Actor-Critic is, what the advantage function means, how the TD error estimates it, and how both the Actor and the Critic learn together at every step.

Frequently Asked Questions

What is the difference between the Actor and the Critic?

The Actor is the policy. It looks at the state and gives the probability of each action. The Critic is a value function. It looks at the state and predicts the total future reward the agent usually gets from that state. Both are usually neural networks, and both learn at the same time.

What does a negative advantage mean?

A negative advantage means the action was worse than what the agent usually gets from that state. So, the Actor makes that action less likely. A positive advantage means the action was better than usual, and the Actor makes it more likely. A zero advantage means the action was just as good as usual.

Why does Actor-Critic not need a separate Q network?

Actor-Critic estimates the advantage with the TD error, which needs only the Critic's values. The TD error is the reward we just got, plus the discounted value of the new state, minus the value of the current state. Keeping two separate networks for Q and V would be extra work, so the TD error removes the need for Q.

Is the TD error a perfect estimate of the advantage?

No. The Critic's predictions are not perfect, especially at the start of training, so the TD error has some bias. In practice, this bias is a good trade for the large drop in variance compared to REINFORCE, which judges every action by the full noisy total of the episode.

What are A2C and A3C?

Both are Actor-Critic algorithms. A2C (Advantage Actor-Critic) usually runs on many copies of the environment at the same time, with updates that wait for all copies. A3C (Asynchronous Advantage Actor-Critic) uses many workers that run their own copies of the environment and update the shared networks on their own, without waiting for each other.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.