What is a Value Function? Q-Learning Explained Step by Step

A Value Function gives a number that tells an agent how much total reward it can expect from a situation, and Q-Learning is a method that learns these numbers for every situation and action by trial and error.

In this blog, we will learn about what a Value Function is and how Q-Learning works step by step. We will also see what return and discount mean, the State Value Function and the Action Value Function, the Bellman idea, the Q-table and the Q-Learning update rule, a step-by-step walkthrough on a tiny corridor, and a short Python implementation along with where the Q-table fails.

We will cover the following:

• Return and Discount
• What is a Value Function?
• State Value Function V(s)
• Action Value Function Q(s, a)
• The Bellman Idea
• What is Q-Learning?
• The Q-Learning Update Rule
• Our Tiny Corridor
• Q-Learning Step by Step
• Epsilon-Greedy
• Q-Learning in Python
• Limitation of the Q-table and Deep Q-Network

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

Return and Discount

In Reinforcement Learning, an agent is in a state (the situation it is in), picks an action, gets a reward (a number that says how good that move was), and lands in the next state. The goal of the agent is to collect as much reward as possible over time.

The return is the total reward the agent collects from now until the end. A reward that comes later is worth a little less than a reward that comes now, so we multiply each future reward by a discount factor called gamma (a number between 0 and 1, for example 0.9).

G = r1 + gamma * r2 + gamma^2 * r3 + ...

Here, r1 is the reward from the next move, r2 is the reward from the move after that, and so on. Each extra step of waiting adds one more gamma.

Let's say the agent gets a reward of 10 after 3 moves and nothing before that. With gamma = 0.9, the return is 0 + 0.9 * 0 + 0.81 * 10 = 8.1. The same reward of 10 after 1 move gives a return of 10. Hence, the discount makes the agent prefer reaching the reward sooner.

We have a detailed blog on Markov Decision Process (MDP) that explains how states, actions, rewards, and the discount factor fit into one framework.

What is a Value Function?

A Value Function tells us how good it is for the agent to be in a state, or to take an action in a state, measured as the return the agent can expect from there.

Let's say we are playing a board game. Being two squares away from the finish line is a better position than being twenty squares away, even though we have not won anything yet. A Value Function puts a number on that "better position" feeling.

There are two kinds of Value Functions. Let's learn about each of them.

State Value Function V(s)

The State Value Function V(s) is the return the agent can expect if it starts in state s and then keeps following its way of choosing actions.

That way of choosing actions is called the policy. So, V(s) depends on the policy. A good policy gives high values, and a bad policy gives low values.

V(s) answers the question: "How good is this state?"

Action Value Function Q(s, a)

The Action Value Function Q(s, a) is the return the agent can expect if it is in state s, takes action a right now, and then keeps following its policy.

Q(s, a) answers a more detailed question: "How good is this action in this state?"

The letter Q comes from the word "quality". Q(s, a) is the quality of action a in state s.

Now, the relation between the two is simple. If the agent always picks the best action, then the value of a state is the value of its best action:

V(s) = max over a of Q(s, a)

Means, if we know Q for every action, we also know V. And with Q, we can choose the action directly: in each state, pick the action with the highest Q. With only V, the agent would also need to know where each action leads. This is why Q-Learning learns Q and not V.

To learn Reinforcement Learning and how an agent judges its states and actions from the ground up, check out our AI and Machine Learning Program at Outcome School.

The Bellman Idea

Now, the next big question is: how do we compute these values?

The answer is the Bellman idea, which in plain words says:

The value of where we are now = the reward we get now + the discounted value of where we land next.

For Q, when the agent picks the best action from the next state onward, it becomes:

Q(s, a) = r + gamma * max Q(s', a')

Here, s' is the next state and a' is any action in that next state. max Q(s', a') is the value of the best action from the next state. For the sake of understanding, we assume that the same action in the same state always gives the same reward and the same next state. If the result is random, we take the average over all the possible results.

We do not need to look all the way to the end of the game. We need one step of real reward plus our current guess of the value of the next state.

What is Q-Learning?

Q-Learning is a Reinforcement Learning method in which the agent learns the Q(s, a) value for every state and action by trying actions, seeing the rewards, and improving its guesses using the Bellman idea.

The agent keeps all its guesses in a table called the Q-table. Each row is a state, each column is an action, and each cell holds the current guess of Q(s, a).

At the start, every cell is 0. The agent updates the cells one by one as it practices, and in the end, it picks the action with the highest value in each state.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

The Q-Learning Update Rule

Every time the agent takes one action, it updates one cell of the Q-table using this rule:

Q(s,a) <- Q(s,a) + alpha * (r + gamma * max Q(s',a') - Q(s,a))

Let's understand each term:

• <-: replace the old value in the cell with the new value on the right side.
• Q(s,a): the current guess for taking action a in state s.
• r: the reward the agent just got.
• gamma: the discount factor, for example 0.9.
• max Q(s',a'): the best value the agent currently believes it can get from the next state s'. If s' is the end of the game, this is 0.
• r + gamma * max Q(s',a'): the target. This is the Bellman idea, a better guess built from one real reward.
• target - Q(s,a): the error. It tells how far the old guess is from the target.
• alpha: the learning rate, a number between 0 and 1, for example 0.5. It decides how big a step we take toward the target.

So, the rule moves the old guess a part of the way toward the target.

Let's see this as a diagram, with gamma = 0.9 and alpha = 0.5:

┌─────────────────────────────────┐
│ Agent is in state s             │
└─────────────────────────────────┘
                 ↓
┌─────────────────────────────────┐
│ Pick action a                   │
└─────────────────────────────────┘
                 ↓
┌─────────────────────────────────┐
│ Get reward r, land in state s'  │
└─────────────────────────────────┘
                 ↓
┌─────────────────────────────────┐
│ target = r + 0.9 * max Q(s',a') │
└─────────────────────────────────┘
                 ↓
┌─────────────────────────────────┐
│ error = target - Q(s,a)         │
└─────────────────────────────────┘
                 ↓
┌─────────────────────────────────┐
│ Q(s,a) = Q(s,a) + 0.5 * error   │
└─────────────────────────────────┘
                 ↓
┌─────────────────────────────────┐
│ s' becomes s, repeat until goal │
└─────────────────────────────────┘

Here, we can see that one action gives one update. The agent repeats this loop again and again until it reaches the goal, and then starts a new attempt from the beginning. Each attempt from start to end is called an episode.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Our Tiny Corridor

The best way to learn this is by taking an example.

Consider a corridor of 4 cells. The agent starts in Cell 0. The goal is in Cell 3. Let's see this as a diagram:

┌────────┬────────┬────────┬────────┐
│ Cell 0 │ Cell 1 │ Cell 2 │ Cell 3 │
│ Start  │        │        │ Goal   │
│        │        │        │ +10    │
└────────┴────────┴────────┴────────┘

Here, we can see the 4 cells in a row, with the start on the left and the goal, worth a reward of 10, on the right. The rules are:

• The agent has two actions: Left and Right.
• Moving Right goes to the next cell. Moving Left goes to the previous cell. Moving Left in Cell 0 hits the wall, and the agent stays in Cell 0.
• Reaching Cell 3 gives a reward of 10, and the episode ends.
• Every other move gives a reward of 0.
• We use alpha = 0.5 and gamma = 0.9.

Cell 3 is the end, so the agent never takes an action there. Hence, our Q-table has 3 rows (Cell 0, Cell 1, Cell 2) and 2 columns (Left, Right). At the start, every cell is 0:

Q-Learning Step by Step

For the sake of understanding, let's say the agent moves Right every time in the first few episodes. Let's update the Q-table after each move.

Episode 1

Step 1: The agent is in Cell 0 and moves Right to Cell 1. The reward is 0. The best value in Cell 1 is 0.

• target = 0 + 0.9 * 0 = 0
• Q(Cell 0, Right) = 0 + 0.5 * (0 - 0) = 0

Nothing changes. The Q-table is still all zeros.

Step 2: The agent is in Cell 1 and moves Right to Cell 2. The reward is 0, and the best value in Cell 2 is 0. So, the target is 0, and Q(Cell 1, Right) stays 0. Nothing changes, because the agent has not found any reward yet.

Step 3: The agent is in Cell 2 and moves Right to Cell 3, the goal. The reward is 10. Cell 3 is the end, so its best value is 0.

• target = 10 + 0.9 * 0 = 10
• Q(Cell 2, Right) = 0 + 0.5 * (10 - 0) = 5

Here, we can see that the first useful number appears right next to the goal. The episode ends, and the agent starts again from Cell 0.

Episode 2

Step 4: The agent is in Cell 0 and moves Right to Cell 1. The reward is 0, and the best value in Cell 1 is still 0. So, Q(Cell 0, Right) stays 0, and the Q-table does not change.

Step 5: The agent is in Cell 1 and moves Right to Cell 2. The reward is 0. But now, the best value in Cell 2 is 5.

• target = 0 + 0.9 * 5 = 4.5
• Q(Cell 1, Right) = 0 + 0.5 * (4.5 - 0) = 2.25

Here, we can notice that the reward was 0 in this move, but Q(Cell 1, Right) still went up. This is because the agent now knows that Cell 2 is a good place. This is the Bellman idea at work.

Step 6: The agent is in Cell 2 and moves Right to the goal. The reward is 10, and the best value of the goal is 0.

• target = 10 + 0.9 * 0 = 10
• Q(Cell 2, Right) = 5 + 0.5 * (10 - 5) = 7.5

The guess moved from 5 to 7.5, halfway toward the target of 10, because alpha is 0.5.

Episode 3

Step 7: The agent is in Cell 0 and moves Right to Cell 1. The reward is 0, but the best value in Cell 1 is now 2.25.

• target = 0 + 0.9 * 2.25 = 2.025
• Q(Cell 0, Right) = 0 + 0.5 * (2.025 - 0) = 1.0125

If the agent finishes Episode 3 by moving Right two more times, the Right column after each episode looks like below:

Here, we can see that the reward at the goal flows backward, one cell at a time, until it reaches the start.

If we keep going for many episodes, and the agent also tries Left sometimes (we will see how in the next section), the values settle at:

Here, Right is higher than Left in every cell, so the agent has learned to always move Right. Using V(s) = max over a of Q(s, a), the state values are V(Cell 0) = 8.1, V(Cell 1) = 9, and V(Cell 2) = 10. The closer the cell is to the goal, the higher its value. Also, Q(Cell 0, Right) = 8.1, which is exactly the return we computed earlier for a reward of 10 after 3 moves.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Epsilon-Greedy

If the agent always picks the action with the highest Q, it can get stuck with the first good path it finds and never try others. So, we use epsilon-greedy: with a small probability epsilon (for example 0.2), the agent picks a random action to explore, and otherwise it picks the action with the highest Q.

Q-Learning in Python

Now, let's see the code for Q-Learning on our corridor.

import random

random.seed(0)
alpha, gamma, epsilon = 0.5, 0.9, 0.2
Q = [[0.0, 0.0] for _ in range(3)]  # 3 cells, 2 actions: 0 = Left, 1 = Right

for episode in range(200):
    s = 0
    while s != 3:
        if random.random() < epsilon or Q[s][0] == Q[s][1]:
            a = random.choice([0, 1])               # explore (or break a tie)
        else:
            a = 0 if Q[s][0] > Q[s][1] else 1       # exploit
        s_next = max(0, s - 1) if a == 0 else s + 1
        r = 10 if s_next == 3 else 0
        best_next = 0 if s_next == 3 else max(Q[s_next])
        Q[s][a] += alpha * (r + gamma * best_next - Q[s][a])
        s = s_next

for s in range(3):
    print(f"Cell {s}: Left = {Q[s][0]:.2f}, Right = {Q[s][1]:.2f}")

Here, we have:

• random.seed(0): makes the random choices the same on every run, so we get the same output.
• Q: the Q-table with 3 rows and 2 columns, all starting at 0.
• for episode in range(200): 200 episodes, each starting in Cell 0 and ending when the agent reaches Cell 3.
• The if block: epsilon-greedy. With probability 0.2, or when both actions have the same value, the agent picks a random action. Otherwise, it picks the action with the higher value.
• s_next and r: the corridor rules. Left in Cell 0 stays in Cell 0, and reaching Cell 3 gives 10.
• best_next: max Q(s', a'), which is 0 when the next state is the goal.
• The line Q[s][a] += ...: the Q-Learning update rule, exactly as we wrote it above.

It will print the following:

Cell 0: Left = 7.29, Right = 8.10
Cell 1: Left = 7.29, Right = 9.00
Cell 2: Left = 8.10, Right = 10.00

Here, we can see that the agent has learned the same values as our table above. It works perfectly.

If we want to go deep into Reinforcement Learning and Exploration vs Exploitation, our AI and Machine Learning Program at Outcome School covers it in depth.

Limitation of the Q-table and Deep Q-Network

The Q-table works well when there are only a few states and actions, like our corridor with 3 states and 2 actions. But in a video game, every screen is a different state, so there are far too many states to store in a table, and the agent never sees most of them even once.

So, here comes the Deep Q-Network (DQN) to the rescue. Instead of a table, DQN uses a neural network that takes the state as input and outputs one Q value for each action. It trains that network with the same Q-Learning target, so it can also give useful values for new states that look similar to the ones it has already seen.

Q-Learning and DQN are value-based methods. The other big family skips the values and learns the policy directly. We have a detailed blog on Policy Gradient and the REINFORCE Algorithm that explains it step by step.

Now we must have understood what a Value Function is, how Q-Learning learns it with the update rule, and how the Q-table fills up step by step.

Frequently Asked Questions

Why does Q-Learning learn Q instead of V?

Q-Learning learns Q because, with Q, the agent can choose the action directly: in each state, it picks the action with the highest Q. With only V, the agent would also need to know where each action leads. And if we know Q for every action, we also know V, because the value of a state is the value of its best action.

What does the learning rate alpha do in Q-Learning?

The learning rate alpha decides how big a step the agent takes toward the target in each update. It is a number between 0 and 1, for example 0.5. With alpha = 0.5, the old guess moves halfway toward the target, so a guess of 5 with a target of 10 becomes 7.5.

Why does a Q value go up even when the reward is 0?

A Q value can go up with a reward of 0 because the target also includes the discounted value of the next state. If the agent lands in a state that it already knows is good, the target becomes higher than the old guess. This is how the reward at the goal flows backward, one state at a time, until it reaches the start.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.