What is Exploration vs Exploitation in Reinforcement Learning?

Exploration vs Exploitation in Reinforcement Learning is the choice an agent faces at every step: use the action it already knows works well (exploitation), or try a different action to learn something new that can work even better (exploration).

In this blog, we will learn about exploration vs exploitation in reinforcement learning. We will also see what exploitation and exploration mean, why balancing them is a dilemma, the multi-armed bandit problem with real numbers, the common strategies from greedy and epsilon-greedy to UCB and softmax and how they compare, and how exploration shows up when we train LLMs with reinforcement learning.

We will cover the following:

• What is Exploitation?
• What is Exploration?
• The Exploration vs Exploitation Dilemma
• The Multi-Armed Bandit Problem
• Greedy Strategy
• Epsilon-Greedy Strategy
• Decaying Epsilon
• Optimistic Initial Values
• Upper Confidence Bound (UCB)
• Softmax Exploration with Temperature
• Comparing the Strategies
• Exploration in LLM Reinforcement Learning

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

In Reinforcement Learning, an agent learns by doing. It takes an action, gets a reward (a number that says how good that action was), and slowly learns which actions give more reward. The agent does not know the right answer in advance. It has to find it by trying.

So, at every step, the agent faces a question: repeat the best action it knows, or try something new? Here comes the exploration vs exploitation into the picture.

What is Exploitation?

Exploitation means picking the action that has given the best reward so far.

Let's say we have moved to a new city. In the first week, we tried 3 restaurants, and one of them served very good food. Now, whenever we feel hungry, we go back to that same restaurant. This is exploitation. We are using what we already know to get a good meal with low risk.

Exploitation gives us a good reward right now. But it has one problem. We never learn whether there is an even better restaurant just two streets away.

This was about exploitation. Now, it's time to learn about exploration.

What is Exploration?

Exploration means trying an action that is not the current best, so that we can learn more about it.

In the same city, there are 50 restaurants, and we have tried only 3 of them. If we try a new restaurant today, this is exploration. The meal can be bad. But it can also turn out to be the best food in the city, and then we will eat better for the rest of our stay.

Exploration costs us something today, and it pays us back later with better knowledge.

The Exploration vs Exploitation Dilemma

Now, the question is: how often should we try a new restaurant, and how often should we go back to our favorite one?

• If we only exploit, we get stuck with the first good option we found, which is often not the best one.
• If we only explore, we keep trying new places forever and never enjoy the best one we already found.

Both extremes collect less total reward. The agent must balance the two. This is called the exploration vs exploitation dilemma, and it is one of the central problems in reinforcement learning.

The best way to learn this is by taking an example.

The Multi-Armed Bandit Problem

The classic example used to study this dilemma is the multi-armed bandit problem.

A slot machine in a casino is sometimes called a "one-armed bandit", because it has one lever (arm) and it takes our money. A multi-armed bandit is a row of such machines, and at every step we choose which one to pull. It is the simplest form of reinforcement learning, because the situation never changes between pulls. Only our choice of machine matters. When the situation does change after every action, the problem is written down as a Markov Decision Process (MDP), and we have a detailed blog that explains it step by step.

Let's say we have 3 machines: A, B, and C. Each pull either pays 1 coin or pays 0 coins. Each machine has a hidden chance of paying:

We, as the agent, do not see this table. We have 1000 pulls, and our goal is to collect as many coins as possible.

If we knew the table, we would pull machine C every time and collect about 800 coins. But we do not know it. We must discover it by pulling.

For each machine, the agent keeps an estimate: the average reward it has seen from that machine so far. For example, if we pulled machine B 4 times and got 1, 0, 1, 1, the estimate for B is:

estimate(B) = (1 + 0 + 1 + 1) / 4 = 0.75

Now, let's see different strategies the agent can use to decide which machine to pull next.

Greedy Strategy

The simplest strategy is greedy: always pull the machine with the highest estimate.

At the start, all estimates are 0. Let's say ties go to the first machine, so the agent pulls A. Now, two things can happen:

• A pays 0 coins. The estimate of A stays at 0, all three machines are still tied, and the tie again goes to A.
• A pays 1 coin. The estimate of A goes above 0, while B and C are still at 0. So, A wins clearly.

Either way, the agent picks A again, and again, and again.

The agent never tries B or C even once. It ends up with about 200 coins instead of about 800.

This is the failure of pure exploitation. The greedy agent locks onto the first option that looked good and never learns that something better exists.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Epsilon-Greedy Strategy

So, here comes epsilon-greedy to the rescue.

Pick a small number called epsilon, for example 0.1. Then, at every step:

• With probability epsilon (10% of the time), explore: pull a random machine.
• With probability 1 - epsilon (90% of the time), exploit: pull the machine with the highest estimate.

Let's see this as a diagram:

     ┌────────────────────────────────┐
     │ Pick a random number r         │
     │ between 0 and 1                │
     └────────────────┬───────────────┘
                      ↓
     ┌────────────────────────────────┐
     │ Is r < epsilon (0.1)?          │
     └────┬───────────────────────┬───┘
          ↓                       ↓
         Yes                     No
          ↓                       ↓
┌───────────────────┐   ┌───────────────────┐
│ EXPLORE           │   │ EXPLOIT           │
│ Pull a random     │   │ Pull the machine  │
│ machine           │   │ with the highest  │
│ (10% of pulls)    │   │ estimate (90%)    │
└─────────┬─────────┘   └─────────┬─────────┘
          ↓                       ↓
┌───────────────────────────────────────────┐
│ Get reward: 1 coin or 0 coins             │
└─────────────────────┬─────────────────────┘
                      ↓
┌───────────────────────────────────────────┐
│ Update the estimate of that machine       │
└───────────────────────────────────────────┘

Here, we can see that 90% of the time the agent uses its best guess, and 10% of the time it tries a random machine. Those random tries are how it discovers that C is better than A.

Let's see the code for both greedy and epsilon-greedy as below:

import random

def run(epsilon, steps=1000, seed=42):
    random.seed(seed)
    win_chance = [0.2, 0.5, 0.8]   # hidden from the agent
    estimate = [0.0, 0.0, 0.0]     # what the agent believes
    count = [0, 0, 0]
    total = 0
    for _ in range(steps):
        if random.random() < epsilon:
            arm = random.randrange(3)              # explore
        else:
            arm = estimate.index(max(estimate))    # exploit
        reward = 1 if random.random() < win_chance[arm] else 0
        count[arm] += 1
        estimate[arm] += (reward - estimate[arm]) / count[arm]
        total += reward
    print(f"epsilon={epsilon}: total={total}, pulls={count}")

run(0.0)
run(0.1)

Here, we have:

• win_chance is the hidden table. The agent never reads it directly. It only sees the reward.
• estimate is the running average reward of each machine.
• If the random number is less than epsilon, we pull a random machine. Otherwise, we pull the machine with the highest estimate.
• The line estimate[arm] += (reward - estimate[arm]) / count[arm] updates the average without storing every past reward.
• run(0.0) is pure greedy, because epsilon is 0. run(0.1) is epsilon-greedy.

This will print the following:

epsilon=0.0: total=186, pulls=[1000, 0, 0]
epsilon=0.1: total=725, pulls=[96, 38, 866]

Here, we can see that the greedy agent pulled machine A all 1000 times and got only 186 coins. The epsilon-greedy agent explored enough to find machine C, pulled it 866 times, and got 725 coins. That is almost 4 times more coins.

But epsilon-greedy has a weakness. Even after the agent is sure that C is the best, it still spends 10% of its pulls on random choices forever, and most of those random choices land on A or B. Let's see how the next strategy solves this issue.

Decaying Epsilon

In the beginning, the agent knows nothing, so it must explore a lot. Later, it knows a lot, so it must explore very less.

Decaying epsilon follows exactly this idea. We start with a large epsilon and make it smaller after every step. For example:

epsilon = max(0.01, 1.0 * 0.99^step)

Let's see how epsilon changes over time:

Here, the max(0.01, ...) part keeps a small amount of exploration alive, so that the agent can still notice if the hidden chances of the machines change later.

This is how decaying epsilon explores a lot at the start, when exploration is most useful, and exploits a lot at the end, when the agent already knows the best action.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Optimistic Initial Values

There is another simple trick that does not use any randomness at all.

Instead of starting every estimate at 0, we start every estimate at a high value, for example 1.0, which is the best reward possible here. It is as if the agent believes "every machine pays every time" until it is proven wrong.

Now, let's use the plain greedy strategy with these optimistic starting values:

• The agent pulls A. A pays 0. The estimate of A drops below 1.0.
• Now B and C still look perfect at 1.0, so the greedy agent tries them.
• Every machine gets tried, and every pull that pays 0 lowers that machine's estimate, until the agent settles on the one that keeps paying, which is usually C.

Means, the optimism itself pushes the agent to explore.

Note: This works well at the start, but it is not a full solution. The early exploration is a one-time push. In our example, it finds C in most runs, but if the agent gets unlucky early, it can still settle on B. And if the hidden chances of the machines change later, this trick does not explore again.

Upper Confidence Bound (UCB)

Epsilon-greedy explores randomly. It can waste a pull on machine A even when we have already pulled A hundreds of times and know it is bad.

Upper Confidence Bound (UCB) explores in a more targeted way. It prefers the machines that we are unsure about.

UCB gives each machine a score, and the agent pulls the machine with the highest score:

score = average_reward + sqrt(2 * ln(t) / n)

Here, in this formula:

• average_reward is the estimate of that machine.
• t is the total number of pulls so far, across all machines.
• n is the number of times this particular machine was pulled.
• ln is the natural logarithm, which grows very slowly as t grows.

The second part is the exploration bonus. It is big when n is small, which means we have pulled this machine only a few times and we are not sure about it. It shrinks as we pull the machine more.

Let's take an example after 20 total pulls (t = 20, so ln(20) is about 3.0):

Here, we can notice that machine C has the best average, but UCB picks machine B. Why? Because B was pulled only 2 times. Its average of 0.50 is based on very little data, and its true value can be much higher. UCB gives B a chance to prove itself.

If B is really worse than C, more pulls bring its average close to its true value of 0.5, its bonus shrinks, and UCB moves back to C. And because ln(t) keeps growing slowly, a machine that has not been pulled for a long time slowly gains bonus again and gets one more check later.

This is the idea behind UCB: when we are unsure about an action, we assume it can be good and give it a chance.

If we want to go deep into Reinforcement Learning and Exploration vs Exploitation, check out our AI and Machine Learning Program at Outcome School.

Softmax Exploration with Temperature

Epsilon-greedy treats all non-best machines the same. When it explores, a bad machine and a nearly-best machine have an equal chance.

Softmax exploration fixes this. It turns the estimates into probabilities, so that better machines get picked more often, but every machine still has some chance. Softmax is a function that converts a list of numbers into probabilities that add up to 1, where a bigger number gets a bigger probability.

It also uses a number called temperature (T):

probability(machine) = exp(estimate / T) / sum of exp(estimate / T) over all machines

Let's say our estimates are A = 0.2, B = 0.5, and C = 0.8.

Here, at T = 0.1, the chance of A is about 0.002, so it shows as 0.00. We can see that:

• A high temperature makes the probabilities close to each other, so the agent explores more.
• A low temperature makes the best machine dominate, so the agent exploits more.
• Even when exploring, the agent picks B more often than A, because B looks better than A.

Just like epsilon, the temperature can also start high and go down over time.

Comparing the Strategies

Let me tabulate the differences between these strategies for your better understanding.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Exploration in LLM Reinforcement Learning

The same dilemma appears when we train an LLM (Large Language Model) with reinforcement learning.

Here, the agent is the LLM. The action is the next token it generates, and a full answer is a long chain of such actions. The reward comes at the end, for example from a reward model or from checking whether a math answer is correct.

In LLM training, exploration mainly comes from two things.

Sampling temperature: The LLM gives a score (called a logit) for every possible next token, and these scores are turned into probabilities using softmax with a temperature, exactly like the softmax exploration we saw above.

• With a temperature close to 0, the LLM almost always picks the most likely token. This is exploitation. Every answer to the same prompt looks almost the same, so the training has nothing new to learn from.
• With a temperature around 1.0, the LLM sometimes picks less likely tokens. This is exploration. It produces different answers, and some of them can be better than its usual answer.

Multiple samples per prompt: During training, we often generate many answers for the same prompt, for example 8 or 16 answers. Each answer is scored with a reward. Then, in a common setup, the training increases the probability of the answers that scored above the average of the group and decreases the probability of the ones that scored below it. We have a detailed blog on Group Relative Policy Optimization (GRPO) that explains this group-based training in depth.

Let's say we ask the LLM a math question and sample 8 answers. 6 are wrong, and 2 are correct. Let's see this as a diagram:

┌─────────────────────────────────────┐
│ Prompt: "What is 17 * 24?"          │
└──────────────────┬──────────────────┘
                   ↓
┌─────────────────────────────────────┐
│ Sample 8 answers                    │
│ (temperature = 1.0)                 │
└──────────────────┬──────────────────┘
                   ↓
┌─────────────────────────────────────┐
│ Reward: 2 correct (1 each),         │
│ 6 wrong (0 each)                    │
│ Group average = 2 / 8 = 0.25        │
└──────────────────┬──────────────────┘
                   ↓
┌─────────────────────────────────────┐
│ Raise probability of 2 correct      │
│ Lower probability of 6 wrong        │
└─────────────────────────────────────┘

Here, we can see that the 2 correct answers scored 1, which is above the group average of 0.25, so they become more likely. The 6 wrong answers scored 0, which is below the average, so they become less likely.

Now, think about what happens without exploration. If we had sampled only one answer with temperature 0, and that answer was wrong, we would get the same wrong answer every time. The model would never see that a correct path exists. By exploring 8 different answers, the model finds the 2 correct paths and learns to make them more likely.

The balance still matters here. If the temperature is too low, the answers are almost the same and nothing new is learned. If the temperature is too high, the answers become close to random text and often get no reward at all. Over training, the model also tends to become more confident and its answers become less varied, so keeping enough exploration is an active concern when training LLMs with reinforcement learning.

To master Reward Models and GRPO for training LLMs from the ground up, explore our AI and Machine Learning Program at Outcome School.

Now we must have understood what exploration vs exploitation in reinforcement learning is, why it is a dilemma, the strategies we use to balance it, and how it appears when we train LLMs.

Frequently Asked Questions

Is the multi-armed bandit problem a reinforcement learning problem?

Yes. The multi-armed bandit problem is the simplest form of reinforcement learning. The agent picks an action, gets a reward, and learns which action pays the most. It is simple because the situation never changes between pulls, so only the choice of machine matters.

Why not stop exploring once the agent finds the best action?

Because the hidden chances of the actions can change later. With decaying epsilon, we keep a small floor, for example 0.01, instead of going all the way to 0. This small amount of exploration lets the agent still notice such a change. Optimistic initial values do not do this, as they explore only at the start.

What is the difference between epsilon-greedy and UCB?

Epsilon-greedy explores randomly, so it can waste a pull on an action that it already knows is bad. UCB explores in a targeted way. It adds an exploration bonus to each action, and this bonus is big for actions tried only a few times. So, UCB prefers the actions it is unsure about.

What happens if the sampling temperature is too low or too high in LLM reinforcement learning?

Both extremes hurt the training. If the temperature is too low, the sampled answers are almost the same, so nothing new is learned. If the temperature is too high, the answers become close to random text and often get no reward at all. Around 1.0, the LLM produces different answers, and some of them can be better than its usual answer.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.