What are Optimizers in Deep Learning? SGD, Momentum, RMSProp, and Adam Explained

An optimizer is the part of deep learning training that decides how to change the weights of a neural network after every training step, so that the network makes fewer mistakes. SGD, Momentum, RMSProp, and Adam are four popular optimizers, and each one improves on the one before it.

In this blog, we will learn about optimizers in deep learning. We will also see what a gradient and a learning rate are, how SGD updates the weights and where it struggles, how Momentum and RMSProp fix its problems one by one, how Adam combines both ideas, what AdamW changes, and which optimizer to use when.

We will cover the following:

• What is an Optimizer?
• Gradient and Learning Rate
• SGD (Stochastic Gradient Descent)
• Momentum
• RMSProp
• Adam
• Adam in Code
• AdamW
• Comparison of SGD, Momentum, RMSProp, and Adam

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is an Optimizer?

A neural network is made of many numbers called weights. The weights decide what the network outputs for a given input. At the start of training, the weights are random, so the outputs are wrong.

During training, we measure how wrong the network is with a single number called the loss. A big loss means many mistakes. A small loss means few mistakes. Our goal is to make the loss as small as possible.

An optimizer is the algorithm that uses the slope of the loss, called the gradient, to update the weights step by step, so that the loss keeps going down.

The best way to learn this is by taking an example.

Let's say we are standing on a hill in thick fog, and we want to reach the lowest point of the valley. We cannot see the valley. We can only feel the slope of the ground under our feet. So, we take a small step in the downhill direction, feel the slope again, and take another step. We keep doing this until the ground becomes flat.

Here, our position on the hill is the weights, the height is the loss, and the rule we follow to take each step is the optimizer.

Let's see where the optimizer sits in the training loop as a diagram:

┌────────────────────────────────────┐
│ Batch of 32 training examples      │
└────────────────────────────────────┘
                   ↓
┌────────────────────────────────────┐
│ Forward pass                       │
│ Network makes 32 predictions       │
└────────────────────────────────────┘
                   ↓
┌────────────────────────────────────┐
│ Loss                               │
│ One number: how wrong it is        │
└────────────────────────────────────┘
                   ↓
┌────────────────────────────────────┐
│ Backward pass                      │
│ One gradient for every weight      │
└────────────────────────────────────┘
                   ↓
┌────────────────────────────────────┐
│ Optimizer                          │
│ Uses the gradients to update       │
│ every weight                       │
└────────────────────────────────────┘
                   ↓
┌────────────────────────────────────┐
│ Updated weights                    │
│ Used for the next batch            │
└────────────────────────────────────┘

Here, we can see that the optimizer is the last stage of every training step. It does not compute the loss and it does not compute the gradient. It takes the gradient and decides how much to change each weight. Then the loop repeats with the updated weights.

We have a detailed blog on backpropagation that explains how the backward pass computes these gradients step by step.

Gradient and Learning Rate

Before jumping into the optimizers, we must know two terms: gradient and learning rate.

The gradient of a weight tells us two things:

• Direction: If we increase this weight a little, does the loss go up or down?
• Size: How strongly does the loss change when we change this weight?

If the gradient is positive, increasing the weight increases the loss. So, we must decrease the weight. If the gradient is negative, we must increase the weight. This is why every optimizer moves the weight in the opposite direction of the gradient.

The learning rate is a small number, like 0.1 or 0.001, that controls how big each step is. We choose it before training starts.

• If the learning rate is too big, we jump over the valley and the loss bounces around.
• If the learning rate is too small, we move so slowly that training takes a very long time.

For all the examples in this blog, we will use one simple loss for the sake of understanding:

loss = w^2

The lowest loss is at w = 0. The gradient of this loss is 2 * w. We will start at w = 2 and use a learning rate of 0.1.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

SGD (Stochastic Gradient Descent)

SGD is the simplest optimizer. It takes a step in the opposite direction of the gradient, scaled by the learning rate.

w_new = w - learning_rate * gradient

The word Stochastic means random. In SGD, we do not compute the gradient on the whole dataset at once. We compute it on a small random batch of examples, which is much faster. Hence, the name Stochastic Gradient Descent.

Let's do two steps with numbers.

Step 1: At w = 2, the gradient is 2 * 2 = 4. So, w_new = 2 - 0.1 * 4 = 1.6.

Step 2: At w = 1.6, the gradient is 2 * 1.6 = 3.2. So, w_new = 1.6 - 0.1 * 3.2 = 1.28.

The weight is moving toward 0, and the loss is going down.

But the following issues arise in real networks:

• Zig-zag: Real loss surfaces are often shaped like a long narrow valley. The slope across the valley is steep, and the slope along the valley is gentle. SGD keeps jumping from one wall of the valley to the other, and makes very little progress along the valley floor.
• Slow in flat areas: When the ground is almost flat, the gradient is tiny. So, the step is tiny, and SGD moves very slowly.

The issue with this approach is that every step only looks at the current gradient and forgets everything before it. Let's see how the next approach solve this issue.

Momentum

Momentum adds memory to SGD. Instead of stepping using only the current gradient, it keeps a running value called the velocity, which collects the gradients from the past steps.

Think of a ball rolling down the hill. The ball does not stop and restart at every step. It builds up speed in the direction it keeps going.

The update rule is as below:

velocity = beta * velocity + gradient

w_new = w - learning_rate * velocity

Here, beta is a number like 0.9. It decides how much of the old velocity we keep. With beta = 0.9, we keep 90 percent of the old velocity and add the new gradient on top.

Let's do the same two steps with numbers. The velocity starts at 0.

Step 1: The gradient is 4. So, velocity = 0.9 * 0 + 4 = 4 and w_new = 2 - 0.1 * 4 = 1.6.

Step 2: The gradient is 3.2. So, velocity = 0.9 * 4 + 3.2 = 6.8 and w_new = 1.6 - 0.1 * 6.8 = 0.92.

Here, we can notice that after two steps, SGD reached 1.28, but Momentum reached 0.92. Momentum is closer to the lowest point at 0, because the velocity has built up speed.

This fixes both SGD problems:

• Zig-zag: The side-to-side gradients point in opposite directions in alternate steps, so they mostly cancel out inside the velocity. The gradients along the valley point the same way every step, so they add up.
• Flat areas: Even when the current gradient is tiny, the velocity still carries speed from the earlier steps.

But the following issue arises. A network has millions of weights. Some weights always get large gradients, and some always get tiny gradients. SGD and Momentum use the same learning rate for all of them. A learning rate that is safe for the weights with large gradients is far too small for the weights with tiny gradients, so those weights barely learn.

The issue with this approach is that every weight gets the same step size. Let's see how the next approach solve this issue.

RMSProp

RMSProp gives every weight its own step size. It divides each gradient by the typical size of that weight's recent gradients.

To know the typical size, RMSProp keeps a running average of the squared gradient for each weight. We square the gradient so that we measure only its size, and negative and positive values do not cancel out.

The update rule is as below:

s = beta * s + (1 - beta) * gradient^2

w_new = w - learning_rate * gradient / (sqrt(s) + epsilon)

Here:

• s is the running average of the squared gradients for this weight. It starts at 0.
• beta is usually 0.9.
• sqrt(s) is the typical size of the gradient for this weight.
• epsilon is a tiny number like 0.00000001, added only to avoid dividing by zero.

RMSProp stands for Root Mean Square Propagation, because sqrt(s) is the root of the mean of the squares.

Let's take two weights, A and B, with very different gradients. Weight A has a gradient of 4, and weight B has a gradient of 0.04. We use a learning rate of 0.1.

Weight A: s = 0.9 * 0 + 0.1 * 16 = 1.6 and sqrt(1.6) = 1.2649. So, the step is 0.1 * 4 / 1.2649 = 0.3162.

Weight B: s = 0.9 * 0 + 0.1 * 0.0016 = 0.00016 and sqrt(0.00016) = 0.012649. So, the step is 0.1 * 0.04 / 0.012649 = 0.3162.

Here, we can see that both weights take the same step of 0.3162, even though their gradients differ by 100 times. With plain SGD, weight A would move by 0.4 and weight B would move by only 0.004.

Means, a weight with large gradients gets its steps shrunk, and a weight with tiny gradients gets its steps boosted.

But RMSProp has no velocity. It does not remember the direction of the past gradients, so it loses the benefits of Momentum.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Adam

So, here comes Adam to the rescue. Adam stands for Adaptive Moment Estimation. It combines the two ideas we just learned:

• From Momentum, it takes a running average of the gradients. This gives the direction and the speed.
• From RMSProp, it takes a running average of the squared gradients. This gives a separate step size for each weight.

The update rule is as below:

m = beta1 * m + (1 - beta1) * gradient

v = beta2 * v + (1 - beta2) * gradient^2

m_hat = m / (1 - beta1^t)

v_hat = v / (1 - beta2^t)

w_new = w - learning_rate * m_hat / (sqrt(v_hat) + epsilon)

Here:

• m is the Momentum part. beta1 is usually 0.9.
• v is the RMSProp part. beta2 is usually 0.999.
• t is the step number: 1, 2, 3, and so on.
• m_hat and v_hat are the bias-corrected versions of m and v.

Now, the question is: why do we need m_hat and v_hat?

The answer is that m and v start at 0. In the first few steps, they are pulled toward 0 and are much smaller than the real gradients. Dividing by (1 - beta1^t) and (1 - beta2^t) fixes this. In the first step, these divisions make a big correction. As t grows, beta1^t and beta2^t go close to 0, so the division stops changing anything.

Let's do one step with numbers. The gradient is 4, t = 1, and the learning rate is 0.1.

Momentum part: m = 0.9 * 0 + 0.1 * 4 = 0.4.

RMSProp part: v = 0.999 * 0 + 0.001 * 16 = 0.016.

Bias correction: m_hat = 0.4 / (1 - 0.9) = 4 and v_hat = 0.016 / (1 - 0.999) = 16.

Update: w_new = 2 - 0.1 * 4 / sqrt(16) = 2 - 0.1 * 4 / 4 = 1.9.

Here, we can notice that without bias correction, m would be only 0.4 instead of the real gradient 4. Bias correction brings it back to 4.

Let's see how Adam combines the two ideas in this step as a diagram:

┌────────────────────────────────────────┐
│ gradient = 4 (goes to both parts)      │
└────────────────────────────────────────┘
          ↓                     ↓
┌──────────────────┐  ┌──────────────────┐
│ Momentum part    │  │ RMSProp part     │
│ m_hat = 4        │  │ v_hat = 16       │
│ gives direction  │  │ gives scale      │
└──────────────────┘  └──────────────────┘
          ↓                     ↓
┌────────────────────────────────────────┐
│ step = 0.1 * 4 / sqrt(16) = 0.1        │
│ w_new = 2 - 0.1 = 1.9                  │
└────────────────────────────────────────┘

Here, we can see that the same gradient feeds two running averages. The Momentum part gives the direction, and the RMSProp part gives the scale. Adam divides the first by the square root of the second to get the final step for this weight.

To master Optimizers and Gradient Descent, and build a Neural Network from scratch, we have a complete program - check out our AI and Machine Learning Program at Outcome School.

Adam in Code

Let's see the code for Adam with two weights. One weight gets a big gradient, and the other gets a gradient that is 100 times smaller.

import numpy as np

w = np.array([2.0, 2.0])          # two weights
m = np.zeros(2)                   # running average of gradients
v = np.zeros(2)                   # running average of squared gradients
lr, beta1, beta2, eps = 0.1, 0.9, 0.999, 1e-8

for t in range(1, 4):
    g = np.array([2.0, 0.02]) * w  # gradients: one big, one tiny
    m = beta1 * m + (1 - beta1) * g
    v = beta2 * v + (1 - beta2) * g * g
    m_hat = m / (1 - beta1 ** t)
    v_hat = v / (1 - beta2 ** t)
    w = w - lr * m_hat / (np.sqrt(v_hat) + eps)
    print(t, np.round(w, 4))

Here, both weights start at 2.0, but the gradient of the first weight is 100 times bigger than the gradient of the second weight. The five lines inside the loop are the five lines of the Adam update rule.

This will print the following:

1 [1.9 1.9]
2 [1.8002 1.8002]
3 [1.7006 1.7006]

Here, we can see that both weights move by almost the same amount at every step, even though their gradients are very different. This is the RMSProp part of Adam at work. With plain SGD, the second weight would have moved 100 times slower.

Note: We used a learning rate of 0.1 to keep the numbers easy to read. In real training, Adam often uses a much smaller learning rate, like 0.001 or 0.0003.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

AdamW

AdamW is a small change to Adam, and it is a common choice for training LLMs (Large Language Models).

To stop the weights from growing too large, we often shrink every weight a little at each step. This is called weight decay. In plain Adam, weight decay is usually added to the gradient, so it also gets divided by sqrt(v_hat), and the shrinking becomes uneven across weights.

AdamW keeps the weight decay separate from the gradient. It does the normal Adam step, and then shrinks every weight directly:

w_new = w - learning_rate * m_hat / (sqrt(v_hat) + epsilon) - learning_rate * weight_decay * w

Here, the last part shrinks the weight by the same fraction for every weight, no matter how big its gradients are. This change makes the weight decay work as intended, and the models often generalize better.

Comparison of SGD, Momentum, RMSProp, and Adam

Let me tabulate the differences between SGD, Momentum, RMSProp, and Adam for your better understanding.

For most deep learning projects today, Adam or AdamW is the default starting point. SGD with Momentum is still used for some image models, where it often generalizes well when the learning rate is tuned carefully.

Now we must have understood what optimizers are, how SGD, Momentum, RMSProp, and Adam update the weights, and when to use which one.

Frequently Asked Questions

Does the optimizer compute the gradient?

No. The optimizer does not compute the loss or the gradient. The backward pass computes one gradient for every weight, and the optimizer takes those gradients and decides how much to change each weight. It is the last stage of every training step, and then the loop repeats with the updated weights.

Which optimizer should we use for deep learning?

For most deep learning projects today, Adam or AdamW is the default starting point. AdamW is a common choice for training LLMs. SGD with Momentum is still used for some image models, where it often generalizes well when the learning rate is tuned carefully.

Why does Adam need bias correction?

Adam needs bias correction because its running averages m and v start at 0. In the first few steps, they are pulled toward 0 and are much smaller than the real gradients. Dividing by (1 - beta1^t) and (1 - beta2^t) fixes this, and as the step number grows, the correction stops changing anything.

What learning rate does Adam use in real training?

In real training, Adam often uses a small learning rate, like 0.001 or 0.0003. The learning rate is chosen before training starts. If it is too big, the loss bounces around, and if it is too small, training takes a very long time.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.