Why do vanishing and exploding gradients happen?

Vanishing and exploding gradients happen because, during training, the learning signal of a neural network gets multiplied by many numbers, one for each layer, on its way back. If those numbers are mostly smaller than 1, the signal shrinks to almost nothing, and if they are mostly bigger than 1, the signal blows up to a huge value.

In this blog, we will learn about why vanishing and exploding gradients happen. We will also see how the gradient travels backward through the layers, why repeated multiplication is the root cause, what makes it worse, what symptoms we notice during training, and how we fix each problem.

We will cover the following:

• What is a gradient?
• How does the gradient travel backward?
• The root cause: repeated multiplication
• What is the vanishing gradient problem?
• What is the exploding gradient problem?
• What makes the numbers smaller or bigger than 1?
• Why do RNNs suffer the most?
• Seeing it in code
• How do we know it is happening?
• How do we fix vanishing gradients?
• How do we fix exploding gradients?
• Comparison of vanishing and exploding gradients

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is a gradient?

Before jumping into vanishing and exploding gradients, we must know what a gradient is.

A neural network is made of many layers. The connections inside it have numbers called weights, which decide how much each input matters.

Training a network means adjusting its weights again and again until its answers become correct. At every training step, the network makes a guess, and we measure how wrong the guess is. This measure of wrongness is a single number called the loss.

A gradient tells each weight in which direction to move, and how strongly, to reduce the loss.

Let's take a real-world comparison. Imagine we are standing on a hill in thick fog, and we want to reach the lowest point of the valley. We cannot see anything, but we can feel the slope under our feet. The slope tells us which way is downhill and how steep it is. The gradient is that slope.

Then, each weight is updated as below:

new_weight = old_weight - learning_rate x gradient

Here, the learning rate is a small number that controls the step size. The gradient decides the direction and the strength of the change.

So, the gradient is the learning signal. If the gradient is healthy, the weight learns. If the gradient is almost 0, the weight barely changes. If the gradient is huge, the weight jumps wildly.

We have a detailed blog on How Does Gradient Descent Work? that explains how this update rule reduces the loss step by step.

How does the gradient travel backward?

A network has many layers stacked one after another. Let's say we have 4 layers.

 Forward pass (making a guess):

 input --> Layer 1 --> Layer 2 --> Layer 3 --> Layer 4 --> guess --> loss


 Backward pass (sending the learning signal):

 Layer 1 <-- Layer 2 <-- Layer 3 <-- Layer 4 <-- loss

Here, we can see two directions:

• In the forward pass, the input goes through every layer, and the network makes a guess. Then, we compute the loss.
• In the backward pass, the learning signal starts from the loss and travels backward, from the last layer to the first. This backward journey is called backpropagation.

Now, the important question is, how does Layer 1 find out how much it contributed to the mistake? Layer 1 does not touch the loss directly. Its output goes to Layer 2, which goes to Layer 3, which goes to Layer 4, which produces the guess.

So, Layer 1 must receive the signal through all the layers after it. At every layer, the signal gets multiplied by a local slope. The local slope of a layer tells how much the output of that layer changes when its input changes a little.

This rule of multiplying the local slopes along the path is called the chain rule.

gradient at Layer 1 = slope4 x slope3 x slope2 x slope1 x (signal from loss)

Here, we can see that the gradient reaching Layer 1 is the result of multiplying the slopes of all the layers along the way.

Let's take a real-world comparison. Think of a chain of gears. When we turn the last gear, the turn passes through every gear to reach the first one. If each gear slows the turn down a little, the first gear barely moves. If each gear speeds the turn up a little, the first gear spins out of control. The gradient travels through the layers in exactly this way.

To learn Backpropagation, Gradient Descent, and how to build a Neural Network from scratch, check out our AI and Machine Learning Program at Outcome School.

The root cause: repeated multiplication

When we multiply many numbers together, a small difference from 1 grows into a huge effect.

Let's see what happens when we multiply the same number again and again:

Here, we can see three cases:

• Numbers below 1 shrink towards 0, and they shrink faster with more layers.
• The number 1 stays exactly the same, no matter how many times we multiply.
• Numbers above 1 grow to giant values.

Notice that 0.5 and 1.5 are not extreme numbers at all. They are close to 1. But after 50 multiplications, one becomes almost nothing, and the other becomes hundreds of millions.

This is why vanishing and exploding gradients happen. The gradient reaching an early layer is a product of many local slopes. If they are mostly below 1, the product vanishes. If they are mostly above 1, the product explodes.

What is the vanishing gradient problem?

The vanishing gradient problem is when the gradient becomes so small by the time it reaches the early layers that those layers almost stop learning.

Let's say we have a 20-layer network, and every layer multiplies the signal by 0.5 on the way back. Then, the gradient reaching Layer 1 is about one millionth of the gradient at the last layer.

Means, the last layers learn normally, but the early layers barely change. They stay close to their random starting values.

Coming back to the gear chain, the last gear turns, but the first gear barely moves.

This is a big problem because the early layers do important work. In an image model, the early layers learn simple things like edges and corners. The later layers build on top of them. If the early layers do not learn, the later layers have poor inputs to build on.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

What is the exploding gradient problem?

The exploding gradient problem is when the gradient becomes so large on its way back that the weight updates become huge and the training breaks.

Let's say every layer multiplies the signal by 1.5 on the way back. With 50 layers, the gradient reaching Layer 1 is hundreds of millions of times bigger than at the last layer.

Remember the update rule: new_weight = old_weight - learning_rate x gradient. When the gradient is huge, the step is huge. The weight jumps to a very bad value. The next guess becomes terrible, the loss shoots up, and the gradients become even bigger. Very soon, the numbers become too big for the computer to store, and they show up as NaN, which means "Not a Number". At this point, the training fails.

Now, let's understand what makes those local slopes smaller or bigger than 1.

What makes the numbers smaller or bigger than 1?

Each local slope depends on two things: the activation function and the weights. On top of that, the depth of the network decides how many of them get multiplied.

Cause 1: The activation function.

An activation function is a small rule inside each neuron that bends its output. Every activation function has its own slope, and that slope gets multiplied into the gradient.

Let's take Sigmoid, which was very popular in early networks. It squeezes any number into the range 0 to 1, using an S-shaped curve.

output
  1 |                ________
    |              /
0.5 |            /
    |          /
  0 |________/
    +------------0----------- input

Here, we can see that the steepest point of Sigmoid has a slope of only 0.25. At the flat ends, the slope is almost 0. So, even in the best case, every Sigmoid layer multiplies the gradient by 0.25 or less. After 10 layers, that is less than one millionth. This is the classic cause of the vanishing gradient problem.

Tanh, another S-shaped activation function, has a steepest slope of 1, which is better, but its flat ends still have a slope close to 0.

Cause 2: The weights.

The gradient also gets multiplied by the weights of each layer on its way back. If the weights are too small, they shrink the gradient. If the weights are too big, they grow it. This is why the starting values of the weights, called weight initialization, matter so much.

Cause 3: The depth.

More layers means more multiplications. A 3-layer network with a slope of 0.5 per layer only shrinks the gradient to 0.125. A 50-layer network with the same slope shrinks it to almost 0. So, the deeper the network, the stronger both problems become.

Why do RNNs suffer the most?

An RNN, which stands for Recurrent Neural Network, is a network that reads a sequence, like a sentence, one word at a time. It keeps a memory and updates it after every word, using the same weights again and again.

Let's say the RNN reads a sentence of 100 words. To learn how the first word affects the final answer, the gradient must travel back through all 100 steps. At every step, it gets multiplied by the same weights.

This is like a network with 100 layers that all share the same weights. Multiplying by the same number 100 times makes vanishing or exploding very likely.

This is why plain RNNs forget the beginning of long sentences. The learning signal from the end of the sentence vanishes before it reaches the beginning. This is the problem that LSTMs were introduced to solve. We will see them in the fixes below.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Seeing it in code

The best way to learn this is by taking an example. Let's simulate a gradient traveling backward through many layers, where each layer multiplies it by the same local slope.

def gradient_at_first_layer(num_layers, slope_per_layer):
    gradient = 1.0  # signal at the last layer
    for _ in range(num_layers):
        gradient = gradient * slope_per_layer
    return gradient

for slope in [0.25, 0.5, 1.0, 1.5]:
    for layers in [10, 50]:
        g = gradient_at_first_layer(layers, slope)
        print(f"slope {slope}, {layers} layers -> gradient {g:.3g}")

Here, we have:

• gradient starts at 1.0, which is the learning signal at the last layer.
• In each loop, we multiply it by slope_per_layer, just like the chain rule does at every layer.
• We try four slopes: 0.25 (the best case of Sigmoid), 0.5, 1.0, and 1.5, for 10 and 50 layers.

This will print the following:

slope 0.25, 10 layers -> gradient 9.54e-07
slope 0.25, 50 layers -> gradient 7.89e-31
slope 0.5, 10 layers -> gradient 0.000977
slope 0.5, 50 layers -> gradient 8.88e-16
slope 1.0, 10 layers -> gradient 1
slope 1.0, 50 layers -> gradient 1
slope 1.5, 10 layers -> gradient 57.7
slope 1.5, 50 layers -> gradient 6.38e+08

Here, e-07 means "divide by 10 seven times", and e+08 means "multiply by 10 eight times". So, 7.89e-31 is a number with 30 zeros after the decimal point, and 6.38e+08 is about 638 million.

So, we can notice that only the slope of 1.0 keeps the gradient healthy, while everything below 1 vanishes and everything above 1 explodes.

How do we know it is happening?

Now, let's see the symptoms during training.

Signs of vanishing gradients:

• The loss goes down very slowly, or stops going down early.
• The weights of the early layers barely change from their starting values.
• The gradients of the early layers are tiny compared to the later layers.

Signs of exploding gradients:

• The loss jumps up and down wildly, or suddenly shoots up.
• The loss or the weights become NaN.
• The gradients become very large numbers.

A simple habit is to print the size of the gradients of each layer during training. This makes the problem visible.

How do we fix vanishing gradients?

Now that we have learned why the problems happen, it's time to learn the fixes. Each fix attacks one cause.

Fix 1: Use ReLU instead of Sigmoid.

ReLU is an activation function that keeps positive numbers as they are and turns negative numbers into 0. For positive numbers, its slope is exactly 1. As we saw, multiplying by 1 again and again keeps the gradient healthy. This change made deep networks much easier to train.

Fix 2: Use good weight initialization.

Methods like Xavier initialization and He initialization pick the starting weights at the right size, based on how many inputs each neuron has. This keeps the signal size close to steady through the layers from the very first step.

Fix 3: Use residual connections.

A residual connection is a shortcut that adds the input of a block directly to its output, skipping over the block.

           +---------------------------+
           |        (shortcut)         |
           |                           ↓
 input ----+--> [ layers of block ] --(+)--> output

Here, we can see that the input has two paths: one through the block, and one straight across the shortcut. On the way back, the gradient also gets two paths. Even if the path through the block shrinks the gradient, the shortcut carries it back almost untouched.

Let's take a real-world comparison. In a city with only narrow lanes, traffic slows down at every turn. A highway lets cars skip all of those turns. Residual connections are highways for the gradient.

Residual connections were made famous by ResNet in 2015, which trained image networks with more than 100 layers. Transformers, which are used in LLMs (Large Language Models), also use them.

Fix 4: Use normalization.

Normalization is a step that rescales the numbers inside the network so that they keep a steady size, with an average near 0 and a typical spread near 1.

Batch Normalization and Layer Normalization are the two popular types. By keeping the numbers at a steady size at every layer, they help stop the signal from drifting too small or too big. Transformers use Layer Normalization.

Fix 5: Use LSTM or GRU instead of a plain RNN.

An LSTM (Long Short-Term Memory) and a GRU (Gated Recurrent Unit) are special kinds of RNNs. They carry their memory forward mostly by adding new information to it, controlled by small switches called gates, instead of multiplying the whole memory again and again. Addition does not shrink the signal the way repeated multiplication does. So, the gradient can travel back across many more words.

If we want to go deep into Activation Functions, Layer Normalization and Batch Normalization, and the rest of Deep Learning, we have a complete program - check out our AI and Machine Learning Program at Outcome School.

How do we fix exploding gradients?

Good initialization, normalization, and residual connections also help here. But there is one popular fix made for exploding gradients.

Gradient clipping is a technique that limits the size of the gradient, so that it can never go above a chosen maximum.

Let's take a real-world comparison. A car can have a speed limiter. No matter how hard we press the pedal, the car never goes above, say, 100 km per hour. Gradient clipping is a speed limiter for the gradient.

The most common type is clipping by norm. The norm is the overall size of all the gradients together, measured like the length of an arrow. If the norm is bigger than the maximum, we shrink the whole gradient down to the maximum, but keep its direction the same.

Let's see the code for it as below:

import numpy as np

def clip_by_norm(gradient, max_norm):
    norm = np.linalg.norm(gradient)       # overall size of the gradient
    if norm > max_norm:
        gradient = gradient * (max_norm / norm)
    return gradient

g = np.array([300.0, 400.0])               # an exploding gradient
print(np.linalg.norm(g))                   # size before clipping
print(clip_by_norm(g, max_norm=1.0))       # after clipping

Here, we have:

• np.linalg.norm(gradient) computes the overall size. For [300, 400], the size is 500, just like the long side of a right-angled triangle with sides 300 and 400.
• If the size is above max_norm, we multiply the whole gradient by max_norm / norm. This shrinks it to the allowed size.
• The direction stays the same, because both numbers shrink by the same factor.

This will print the following:

500.0
[0.6 0.8]

Here, we can see that the gradient shrank from size 500 to size 1, and [0.6, 0.8] still points in exactly the same direction as [300, 400].

Note: In PyTorch, this is one line: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0). A maximum of 1.0 is a common choice when training LLMs.

A lower learning rate and a short warmup, which starts training with tiny steps and slowly increases them, also help keep the updates under control in the early, unstable steps.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Comparison of vanishing and exploding gradients

Let me tabulate the differences between vanishing and exploding gradients for your better understanding.

Now, we must have understood why vanishing and exploding gradients happen.

Frequently Asked Questions

Why does Sigmoid cause vanishing gradients?

Sigmoid causes vanishing gradients because its slope is very small. At its steepest point, the slope is only 0.25, and at its flat ends, the slope is almost 0. So, every Sigmoid layer multiplies the gradient by 0.25 or less, and after 10 layers, the gradient becomes less than one millionth of its size.

How do Transformers avoid vanishing gradients?

Transformers use residual connections and Layer Normalization. A residual connection adds the input of a block directly to its output, so on the way back, the shortcut carries the gradient almost untouched. Layer Normalization keeps the numbers at a steady size at every layer, which helps stop the signal from drifting too small or too big.

Does gradient clipping change the direction of the gradient?

No. Clipping by norm does not change the direction of the gradient. If the overall size of the gradient is above the chosen maximum, every number in the gradient is shrunk by the same factor. For example, the gradient [300, 400] clipped to a maximum size of 1 becomes [0.6, 0.8], which points in exactly the same direction.

What maximum value is commonly used for gradient clipping?

A maximum of 1.0 is a common choice when training LLMs. In PyTorch, gradient clipping by norm is one line: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0). A lower learning rate and a short warmup also help keep the weight updates under control in the early, unstable steps of training.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.