Outcome School

Why Do We Initialize Weights Randomly in Neural Networks?

Authors
Why Do We Initialize Weights Randomly in Neural Networks?

We initialize weights randomly in neural networks so that every neuron starts a little different from the others and can learn something different. If all the weights start with the same value, all the neurons in a layer learn the exact same thing, and the network wastes most of its neurons.

In this blog, we will learn about why we initialize weights randomly in neural networks. We will also see what weights are and why training needs starting values, what goes wrong when all the weights start the same, why plain random values can be too big or too small, how Xavier and He initialization pick the right size, what we do with the biases, and why we fix a random seed.

We will cover the following:

  • What are weights?
  • Approach 1: Start all the weights with the same value
  • Approach 2: Start the weights with random values
  • Approach 3: Start the weights with scaled random values
  • What about the biases?
  • Why do we fix the random seed?

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What are weights?

A neural network is made of layers of small units called neurons. Each neuron takes some numbers as input, multiplies each input by a number, adds them up, and passes the result forward.

The numbers that the neuron multiplies with are called weights. A weight decides how much one input matters to that neuron.

Let's say a neuron gets two inputs, x1 = 1 and x2 = 2, and its weights are w1 = 0.5 and w2 = 0.5. Then, the neuron computes:

output = w1 * x1 + w2 * x2 = 0.5 * 1 + 0.5 * 2 = 1.5

Training is the process of slowly changing these weights so that the network gives better answers. In every training step, the network makes a guess, checks how wrong the guess is, and then nudges every weight a little in the direction that reduces the error. The amount by which each weight is to blame for the error is called its gradient.

We have a detailed blog on how backpropagation works that explains step by step how these gradients are calculated.

But, training changes the weights. It does not create them. Before the very first training step, every weight must already have some value. So, the question is: what values should the weights start with?

This step of picking the starting values is called weight initialization. It decides whether the network learns well, learns slowly, or does not learn at all.

Let's see this as a diagram:

┌──────────────────────────────┐
│ Weight initialization        │
│ pick starting values (once)  │
└──────────────────────────────┘
               │
               ↓
┌──────────────────────────────┐
│ Training step 1              │
│ guess, error, update weights │
└──────────────────────────────┘
               │
               ↓
┌──────────────────────────────┐
│ Training step 2              │
│ guess, error, update weights │
└──────────────────────────────┘
               │
               ↓
┌──────────────────────────────┐
│ ... thousands more steps     │
└──────────────────────────────┘

Here, we can see that weight initialization happens only once, at the very beginning. Every training step after that starts from the weights that the previous step left behind.

The best way to learn this is by taking an example. Let's try different approaches one by one.

Approach 1: Start all the weights with the same value

The simplest idea is to start every weight with the same value, for example 0 or 0.5. It looks fair, because no weight gets any special treatment.

Let's take a tiny network for the sake of understanding:

  • 2 inputs: x1 = 1 and x2 = 2
  • 1 hidden layer with 2 neurons: h1 and h2
  • 1 output neuron: y
  • The correct answer (target) is 3

We start all 6 weights with the same value 0.5. We also skip the activation function here just for the sake of understanding.

Step 1: Both hidden neurons compute their output.

h1 = 0.5 * 1 + 0.5 * 2 = 1.5

h2 = 0.5 * 1 + 0.5 * 2 = 1.5

Both neurons see the same inputs and have the same weights. So, both give the same output, 1.5.

Step 2: The output neuron computes the final answer.

y = 0.5 * h1 + 0.5 * h2 = 0.5 * 1.5 + 0.5 * 1.5 = 1.5

Step 3: We find the error.

error = y - target = 1.5 - 3 = -1.5

Step 4: We find the gradient of each incoming weight of the hidden neurons. For a weight going from input x into a hidden neuron, the gradient is:

gradient = x * error * (weight from that hidden neuron to the output)

For h1, the weight from x1 gets 1 * -1.5 * 0.5 = -0.75 and the weight from x2 gets 2 * -1.5 * 0.5 = -1.5.

For h2, we get the exact same numbers: -0.75 and -1.5. This is because h2 has the same inputs, the same outgoing weight, and the same error as h1.

Step 5: We update the weights with a learning rate of 0.01.

new weight = old weight - 0.01 * gradient

Both h1 and h2 move from [0.5, 0.5] to [0.5075, 0.515].

Let's see this as a diagram:

┌────────────────────────────────────────────┐
│ inputs: x1 = 1, x2 = 2                     │
└────────────────────────────────────────────┘
          │                       │
          ↓                       ↓
┌────────────────────┐  ┌────────────────────┐
│ h1                 │  │ h2                 │
│ weights: 0.5, 0.5  │  │ weights: 0.5, 0.5  │
│ output: 1.5        │  │ output: 1.5        │
└────────────────────┘  └────────────────────┘
          │                       │
          ↓                       ↓
┌────────────────────────────────────────────┐
│ y = 1.5, target = 3, error = -1.5          │
└────────────────────────────────────────────┘
          │                       │
          ↓                       ↓
┌────────────────────┐  ┌────────────────────┐
│ h1 gradients:      │  │ h2 gradients:      │
│ -0.75, -1.5        │  │ -0.75, -1.5        │
│ new: 0.5075, 0.515 │  │ new: 0.5075, 0.515 │
└────────────────────┘  └────────────────────┘

Here, we can see that the two neurons start the same, give the same output, receive the same update, and end up the same again. The next step repeats the same thing, so they stay identical forever.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Let's verify this with code. We train this tiny network for 50 steps as below:

import numpy as np

x = np.array([1.0, 2.0])     # 2 inputs
target = 3.0

W1 = np.full((2, 2), 0.5)    # inputs -> 2 hidden neurons
W2 = np.full(2, 0.5)         # 2 hidden neurons -> output

for step in range(50):
    h = x @ W1               # hidden neuron outputs
    y = h @ W2               # final output
    error = y - target
    grad_W2 = error * h               # blame for each outgoing weight
    grad_W1 = np.outer(x, error * W2) # blame for each incoming weight
    W2 -= 0.01 * grad_W2
    W1 -= 0.01 * grad_W1

print("Prediction:", round(float(y), 4))
print("Neuron 1 weights:", W1[:, 0].round(4))
print("Neuron 2 weights:", W1[:, 1].round(4))

Here, W1 holds the weights going into the hidden neurons (column 0 for neuron 1, column 1 for neuron 2), and W2 holds the weights going to the output. In each step, we compute the output, find the error, compute the gradients, and update the weights.

It will print the following:

Prediction: 2.9975
Neuron 1 weights: [0.5942 0.6884]
Neuron 2 weights: [0.5942 0.6884]

Here, we can see that the network did learn to predict close to 3, but the two hidden neurons still have the exact same weights after 50 steps.

This is called the symmetry problem. When all the neurons in a layer start with the same weights, they compute the same output, get the same gradient, and receive the same update. So, they stay the same forever.

Means, a layer with 512 neurons behaves like a layer with just 1 neuron copied 512 times. All that extra capacity is wasted, because every neuron learns the same thing.

Starting all the weights at 0 is even worse. In our tiny network, the hidden neurons output 0, and every gradient also becomes 0, so the weights do not move at all.

The issue with this approach is that identical starting values give identical neurons. Let's see how the next approach solve this issue.

Approach 2: Start the weights with random values

So, here comes the randomness to the rescue. If every weight starts with a different random value, then every neuron computes a different output from the very first step. Different outputs give different gradients, and different gradients push each neuron in its own direction.

This is called breaking the symmetry.

Let's run the same tiny network again, but this time with random starting weights:

np.random.seed(42)
W1 = np.random.randn(2, 2)   # random starting weights
W2 = np.random.randn(2)

Here, np.random.randn gives random numbers from a bell-shaped curve centered at 0. Most of the numbers are between -1 and 1, and a few are larger. We will learn about np.random.seed at the end of this blog. Everything else in the training code stays the same.

It will print the following:

Prediction: 2.999
Neuron 1 weights: [0.507  0.6682]
Neuron 2 weights: [-0.0702  1.6593]

Here, we can see that the network again predicts close to 3, but now the two neurons have different weights. Each neuron has learned its own way of looking at the inputs. The symmetry is broken.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Random values solve the symmetry problem, but the size of those random values matters a lot.

Let's understand why. In a real network, a neuron often has hundreds of inputs, and a deep network has many layers. The output of one layer becomes the input of the next layer. So, whatever happens in one layer gets multiplied again and again.

  • If the weights are a bit too big, every layer makes the numbers bigger. After many layers, the numbers become huge. This is called exploding.
  • If the weights are a bit too small, every layer makes the numbers smaller. After many layers, the numbers become almost zero. This is called vanishing.

Let's see this with an experiment. We pass 1000 inputs through 10 layers of 512 neurons each. We use the ReLU activation function, which simply turns every negative number into 0 and keeps positive numbers as they are. We try random weights with two sizes: normal size (1.0) and very small size (0.01).

import numpy as np

np.random.seed(42)
x = np.random.randn(1000, 512)  # 1000 inputs, 512 numbers each

for scale in [1.0, 0.01]:
    h = x
    for layer in range(1, 11):
        W = np.random.randn(512, 512) * scale
        h = np.maximum(0, h @ W)  # ReLU
    print(f"scale = {scale}: average size at layer 10 = {h.mean():.3g}")

Here, scale controls the size of the random weights. In each layer, we multiply the current numbers by a new random weight matrix and apply ReLU. After 10 layers, we print the average size of the numbers.

It will print the following:

scale = 1.0: average size at layer 10 = 6.13e+11
scale = 0.01: average size at layer 10 = 6.88e-09

Here, 6.13e+11 means about 613 billion, and 6.88e-09 means about 0.00000000688. The input numbers we started with were around 1 in size. So, with plain random weights, the numbers either blow up or die out after only 10 layers. Many real networks have far more layers than this.

Both cases break training:

  • Exploding: The numbers and the gradients become so large that each update throws the weights far away. Training jumps around wildly or the numbers overflow.
  • Vanishing: The numbers and the gradients become almost zero. The early layers get almost no update, so they barely learn.

We have a detailed blog on why vanishing and exploding gradients happen that covers this end to end.

There is one more problem for activation functions like tanh and sigmoid, which squash every input into a fixed range. For example, tanh squashes everything between -1 and 1. When the weights are too big, almost every output gets stuck near -1 or 1. This is called saturation. In a stuck region, the curve is almost flat, so the gradient is almost zero, and the neuron stops learning. If we run the same experiment with tanh instead of ReLU and scale = 1.0, about 91 percent of the outputs of the very first layer are already stuck above 0.99 or below -0.99.

To learn Neural Networks, Activation Functions, and Backpropagation from the ground up, check out our AI and Machine Learning Program at Outcome School.

The issue with this approach is that random values break the symmetry, but the wrong size makes the signal explode or vanish. Let's see how the next approach solve this issue.

Approach 3: Start the weights with scaled random values

Now, the question is: what is the right size?

The answer is: a size that keeps the numbers roughly the same from one layer to the next.

The key observation is that a neuron adds up many inputs. The more inputs it has, the bigger the sum gets. So, a neuron with many inputs needs smaller weights, and a neuron with few inputs can have bigger weights. Means, the size of the random weights must depend on the number of inputs.

The number of inputs to a neuron is called fan-in, and we will write it as n_in. The number of outputs of a layer is called fan-out, and we will write it as n_out.

We still pick random numbers centered at 0. We only control their spread, which is measured by the standard deviation. A small standard deviation means the random numbers stay close to 0. A large standard deviation means they spread far from 0.

There are two popular ways to pick this standard deviation.

Xavier (Glorot) initialization

Xavier initialization, also called Glorot initialization, was proposed by Xavier Glorot and Yoshua Bengio. It is designed for activation functions like tanh and sigmoid.

std = sqrt(2 / (n_in + n_out))

Here, it looks at both the number of inputs and the number of outputs, so that the numbers stay stable both when going forward and when the gradients flow backward.

For our layer with 512 inputs and 512 outputs, the standard deviation is sqrt(2 / 1024) = sqrt(1 / 512), which is about 0.044.

He (Kaiming) initialization

He initialization, also called Kaiming initialization, was proposed by Kaiming He and his co-authors. It is designed for the ReLU activation function.

std = sqrt(2 / n_in)

Here, the extra 2 comes from ReLU itself. ReLU turns roughly half of the numbers into 0. So, the signal loses about half of its strength in every layer. The 2 doubles the spread of the weights to make up for that loss.

For our layer with 512 inputs, the standard deviation is sqrt(2 / 512), which is 0.0625.

Let's run our experiment again with He initialization:

import numpy as np

np.random.seed(42)
x = np.random.randn(1000, 512)

h = x
for layer in range(1, 11):
    W = np.random.randn(512, 512) * np.sqrt(2 / 512)  # He
    h = np.maximum(0, h @ W)  # ReLU
    if layer in (1, 5, 10):
        print(f"layer {layer}: average size = {h.mean():.3f}")

Here, the only change is the size of the weights. Instead of 1.0 or 0.01, we multiply them by np.sqrt(2 / 512).

It will print the following:

layer 1: average size = 0.563
layer 5: average size = 0.532
layer 10: average size = 0.557

Here, we can see that the numbers stay around 0.55 from layer 1 to layer 10. They do not explode and they do not vanish. The signal travels through the whole network in good shape, and the gradients can travel back in good shape too.

If we want to go deep into Deep Learning and Gradient Descent, and build a Neural Network from scratch, we have a complete program - check out our AI and Machine Learning Program at Outcome School.

Let's compare all three experiments as a diagram:

┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
│ input: about 1  │  │ input: about 1  │  │ input: about 1  │
│ scale = 1.0     │  │ scale = 0.01    │  │ He scale: 0.0625│
└─────────────────┘  └─────────────────┘  └─────────────────┘
         │                    │                    │
         ↓                    ↓                    ↓
┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
│ 10 ReLU layers  │  │ 10 ReLU layers  │  │ 10 ReLU layers  │
└─────────────────┘  └─────────────────┘  └─────────────────┘
         │                    │                    │
         ↓                    ↓                    ↓
┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
│ layer 10:       │  │ layer 10:       │  │ layer 10:       │
│ 613 billion     │  │ 0.00000000688   │  │ 0.557           │
│ exploding       │  │ vanishing       │  │ stable          │
└─────────────────┘  └─────────────────┘  └─────────────────┘

Here, we can see that the same 10 layers give completely different results, and the only difference is the size of the starting weights.

We must match the method with the activation function. If we use the Xavier size with ReLU in the same experiment, the average size at layer 10 drops to about 0.017, because Xavier does not make up for the half of the numbers that ReLU turns into 0.

Let me tabulate the differences between Xavier and He initialization for your better understanding.

PropertyXavier (Glorot)He (Kaiming)
Formulastd = sqrt(2 / (n_in + n_out))std = sqrt(2 / n_in)
Best fortanh and sigmoidReLU and its variants
Size for 512 inputsabout 0.0440.0625

Popular libraries such as PyTorch and Keras already use a scaled random initialization by default for their layers, so most of the time we do not need to write it ourselves.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

What about the biases?

Each neuron also has one extra number called the bias, which is added after the weighted sum. It lets the neuron shift its output up or down.

output = w1 * x1 + w2 * x2 + bias

Biases usually start at 0.

The question is: does this not bring back the symmetry problem? The answer is no. The symmetry is already broken by the random weights. Since every neuron has different weights, every neuron gets a different gradient for its bias too. So, the biases become different after the first update.

Why do we fix the random seed?

Random weights have one side effect. Every time we run the training, the starting weights are different, so the final result can also be a little different. This makes it hard to compare two experiments or to find a bug.

To solve this, we fix the random seed. A random seed is a starting number for the random number generator. With the same seed, the computer produces the exact same sequence of "random" numbers every time.

import numpy as np

np.random.seed(42)
print(np.random.randn(3).round(4))

np.random.seed(42)
print(np.random.randn(3).round(4))

It will print the following:

[ 0.4967 -0.1383  0.6477]
[ 0.4967 -0.1383  0.6477]

Here, we can see that both runs give the same three numbers because we used the same seed 42. The weights still look random to the network, so they still break the symmetry, but now our experiment can be repeated exactly.

Now we must have understood why we initialize weights randomly in neural networks, what goes wrong with the same values, why the size of the random values matters, and how Xavier and He initialization pick the right size.

Frequently Asked Questions

What happens if we initialize all the weights to zero?

The weights do not move at all. When every weight starts at 0, the hidden neurons output 0 and every gradient also becomes 0, so training cannot change anything. Even a non-zero value that is the same for every weight is a problem, because all the neurons in a layer stay identical forever.

Should we use Xavier or He initialization?

We should match the method with the activation function. Xavier initialization is designed for tanh and sigmoid, and He initialization is designed for ReLU and its variants. If we use the Xavier size with ReLU, the signal keeps shrinking, because Xavier does not make up for the half of the numbers that ReLU turns into 0.

Why not just use very small random weights?

Very small random weights make the signal vanish. Every layer makes the numbers smaller, and with a scale of 0.01, the average size drops to about 0.00000000688 after only 10 layers. The gradients also become almost zero, so the early layers barely learn.

Do we need to write weight initialization code ourselves?

Most of the time, no. Popular libraries such as PyTorch and Keras already use a scaled random initialization by default for their layers. We still need to understand it, because the right size of the starting weights decides whether the network learns well, learns slowly, or does not learn at all.

Does fixing the random seed bring back the symmetry problem?

No. A fixed seed only makes the computer produce the same sequence of random numbers every time we run the training. The weights still look random to the network and are still different from each other, so the symmetry is still broken. The seed only makes our experiment repeatable.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.