What are Activation Functions in Neural Networks? Sigmoid, Tanh, ReLU, and Softmax Explained

Activation Functions are math functions that take the number calculated inside a neuron and change it into the final output of that neuron, which allows a neural network to learn curved, complex patterns instead of only straight lines. Sigmoid, Tanh, ReLU, and Softmax are the most common ones.

In this blog, we will learn about Activation Functions in Neural Networks. We will also see where an activation function sits inside a neuron, why a neural network needs non-linearity, how Sigmoid, Tanh, ReLU, and Softmax work with small worked numbers, the problems of vanishing gradients and dying ReLU, and which activation function to use in which layer.

We will cover the following:

• What is an Activation Function?
• Where does the Activation Function sit inside a Neuron?
• Why do we need Non-Linearity?
• Sigmoid
• Tanh
• ReLU
• Softmax
• Which Activation Function goes where?
• Comparison of Activation Functions

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is an Activation Function?

An Activation Function is a function that takes the number calculated inside a neuron and converts it into the output of that neuron.

A neural network is made of many small units called neurons, arranged in layers. Each neuron takes some numbers as input, does a small calculation, and passes a number to the next layer. The activation function is the last step of that calculation.

Let's say a light dimmer is connected to a bulb. The electricity comes in, and the dimmer decides how bright the bulb glows: fully off, a little on, or fully on. The activation function does the same job for a neuron. The calculated number comes in, and the activation function decides how strongly the neuron "fires".

Where does the Activation Function sit inside a Neuron?

The best way to learn this is by taking an example.

Let's say a neuron receives two inputs: x1 = 2 and x2 = 3. The neuron has one weight for each input, w1 = 0.5 and w2 = 0.25, and one bias, b = -0.25. Weights and bias are the numbers that the network learns during training.

Step 1: The neuron calculates the weighted sum. We call this number z.

z = (w1 * x1) + (w2 * x2) + b

Putting our numbers:

z = (0.5 * 2) + (0.25 * 3) + (-0.25) = 1 + 0.75 - 0.25 = 1.5

Step 2: The neuron passes z through the activation function. Let's use Sigmoid here. We will learn about it in detail soon.

sigmoid(1.5) = 0.818

So, the output of this neuron is 0.818, and this number goes to the next layer.

Let's see this as a diagram:

┌──────────────────┐  ┌──────────────────┐
│ x1 * w1          │  │ x2 * w2          │
│ 2 * 0.5 = 1      │  │ 3 * 0.25 = 0.75  │
└────────┬─────────┘  └────────┬─────────┘
         └──────────┬──────────┘
                    ↓
┌────────────────────────────────────────┐
│ Weighted sum + bias                    │
│ z = 1 + 0.75 + (-0.25) = 1.5           │
└───────────────────┬────────────────────┘
                    ↓
┌────────────────────────────────────────┐
│ Activation function                    │
│ sigmoid(1.5) = 0.818                   │
└───────────────────┬────────────────────┘
                    ↓
┌────────────────────────────────────────┐
│ Output 0.818 goes to the next layer    │
└────────────────────────────────────────┘

Here, we can see that the activation function sits right after the weighted sum. Every neuron in a layer does these two steps: first the weighted sum, then the activation function.

To learn Neural Networks, Feed Forward Neural Networks (FFNNs), and Activation Functions from the ground up, check out our AI and Machine Learning Program at Outcome School.

Why do we need Non-Linearity?

Now, the question is: why can we not just pass z directly to the next layer? Why do we need this extra step at all?

The answer is non-linearity.

The weighted sum is a linear calculation. Linear means that if we draw it on a graph, it is always a straight line. It only multiplies and adds.

Here comes the problem. If we stack two linear calculations, we still get a linear calculation.

Let's say layer 1 does y = 2x + 1 and layer 2 does out = 3y + 4. If we put the first into the second:

out = 3 * (2x + 1) + 4 = 6x + 7

This is again a straight line. Even if we stack 100 such layers, the result is still one straight line. Means, without an activation function, a deep neural network is no more powerful than a single layer.

But real-world patterns are not straight lines. Recognizing a cat in a photo, understanding a sentence, or predicting house prices needs curves, bends, and complex shapes.

So, here comes the activation function to the rescue. It is a non-linear function, which means its graph is not a straight line. When we put a non-linear function after every layer, stacking layers finally adds real power. Each layer can bend the pattern a little more, and many layers together can learn very complex patterns.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Now, let's learn the most common activation functions one by one.

Sigmoid

Sigmoid is an activation function that squeezes any number into a value between 0 and 1.

The formula is:

sigmoid(z) = 1 / (1 + e^(-z))

Here, e is a fixed math number, about 2.718.

Let's see some values:

Here, we can notice that a very negative number goes close to 0, a very positive number goes close to 1, and 0 becomes exactly 0.5. The graph looks like a smooth "S" shape.

Since the output is always between 0 and 1, we can read it as a probability. For example, 0.881 can mean "88.1% chance that this email is spam".

Advantage:

• The output is between 0 and 1, so it works well as a probability for a yes/no answer.

Disadvantage: Vanishing Gradients

Before jumping into this problem, we must know one thing about training. During training, the network finds out how much each weight must change to reduce the error. This value is called the gradient. A big gradient means a big change. A tiny gradient means almost no change, so the weight almost stops learning.

The gradient passes backward through every layer, and at each layer it gets multiplied by the slope of the activation function at that point, along with the weights. We have a detailed blog on Backpropagation that explains how the gradient flows backward step by step.

The slope of sigmoid is at most 0.25 (at z = 0). At z = 10, the slope is only about 0.00005, because the curve is almost flat there.

Now, let's say the gradient passes through 10 layers, and each sigmoid gives its best slope of 0.25:

0.25 * 0.25 * ... (10 times) = 0.00000095

Let's see this as a diagram:

┌────────────────────────────────┐
│ Gradient at the output         │
│ gradient = 1                   │
└────────────────┬───────────────┘
                 ↓
              * 0.25
                 ↓
┌────────────────────────────────┐
│ After 1 sigmoid layer          │
│ gradient = 0.25                │
└────────────────┬───────────────┘
                 ↓
              * 0.25
                 ↓
┌────────────────────────────────┐
│ After 2 sigmoid layers         │
│ gradient = 0.0625              │
└────────────────┬───────────────┘
                 ↓
              * 0.25
                 ↓
┌────────────────────────────────┐
│ After 3 sigmoid layers         │
│ gradient = 0.015625            │
└────────────────┬───────────────┘
                 ↓
         * 0.25, 7 more times
                 ↓
┌────────────────────────────────┐
│ After 10 sigmoid layers        │
│ gradient = 0.00000095          │
└────────────────────────────────┘

Here, we can see that the gradient becomes 4 times smaller at every sigmoid layer, and it is almost zero by the time it reaches the early layers. So, the early layers almost stop learning. This is called the vanishing gradient problem.

Another small issue is that sigmoid outputs are always positive, never centered around 0. This makes training a little slower.

Where it is used: Sigmoid is mostly used in the output layer for a yes/no (binary) classification, such as spam or not spam. It is rarely used in hidden layers today.

This was all about Sigmoid. Now, let's move to Tanh.

Tanh

Tanh (Hyperbolic Tangent) is an activation function that squeezes any number into a value between -1 and 1.

The formula is:

tanh(z) = (e^z - e^(-z)) / (e^z + e^(-z))

Let's see some values:

Here, we can see that tanh also has an "S" shape, just like sigmoid. The difference is that the output goes from -1 to 1, and 0 stays 0.

Advantages:

• The output is centered around 0. Negative inputs give negative outputs and positive inputs give positive outputs. This usually makes training faster than with sigmoid.
• The slope at z = 0 is 1, which is 4 times bigger than the best slope of sigmoid.

Disadvantage:

• Tanh still becomes flat for large positive or large negative numbers. At z = 3, the slope is already about 0.0099. So, deep networks with tanh still suffer from vanishing gradients.

Where it is used: Tanh is used in hidden layers of smaller networks and inside older sequence models like RNNs and LSTMs. It is also used in an output layer when the answer must be between -1 and 1.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Now, it's time to learn about the most popular one: ReLU.

ReLU

ReLU (Rectified Linear Unit) is an activation function that keeps a positive number as it is and turns a negative number into 0.

The formula is:

relu(z) = max(0, z)

Let's see some values:

If the number is negative, the output is 0. Otherwise, the number passes through unchanged.

Even though each half is a straight line, the bend at 0 makes ReLU non-linear. And many ReLU neurons together, each bending at a different place, can build very complex shapes.

Advantages:

• For positive numbers, the slope is always exactly 1. It never becomes flat on the positive side, so ReLU does not shrink the gradient while it passes backward through many layers. This is one of the main reasons why very deep networks became much easier to train.
• It is very fast to calculate. There is no e and no division, just a comparison with 0.
• Many neurons output exactly 0, so only some neurons are active at a time. This is called sparsity, and it makes the network simpler and more efficient.

Disadvantage: Dying ReLU

For negative numbers, the output is 0 and the slope is also 0.

Now, let's say during training, the weights of a neuron change in such a way that its z becomes negative for every input. Then, the neuron always outputs 0, and its gradient is always 0. Since the gradient is 0, its weights never change again. The neuron is stuck and stops learning. This is called the dying ReLU problem.

So, how can we solve this? The answer is Leaky ReLU. Instead of giving 0 for negative numbers, it gives a tiny value:

leaky_relu(z) = max(0.01 * z, z)

For z = -2, Leaky ReLU gives -0.02 instead of 0. The slope for negative numbers is now 0.01, not 0, so the neuron can still learn and become active again.

Where it is used: ReLU and its variants are the default choice for hidden layers in most modern neural networks. Many modern LLMs use smoother relatives of ReLU, such as GELU and SwiGLU, which follow the same central idea.

If we want to go deep into Backpropagation, Gradient Descent, and Loss Functions, and build a Neural Network from scratch, we can check out our AI and Machine Learning Program at Outcome School.

This was all about ReLU. Now, let's move to Softmax.

Softmax

Softmax is an activation function that converts a list of numbers into a list of probabilities that add up to 1.

Sigmoid, Tanh, and ReLU work on one number at a time. Softmax is different. It works on the whole list of numbers in the output layer together.

The formula is:

softmax(z_i) = e^(z_i) / (sum of e^(z_j) for all j)

Here, z_i is the score of one class. We take e to the power of that score, and divide it by the total of e to the power of every score in the list. This makes every output positive, and all the outputs add up to 1.

Let's say a network must decide whether an image is a cat, a dog, or a bird. The output layer gives three raw scores: [2.0, 1.0, 0.1]. These raw scores are called logits.

Step 1: We take e to the power of each score: e^2.0 = 7.389, e^1.0 = 2.718, and e^0.1 = 1.105.

Step 2: We add them up: 7.389 + 2.718 + 1.105 = 11.212.

Step 3: We divide each one by the total. For Cat, 7.389 / 11.212 = 0.659.

After softmax:

Here, we can see that the probabilities add up to 0.659 + 0.242 + 0.099 = 1.0, and the biggest score gets the biggest probability. We have a detailed blog on Cross-Entropy Loss that explains how these probabilities are used to measure the error during training.

Where it is used: Softmax is used in the output layer when the network must pick one answer from many classes. An LLM also uses softmax to turn its scores into probabilities for every token in its vocabulary, and then picks the next token from them.

Now, let's see all four functions in code.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def tanh(z):
    return np.tanh(z)

def relu(z):
    return np.maximum(0, z)

def softmax(z):
    e = np.exp(z)
    return e / e.sum()

z = np.array([-2.0, 0.0, 2.0])
print("sigmoid:", np.round(sigmoid(z), 3))
print("tanh:   ", np.round(tanh(z), 3))
print("relu:   ", relu(z))
print("softmax:", np.round(softmax(np.array([2.0, 1.0, 0.1])), 3))

Here, we have written each formula as a small Python function using NumPy. We pass the same three numbers -2, 0, and 2 to sigmoid, tanh, and relu. For softmax, we pass the three logits of our cat, dog, and bird example.

It will print the following:

sigmoid: [0.119 0.5   0.881]
tanh:    [-0.964  0.     0.964]
relu:    [0. 0. 2.]
softmax: [0.659 0.242 0.099]

These are exactly the numbers that we calculated in the tables above.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Which Activation Function goes where?

Now, the next big question is: which activation function must we use in which layer?

A neural network has three kinds of layers:

• Input layer: It receives the raw data. It has no activation function.
• Hidden layers: These are the layers in the middle that learn the patterns.
• Output layer: It gives the final answer.

For hidden layers, the job of the activation function is to add non-linearity and let the gradient flow well. For the output layer, the job is to give the answer in the right form.

Let's see this as a diagram:

┌──────────────────────────────────┐
│ Input layer                      │
│ raw data, no activation          │
└────────────────┬─────────────────┘
                 ↓
┌──────────────────────────────────┐
│ Hidden layer 1                   │
│ ReLU (default choice)            │
└────────────────┬─────────────────┘
                 ↓
┌──────────────────────────────────┐
│ Hidden layer 2                   │
│ ReLU (default choice)            │
└────────────────┬─────────────────┘
                 ↓
┌──────────────────────────────────┐
│ Output layer                     │
│ activation depends on the task   │
└────────────────┬─────────────────┘
     ┌───────────┼───────────┐
     ↓           ↓           ↓
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Spam?   │ │ Cat/Dog │ │ Price   │
│ Sigmoid │ │ Softmax │ │ None    │
└─────────┘ └─────────┘ └─────────┘

Here, we can see that the hidden layers use ReLU, and the output layer uses the activation function that matches the type of answer:

• Yes or no (spam or not spam): Sigmoid, which gives one probability between 0 and 1.
• One of many classes (cat, dog, or bird): Softmax, which gives probabilities that add up to 1.
• Any number (the price of a house): no activation function, so the output can be any value.

Comparison of Activation Functions

Let me tabulate the differences between Sigmoid, Tanh, ReLU, and Softmax for your better understanding.

Now we must have understood what activation functions are, why a neural network needs non-linearity, and where to use Sigmoid, Tanh, ReLU, and Softmax.

Frequently Asked Questions

Can a neural network work without an activation function?

It can run, but it cannot learn complex patterns. Without an activation function, every layer is a linear calculation, and stacking linear calculations still gives one straight line. So, even a deep network with 100 layers would be no more powerful than a single layer.

Why is ReLU preferred over Sigmoid in hidden layers?

ReLU is preferred because its slope is always exactly 1 for positive numbers, so it does not shrink the gradient as it passes backward through many layers. Sigmoid has a slope of at most 0.25, so the gradient becomes tiny and the early layers almost stop learning. ReLU is also very fast to calculate.

What is the difference between ReLU and Leaky ReLU?

ReLU turns every negative number into 0, so a neuron whose input is always negative gets a gradient of 0 and stops learning. Leaky ReLU gives a tiny value instead, 0.01 times the input, for negative numbers. Its slope on the negative side is 0.01, so the neuron can still learn and become active again.

Do LLMs use activation functions?

Yes. Many modern LLMs use smoother relatives of ReLU, such as GELU and SwiGLU, which follow the same central idea. An LLM also uses softmax to turn its scores into probabilities for every token in its vocabulary, and then picks the next token from them.

Why is there no activation function in the output layer when predicting a price?

When the network must predict any number, such as the price of a house, the output must not be squeezed into a fixed range. Sigmoid would force it between 0 and 1, and ReLU would block negative values. So, we use no activation function, and the output can be any value.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.