What is the Softmax Function and How Does It Work?
- Authors
- Name
- Amit Shekhar
- Published on
The Softmax Function is a formula that turns a list of raw scores into probabilities, which means every value becomes a number between 0 and 1, and all the values add up to exactly 1.
In this blog, we will learn about the Softmax Function and how it works. We will also see why simply dividing by the sum fails, how the exponent step fixes it with a step-by-step worked example, its important properties, how temperature controls it, the trick that keeps it stable on a computer, and where it is used in classification models, LLMs, and attention.
We will cover the following:
- The Problem: Raw Scores Are Not Probabilities
- Why Not Just Divide by the Sum?
- What is the Softmax Function?
- A Step-by-Step Worked Example
- Softmax in Python
- Important Properties of Softmax
- Temperature in Softmax
- The Numerical Stability Trick
- Where is Softmax Used?
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning at Outcome School.
Let's get started.
The Problem: Raw Scores Are Not Probabilities
Let's say we have built a model that looks at a photo and tells us whether it is a cat, a dog, or a bird.
At the end, the model gives one raw score for each option. These raw scores are called logits. A higher score means the model believes more in that option.
Suppose the model looks at a photo and gives these scores:
- Cat: 2.0
- Dog: 1.0
- Bird: 0.1
We can see that the model likes "Cat" the most. But these numbers are hard to use. Can we say the model is 2.0 sure? 2.0 out of what?
What we really want is something like this: "The model is 66% sure it is a cat, 24% sure it is a dog, and 10% sure it is a bird."
Means, we want probabilities. For a list of numbers to be valid probabilities, two rules must hold:
- Every number must be between 0 and 1.
- All the numbers must add up to exactly 1.
Raw scores follow neither rule. A score can be 7.5, it can be -3.2, and the scores can add up to anything.
So, we need a way to convert raw scores into probabilities. Here comes the Softmax Function into the picture.
Why Not Just Divide by the Sum?
Before jumping into softmax, let's try the simplest idea first. We can divide each score by the total of all the scores.
For our scores 2.0, 1.0, and 0.1, the total is 3.1. Dividing each score by 3.1 gives 0.65, 0.32, and 0.03. They add up to 1. It looks like it works.
But, scores can be negative. Let's say the scores are 3, 1, and -2.
The total is 3 + 1 + (-2) = 2. Now, let's divide each score by 2:
- 3 / 2 = 1.5
- 1 / 2 = 0.5
- -2 / 2 = -1.0
We got 1.5, which is more than 1, and -1.0, which is a negative probability. Neither makes any sense.
Also, if the scores are 2 and -2, the total is 0, and we cannot divide by zero at all.
The issue with this approach is negative numbers. We need a way to first turn every score into a positive number, without losing the order of the scores. Let's see how softmax solves this issue.
What is the Softmax Function?
The Softmax Function takes a list of raw scores, raises the number e to the power of each score, and then divides each result by the total of all the results.
Here, e is a fixed mathematical constant, roughly 2.718. Raising e to the power of a score is called taking the exponent of that score, and it is written as exp(score).
The formula is as below:
softmax(x_i) = exp(x_i) / (exp(x_1) + exp(x_2) + ... + exp(x_n))
Here, x_i is the score of one option, and the bottom part is the sum of the exponents of all the scores.
Why the exponent? Because exp has two useful properties:
- The result is always positive, even for a negative input. For example,
exp(-2)is about 0.135, andexp(-100)is a tiny positive number very close to 0. - A bigger input always gives a bigger output. So, the order of the scores stays the same.
This fixes exactly the problem we had. First, the exponent makes every score positive. Then, dividing by the total makes everything add up to 1.
Let's see this as a diagram:
┌──────────────────────────────┐
│ Raw scores (logits) │
│ Cat 2.0, Dog 1.0, Bird 0.1 │
└──────────────────────────────┘
↓
┌──────────────────────────────┐
│ Step 1: Take exp of each │
│ 7.389, 2.718, 1.105 │
└──────────────────────────────┘
↓
┌──────────────────────────────┐
│ Step 2: Add them up │
│ Total = 11.212 │
└──────────────────────────────┘
↓
┌──────────────────────────────┐
│ Step 3: Divide each by total │
│ 0.66, 0.24, 0.10 │
└──────────────────────────────┘
↓
┌──────────────────────────────┐
│ Probabilities (sum = 1) │
│ Cat 66%, Dog 24%, Bird 10% │
└──────────────────────────────┘
Here, we can see that softmax is a short pipeline of three steps: exponent, add, and divide. We have a detailed blog on Activation Functions that explains how softmax compares with Sigmoid, Tanh, and ReLU.
Planning your move into AI?
No matter which tech domain you work in today, with your effort, you can transition to these roles:
- AI Engineer
- Forward Deployed Engineer
- AI Agent Engineer
- Harness Engineer
- Voice AI Engineer
- AI Architect
- LLM Engineer
- LLM Inference Engineer
- Gen AI Engineer
- MLOps Engineer
- ML Engineer
- Deep Learning Engineer
Our AI and Machine Learning Program at Outcome School takes you there.
A Step-by-Step Worked Example
The best way to learn this is by taking an example. Let's use our scores: Cat 2.0, Dog 1.0, Bird 0.1.
Step 1: We take the exponent of each score.
- Cat: exp(2.0) = 7.389
- Dog: exp(1.0) = 2.718
- Bird: exp(0.1) = 1.105
All three are positive now.
Step 2: We add all the exponents.
Total = 7.389 + 2.718 + 1.105 = 11.212
Step 3: We divide each exponent by the total.
- Cat: 7.389 / 11.212 = 0.659
- Dog: 2.718 / 11.212 = 0.242
- Bird: 1.105 / 11.212 = 0.099
Step 4: We check the result.
0.659 + 0.242 + 0.099 = 1.000. Every value is between 0 and 1, and they add up to 1.
So, the model is about 66% sure it is a cat, 24% sure it is a dog, and 10% sure it is a bird. This is how softmax turns raw scores into probabilities.
Now, let's try the scores that broke the "divide by the sum" approach: 3, 1, and -2.
- exp(3) = 20.086
- exp(1) = 2.718
- exp(-2) = 0.135
The total is 22.939. Dividing gives 0.876, 0.118, and 0.006. No negative values, nothing above 1, and they add up to 1. It works perfectly.
Softmax in Python
Now, let's write the code for softmax in Python using NumPy, a popular library for working with lists of numbers, as below:
import numpy as np
def softmax(scores):
exps = np.exp(scores)
return exps / exps.sum()
scores = np.array([2.0, 1.0, 0.1])
probs = softmax(scores)
print(probs.round(2))
print(probs.sum().round(2))
Here, we have:
np.exp(scores)takes the exponent of every score in one go.exps.sum()adds all the exponents.exps / exps.sum()divides each exponent by the total.
This will print the following:
[0.66 0.24 0.1 ]
1.0
Here, we can see the same 0.66, 0.24, and 0.10 that we calculated by hand, and the sum is 1.
A quick note for you
No matter which tech domain you work in, get familiar with these topics:
- LLM
- RAG
- MCP
- Agent
- Fine-tuning
- Quantization
We put it all together in one video:
AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization
No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.
Now, let's get back to the topic.
Important Properties of Softmax
Now, let's understand the properties that make softmax so useful.
Property 1: The outputs always add up to 1. We divide every exponent by the sum of all the exponents. So, each output is a share of the total, and all the shares together make the whole. This is why we can read them as probabilities.
Property 2: It keeps the order. The highest score always gets the highest probability, and the lowest score always gets the lowest probability. In our example, Cat had the highest score and got the highest probability.
Property 3: It amplifies the gaps. When one score is higher than another score by 1, its probability is about 2.718 times bigger. In our example, Cat (2.0) is 1 more than Dog (1.0), and 0.659 / 0.242 is about 2.72. If one score is higher than another by 5, its probability is about 148 times bigger. So, a small lead in the scores becomes a bigger lead in the probabilities.
This is also where the name comes from. Softmax is a "soft" version of picking the maximum. A "hard" max would give 100% to the top option and 0% to all others. Softmax gives most of the share to the top option, but still keeps some share for the others.
Property 4: Only the differences between scores matter. If we add the same number to every score, the probabilities do not change. The scores 2.0, 1.0, 0.1 and the scores 102.0, 101.0, 100.1 give exactly the same result. We will use this property very soon.
To master Neural Networks, Activation Functions, Logits, and Cross-Entropy, and to build a Neural Network from scratch, we can check out our AI and Machine Learning Program at Outcome School.
Temperature in Softmax
Sometimes, we want to control how "sharp" or how "flat" the probabilities are. For this, we use temperature.
Temperature is one positive number. We divide every score by the temperature before applying softmax:
softmax(x_i / T)
Here, T is the temperature.
Let's see what happens with our scores 2.0, 1.0, and 0.1:
| Temperature | Cat | Dog | Bird | Effect |
|---|---|---|---|---|
| 0.5 | 0.86 | 0.12 | 0.02 | Sharper, top option wins |
| 1.0 | 0.66 | 0.24 | 0.10 | Normal softmax |
| 2.0 | 0.50 | 0.30 | 0.19 | Flatter, options are closer |
Note: The values in each row are rounded to two decimal places, so a row can add up to 0.99 instead of 1.
Here, we can notice that:
- A low temperature (below 1) makes the gaps between scores bigger. The top option gets almost all the probability.
- A high temperature (above 1) makes the gaps smaller. The probabilities become more equal.
Let's see the low temperature case of T = 0.5 as a diagram:
┌──────────────────────────────┐
│ Scores: 2.0, 1.0, 0.1 │
│ Gaps: 1.0 and 0.9 │
└──────────────────────────────┘
↓
┌──────────────────────────────┐
│ Divide each score by 0.5 │
│ 4.0, 2.0, 0.2 │
│ Gaps: 2.0 and 1.8 │
└──────────────────────────────┘
↓
┌──────────────────────────────┐
│ Softmax │
│ 0.86, 0.12, 0.02 │
└──────────────────────────────┘
Here, we can see that dividing by 0.5 doubles the gaps between the scores. Softmax amplifies these bigger gaps, so Cat jumps from 0.66 to 0.86. Dividing by 2.0 does the opposite. It halves the gaps, and the probabilities move closer together.
This is exactly the "temperature" setting we see in LLM tools. A low temperature makes the LLM pick the most likely token almost every time, so the answer is focused and predictable. A high temperature gives other tokens a fair chance, so the answer is more varied and creative.
Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.
The Numerical Stability Trick
Now, there is a practical problem when we run softmax on a computer.
The exponent grows very fast. exp(10) is about 22,026. exp(100) is a number with 44 digits. And exp(1000) is so big that a computer cannot store it at all. So, the computer stores it as infinity.
Let's say the scores are 1002.0, 1001.0, and 1000.1. Let's run our softmax function on them:
print(softmax(np.array([1002.0, 1001.0, 1000.1])))
NumPy shows an overflow warning, and then it prints the following:
[nan nan nan]
Here, nan means "not a number". Every exponent became infinity, and infinity divided by infinity has no answer.
So, how can we fix this? The answer is Property 4. Only the differences between scores matter. So, we can subtract the same number from every score, and the probabilities will not change.
The trick is to subtract the maximum score from every score before taking the exponent.
- 1002.0 - 1002.0 = 0
- 1001.0 - 1002.0 = -1.0
- 1000.1 - 1002.0 = -1.9
Now, the biggest score is 0, so the biggest exponent is exp(0) = 1. All the other exponents are between 0 and 1. Nothing can grow too large anymore.
Let's see this as a diagram:
┌──────────────────────────────┐
│ Scores: 1002, 1001, 1000.1 │
└──────────────────────────────┘
↓
┌──────────────────────────────┐
│ Subtract max (1002) │
│ 0, -1.0, -1.9 │
└──────────────────────────────┘
↓
┌──────────────────────────────┐
│ Take exp of each │
│ 1.000, 0.368, 0.150 │
└──────────────────────────────┘
↓
┌──────────────────────────────┐
│ Divide each by total (1.518) │
│ 0.66, 0.24, 0.10 │
└──────────────────────────────┘
Here, we can see that we get 0.66, 0.24, and 0.10 again. These scores have the same differences as 2.0, 1.0, and 0.1, so they give the same probabilities.
Our updated code with temperature and the stability trick:
import numpy as np
def softmax(scores, temperature=1.0):
scores = scores / temperature
scores = scores - scores.max()
exps = np.exp(scores)
return exps / exps.sum()
print(softmax(np.array([1002.0, 1001.0, 1000.1])).round(2))
print(softmax(np.array([2.0, 1.0, 0.1]), temperature=0.5).round(2))
print(softmax(np.array([2.0, 1.0, 0.1]), temperature=2.0).round(2))
Here, we first divide by the temperature, then subtract the maximum score, and then do the usual exponent and divide.
This will print the following:
[0.66 0.24 0.1 ]
[0.86 0.12 0.02]
[0.5 0.3 0.19]
It works perfectly. Popular libraries like PyTorch use this same trick inside their softmax.
Where is Softmax Used?
Now, let's see where softmax is used in real AI systems.
The output of a classification model:
When a model has to pick one answer from a fixed list of options, the last step is usually softmax. Our cat, dog, and bird model is an example. Another example is a model that reads a review and says whether it is positive, negative, or neutral. Softmax turns the final scores into probabilities, and we pick the option with the highest probability. We have a detailed blog on Cross-Entropy Loss that explains how these probabilities are used to measure the error during training.
Next-token prediction in LLMs:
An LLM (Large Language Model) writes text one token at a time. A token is a word or a piece of a word. At every step, the LLM gives a raw score to every token in its vocabulary, which is the full list of tokens it knows. This list can have more than 100,000 tokens. Softmax turns these scores into probabilities. Then, the next token is picked based on these probabilities. This is also where the temperature setting comes into the picture.
Attention in Transformers:
Transformers are the architecture behind LLMs. Inside them, a mechanism called attention decides how much each token must focus on every other token. It first calculates a score for every pair of tokens. Then, softmax turns these scores into weights that add up to 1. Means, each token splits its total focus of 100% among the tokens it can see, and gives more focus to the most relevant ones.
Now we must have understood what the Softmax Function is, how it works, and where it is used.
Frequently Asked Questions
Why is it called softmax?
Softmax is called so because it is a "soft" version of picking the maximum. A hard max gives 100% to the top option and 0% to all the others. Softmax gives most of the share to the top option, but still keeps some share for the other options, and the highest score always gets the highest probability.
What are logits?
Logits are the raw scores that a model gives at the end, one score for each option. A higher logit means the model believes more in that option. Logits can be any number, like 7.5 or -3.2, and they can add up to anything, so we use softmax to turn them into probabilities.
Can softmax give a probability of exactly 0?
No. Softmax takes the exponent of every score, and the exponent is always positive, even for a negative score. So every option gets some share of the total. For a very low score like -100, the exponent is a tiny positive number very close to 0, so that option gets a very small share, but not exactly 0.
Does adding the same number to every score change the softmax output?
No. Adding the same number to every score does not change the probabilities, because only the differences between the scores matter. The scores 2.0, 1.0, 0.1 and the scores 102.0, 101.0, 100.1 give exactly the same result. This is why we can subtract the maximum score before taking the exponent to keep softmax stable on a computer.
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
You can connect with me on:
Follow Outcome School on: