What are Logits in Machine Learning?

Q: Should we apply softmax before passing logits to the loss function?

No. Loss functions like nn.CrossEntropyLoss in PyTorch take the raw logits and combine softmax and log into one stable calculation. If we apply softmax in the model and then pass the result to a loss that already applies softmax, softmax gets applied twice and the model learns poorly.

Logits are the raw scores that a machine learning model gives to each possible answer before those scores are turned into probabilities. A higher logit means the model is more confident about that answer, but a logit by itself is not a probability.

In this blog, we will learn about logits in machine learning. We will also see where logits come from inside a model, how softmax turns logits into probabilities, how sigmoid does the same for yes or no questions, why the name logit comes from log-odds, why models output logits instead of probabilities, and how LLMs use logits to pick the next token.

We will cover the following:

What are Logits?
Where do Logits come from?
From Logits to Probabilities using Softmax
Logits for Yes or No Questions using Sigmoid
Why is it called a Logit?
Why do Models output Logits instead of Probabilities?
Logits in LLMs

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What are Logits?

Logits are the raw, unnormalized scores that come out of the last layer of a model, one score for each possible answer.

Here, "unnormalized" means that these scores are not squeezed into any fixed range. A logit can be any number: positive, negative, small, or big. They also do not add up to 1.

Let's say we have a model that looks at a photo and tells us whether it is a cat, a dog, or a bird. For one photo, the model gives the following three numbers:

cat: 2.0
dog: 1.0
bird: 0.1

These three numbers are the logits.

Here, we can see that the cat has the highest score, so the model believes that the photo is most likely a cat. But we cannot say "the model is 2.0 sure". The number 2.0 is not a percentage. It is only a score that we can compare with the other scores.

To get real probabilities, we pass the logits through a function like softmax or sigmoid. We will learn both in this blog.

Where do Logits come from?

A neural network is made of layers. Each layer takes numbers in, multiplies them by its weights, adds a bias, and passes the result to the next layer. The weights and the bias are the numbers that the model learns during training.

The last layer has one output for each possible answer. If our model chooses between 3 answers (also called classes), the last layer gives 3 numbers. These numbers are the logits. For a yes or no question, the last layer gives only 1 number, which means only 1 logit.

Let's see this as a diagram:

┌────────────────────────────┐
│     Input: a cat photo     │
└────────────────────────────┘
              ↓
┌────────────────────────────┐
│       Hidden layers        │
└────────────────────────────┘
              ↓
┌────────────────────────────┐
│   Last layer: 3 outputs    │
└────────────────────────────┘
              ↓
┌────────────────────────────┐
│ Logits                     │
│ cat:  2.0                  │
│ dog:  1.0                  │
│ bird: 0.1                  │
└────────────────────────────┘
              ↓
┌────────────────────────────┐
│          Softmax           │
└────────────────────────────┘
              ↓
┌────────────────────────────┐
│ Probabilities              │
│ cat:  0.659                │
│ dog:  0.242                │
│ bird: 0.099                │
└────────────────────────────┘

Here, we can see that the logits sit between the last layer and the final probabilities. The last layer does not apply any activation function on its outputs. So, the logits are the plain result of multiplying by the weights and adding the bias.

Now, the next big question is: how do we turn these raw scores into probabilities? Let's understand that.

From Logits to Probabilities using Softmax

When a model picks one answer out of many, we use a function called softmax to convert logits into probabilities.

Softmax does two things:

It makes every number positive by raising e (a special number, about 2.718) to the power of each logit.
It divides each of those positive numbers by their total, so that all of them add up to 1.

The formula is:

softmax(z_i) = exp(z_i) / (exp(z_1) + exp(z_2) + ... + exp(z_n))

Here, z_i is the logit of answer i, n is the number of answers, and exp(z_i) means e raised to the power z_i.

The best way to learn this is by taking an example. Let's use our logits: cat 2.0, dog 1.0, bird 0.1.

Step 1: We raise e to the power of each logit.

cat: exp(2.0) = 7.389
dog: exp(1.0) = 2.718
bird: exp(0.1) = 1.105

Step 2: We add them up.

7.389 + 2.718 + 1.105 = 11.212

Step 3: We divide each one by the total.

cat: 7.389 / 11.212 = 0.659
dog: 2.718 / 11.212 = 0.242
bird: 1.105 / 11.212 = 0.099

Now, we have 0.659 + 0.242 + 0.099 = 1.000. These are real probabilities. Means, the model thinks there is a 65.9% chance that the photo is a cat.

One more thing to notice: the order never changes. The highest logit always gets the highest probability. Softmax only changes the scale, not the ranking.

Another important property: if we add the same number to all the logits, the probabilities do not change at all. For example, the logits 12.0, 11.0, and 10.1 also give 0.659, 0.242, and 0.099. Only the gaps between the logits matter.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

AI Engineer
Forward Deployed Engineer
AI Agent Engineer
Harness Engineer
Voice AI Engineer
AI Architect
LLM Engineer
LLM Inference Engineer
Gen AI Engineer
MLOps Engineer
ML Engineer
Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Logits for Yes or No Questions using Sigmoid

Some models answer only a yes or no question. For example, "Is this email spam?"

Such a model gives only one logit. To turn that one logit into a probability, we use a function called sigmoid.

sigmoid(z) = 1 / (1 + exp(-z))

Sigmoid squeezes any number into the range 0 to 1:

A logit of 0 gives a probability of 0.5. The model is unsure.
A big positive logit gives a probability close to 1. The model says yes.
A big negative logit gives a probability close to 0. The model says no.

Let's say our spam model gives a logit of 2.0. Then:

sigmoid(2.0) = 1 / (1 + exp(-2.0)) = 1 / (1 + 0.135) = 0.881

So, the model thinks there is an 88.1% chance that the email is spam.

If the logit is -1.0, then sigmoid(-1.0) = 0.269, which means only a 26.9% chance of spam.

Why is it called a Logit?

The name comes from the logit function in statistics. To understand it, we must first know what odds are.

Odds compare the chance of something happening with the chance of it not happening:

odds = p / (1 - p)

Here, p is the probability. Suppose the chance of rain tomorrow is 0.8. Then the chance of no rain is 0.2, and the odds are 0.8 / 0.2 = 4. Means, rain is 4 times as likely as no rain.

Now, the logit of a probability is the natural log of its odds. The natural log of a number answers one question: e raised to what power gives this number? For example, log(4) = 1.386, because exp(1.386) is about 4.

So, the formula is:

logit(p) = log(p / (1 - p))

So, logit = log + odds = log-odds. That is where the name comes from.

Let's put our rain example in it:

logit(0.8) = log(0.8 / 0.2) = log(4) = 1.386

Now, if we put 1.386 into sigmoid, we get back 0.8:

sigmoid(1.386) = 1 / (1 + exp(-1.386)) = 1 / (1 + 0.25) = 0.8

Let's see this as a diagram:

┌──────────────────────────┐
│     Probability: 0.8     │
└──────────────────────────┘
             ↓
      logit function
             ↓
┌──────────────────────────┐
│       Logit: 1.386       │
└──────────────────────────┘
             ↓
          sigmoid
             ↓
┌──────────────────────────┐
│     Probability: 0.8     │
└──────────────────────────┘

Here, we can see that sigmoid and the logit function undo each other. The logit function takes a probability and gives a score. Sigmoid takes that score and gives the same probability back.

This is why the raw score of a yes or no model is called a logit. It is exactly the log-odds of the "yes" answer.

A few useful values:

p = 0.5 gives logit 0 (equal chance)
p = 0.9 gives logit 2.197
p = 0.1 gives logit -2.197

Over time, the name "logits" was also used for the raw scores that go into softmax in models with many classes. There, a logit is not exactly the log-odds of one class, but the idea is the same: a raw score that a function turns into a probability.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

LLM
RAG
MCP
Agent
Fine-tuning
Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Why do Models output Logits instead of Probabilities?

Now, the question is: if we finally need probabilities, why does the model not give probabilities directly? There are two main reasons.

Reason 1: Numerical stability.

Computers store numbers with limited precision. The value of exp(1000) is too big for a computer to store, so it becomes infinity. In fact, many models use 32-bit numbers, and there, even exp(100) is too big. One infinity breaks the whole calculation.

When we keep the raw logits, we can apply a trick. We subtract the biggest logit from all the logits before doing exp. As we learned, adding or subtracting the same number from all the logits does not change the probabilities. But now, the biggest value going into exp is 0, and exp(0) = 1. So, no value becomes too big to store.

Let's see the code for the same:

import numpy as np

def softmax(logits):
    shifted = logits - np.max(logits)   # subtract the biggest logit
    exps = np.exp(shifted)
    return exps / np.sum(exps)

logits = np.array([2.0, 1.0, 0.1])     # cat, dog, bird
print(softmax(logits).round(3))

big_logits = np.array([1000.0, 999.0, 998.1])
print(softmax(big_logits).round(3))

This will print the following:

[0.659 0.242 0.099]
[0.659 0.242 0.099]

Here, we can see that both sets of logits give the same probabilities, because their gaps are the same. Without the subtraction, np.exp(1000.0) gives inf (infinity), and the result becomes nan, which means "not a number".

Reason 2: Loss functions take logits directly.

During training, we measure how wrong the model is using a loss function. For classification, the most common one is cross-entropy loss. It needs the log of the probability of the correct answer.

If we first do softmax and then take the log, there is a problem. A very tiny probability can get rounded to exactly 0 by the computer, and the log of 0 is minus infinity. Instead, the loss function takes the logits directly and combines softmax and log into one stable calculation:

log(softmax(z_i)) = z_i - log(exp(z_1) + exp(z_2) + ... + exp(z_n))

Here, the right side works directly on the logits. With the same trick of subtracting the biggest logit first, the sum inside the log is always at least 1, so the result never becomes minus infinity by mistake.

This is why, in popular libraries, the loss functions expect logits as input:

In PyTorch, nn.CrossEntropyLoss takes logits, and nn.BCEWithLogitsLoss takes the one logit of a yes or no model.
In TensorFlow, we pass from_logits=True to tell the loss that the input is logits.

Note: Many times, we do mistakes here by applying softmax in the model and then passing the result to a loss that already applies softmax. Softmax gets applied twice, and the model learns poorly.

There is one more practical benefit. When we only need the best answer, we can pick the highest logit directly. We do not need softmax at all, because softmax never changes the ranking.

To master Neural Networks, Logits, and Cross-Entropy, and to build a Neural Network from scratch, check out our AI and Machine Learning Program at Outcome School.

Logits in LLMs

Now, let's see how logits work in an LLM (Large Language Model).

An LLM generates text one token at a time. A token is a word or a piece of a word. The LLM knows a fixed list of tokens called its vocabulary.

To pick the next token, the LLM gives one logit for every token in its vocabulary. For example, if the vocabulary has 50,000 tokens, the LLM gives 50,000 logits at every step. A higher logit means that the LLM thinks that the token is a better fit as the next token.

Let's say the input is "The sky is". Let's see this as a diagram:

┌──────────────────────────────┐
│     Input: "The sky is"      │
└──────────────────────────────┘
               ↓
┌──────────────────────────────┐
│             LLM              │
└──────────────────────────────┘
               ↓
┌──────────────────────────────┐
│ 50,000 logits, one per token │
│ "blue"   :  2.0              │
│ "clear"  :  1.0              │
│ "dark"   :  0.1              │
│ "banana" : -5.0              │
│ ... and 49,996 more tokens   │
└──────────────────────────────┘
               ↓
┌──────────────────────────────┐
│ Temperature and top-k        │
│ (change the logits)          │
└──────────────────────────────┘
               ↓
┌──────────────────────────────┐
│ Softmax gives probabilities  │
└──────────────────────────────┘
               ↓
┌──────────────────────────────┐
│ Pick next token, like "blue" │
└──────────────────────────────┘

Here, we can see that the logits are the place where we can control how the LLM picks the next token. Two common controls act directly on the logits.

Temperature: We divide every logit by a number called the temperature before softmax. A temperature of 1.0 keeps the logits as they are.

Just for the sake of understanding, let's take a vocabulary of only three tokens with logits 2.0, 1.0, and 0.1.

Temperature	Logits after dividing	Probabilities
0.5	4.0, 2.0, 0.2	0.864, 0.117, 0.019
1.0	2.0, 1.0, 0.1	0.659, 0.242, 0.099
2.0	1.0, 0.5, 0.05	0.502, 0.304, 0.194

Here, we can notice that:

A low temperature makes the gaps between logits bigger, so the top token gets most of the probability. The output becomes more focused and predictable.
A high temperature makes the gaps smaller, so the probabilities become closer to each other. The output becomes more varied and creative.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Top-k: We keep only the k highest logits and set all the other logits to minus infinity. After softmax, minus infinity becomes a probability of 0, so those tokens can never be picked.

For example, with top-k = 2 on our "The sky is" example, only "blue" (2.0) and "clear" (1.0) stay. Softmax on 2.0 and 1.0 gives 0.731 and 0.269. A token that does not fit, like "banana", can never be picked.

We have a detailed blog on Top-K and Top-P Sampling that explains this step by step.

After this, the next token is picked at random using these probabilities, so a token with a higher probability gets picked more often. When we instead always pick the token with the highest logit, it is called greedy decoding. Again, we do not need softmax for this, because the highest logit is always the highest probability.

Now we must have understood what logits are, how they turn into probabilities, why models output them, and how LLMs use them to pick the next token.

Frequently Asked Questions

Can a logit be negative?

Yes. A logit can be any number: positive, negative, small, or big. Logits are not squeezed into any fixed range and they do not add up to 1. For a yes or no model, a big negative logit gives a probability close to 0 after sigmoid, which means the model says no.

What does a logit of 0 mean?

A logit of 0 means the model is unsure in a yes or no model. When we pass 0 through sigmoid, we get a probability of 0.5. In terms of odds, a probability of 0.5 means equal chance of yes and no, and the log of those odds is 0.

Should we apply softmax before passing logits to the loss function?

No. Loss functions like nn.CrossEntropyLoss in PyTorch take the raw logits and combine softmax and log into one stable calculation. If we apply softmax in the model and then pass the result to a loss that already applies softmax, softmax gets applied twice and the model learns poorly.

Do we need softmax to pick the most likely answer?

No. Softmax only changes the scale, not the ranking, so the highest logit always gets the highest probability. When we only need the best answer, we can pick the highest logit directly. This is also how greedy decoding works in an LLM: it always picks the token with the highest logit.

Does adding the same number to all logits change the probabilities?

No. If we add or subtract the same number from all the logits, the probabilities after softmax do not change at all, because only the gaps between the logits matter. This is why we can subtract the biggest logit before doing exp, so that no value becomes too big to store.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.