What is Entropy in Information Theory?

Entropy in information theory is a measure of uncertainty. It tells us how surprised we are, on average, by the outcome of something, so low entropy means easy to guess.

In this blog, we will learn about Entropy in Information Theory. We will also see the idea of surprise behind it, how we measure information in bits, the formula and each of its parts, a step-by-step example with a fair coin and a biased coin, and where entropy is used.

We will cover the following:

What is Entropy?
The simple idea of surprise
Measuring information in bits
The formula, decomposed
A step-by-step example
Low entropy vs high entropy
Where Entropy is used

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is Entropy?

Entropy is a measure of uncertainty. It tells us how surprised we are, on average, by the outcome of something.

Entropy measures how unpredictable a thing is. If we can easily guess the outcome, the entropy is low. If the outcome is very hard to guess, the entropy is high.

Let's say we have a coin that always lands on heads. There is no surprise at all, because we already know the answer. So the entropy is zero. Now let's say we have a fair coin that can land heads or tails equally. Now we cannot guess the outcome. So the entropy is high.

The simple idea of surprise

Before jumping into numbers, we must understand the idea of surprise.

An event that always happens gives us no surprise. If someone tells us "the sun rose today", we learn nothing new, because we already knew it.

An event that rarely happens gives us a big surprise. If someone tells us "it snowed in the desert today", we are very surprised, because it almost never happens.

So, surprise and probability are connected:

   High probability  ->  Low surprise   ->  Little information
   Low probability   ->  High surprise  ->  Lots of information

Entropy is the average surprise across all possible outcomes.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

AI Engineer
Forward Deployed Engineer
AI Agent Engineer
Harness Engineer
Voice AI Engineer
AI Architect
LLM Engineer
LLM Inference Engineer
Gen AI Engineer
MLOps Engineer
ML Engineer
Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Measuring information in bits

Now, we need a unit to measure information. In information theory, we measure it in bits.

One bit is the amount of information we get from one fair coin flip. A fair coin has two equally likely outcomes, heads or tails. Learning the result of one such flip gives us exactly one bit.

Here is a simple way to think about bits. When all outcomes are equally likely, the number of bits is the number of yes-or-no questions we need to ask to figure out the answer.

   2 equally likely outcomes  ->  1 question  ->  1 bit
   4 equally likely outcomes  ->  2 questions ->  2 bits
   8 equally likely outcomes  ->  3 questions ->  3 bits

Each yes-or-no question cuts the possibilities in half. So the number of bits grows slowly as the number of outcomes grows. This is why we use the logarithm with base 2 in the formula.

The formula, decomposed

Now, let's see the actual formula for entropy, written as H.

H = - sum over every outcome of (p * log2(p))

Let's break this down.

p is the probability of one outcome.
-log2(p) measures the surprise of that outcome. Rarer outcomes give a bigger surprise.
We multiply the surprise by its probability p, so common outcomes count more toward the average.
We add this up for every possible outcome.
The minus sign in front turns log2(p) into the surprise -log2(p). It makes the final number positive, because log2 of a probability is zero or negative.

So, in plain words, entropy is the average surprise, weighted by how often each outcome happens.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

LLM
RAG
MCP
Agent
Fine-tuning
Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

A step-by-step example

The best way to learn this is by taking an example.

Let's take a fair coin. It has two outcomes, each with probability 0.5.

Step 1: Write log2(p) for each outcome. log2(0.5) is -1.

Step 2: Multiply by the probability. For heads, 0.5 * (-1) = -0.5. For tails, 0.5 * (-1) = -0.5.

Step 3: Add them. -0.5 + (-0.5) = -1.

Step 4: Apply the minus sign in front. -(-1) = 1.

So the entropy of a fair coin is 1 bit. This matches our earlier idea that one fair coin flip gives one bit.

Now let's take a biased coin that lands heads 90% of the time and tails 10% of the time.

   Heads: 0.9 * log2(0.9) = 0.9 * (-0.152) = -0.137
   Tails: 0.1 * log2(0.1) = 0.1 * (-3.322) = -0.332
   Sum   = -0.469
   H     = -(-0.469) = 0.469 bits

The biased coin has only about 0.47 bits of entropy, which is much less than the fair coin's 1 bit. This makes sense, because the biased coin is easier to guess. We would bet on heads and be right most of the time.

Low entropy vs high entropy

Let's place the two coins side by side to see the pattern.

        FAIR COIN (50/50)                 BIASED COIN (90/10)

   Hard to guess the result           Easy to guess the result
   Maximum uncertainty                Low uncertainty
   Entropy = 1 bit                    Entropy = 0.47 bits

   |####################|             |#########           |
   full uncertainty                   less uncertainty

Here, we can see the rule. The more balanced the outcomes, the higher the entropy. The more one outcome dominates, the lower the entropy. Entropy is highest when everything is equally likely, because that is when we are the most unsure.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Where Entropy is used

Let's see where entropy helps us.

Data compression. Entropy tells us the smallest average number of bits needed per symbol to store a message. On average, we cannot compress below the entropy without losing information.
Decision trees. Machine learning models like decision trees use entropy to pick the question that reduces uncertainty the most at each step.
Machine learning loss. A very common loss function called cross-entropy comes directly from this idea. It measures how surprised a model is by the true answer.
Communication. Ideas built on entropy set the limit on how much information a channel can carry.

We have a detailed blog on KL Divergence that explains how entropy and cross-entropy are related.

Frequently Asked Questions

What is the entropy of a coin that always lands on heads?

The entropy is zero. If a coin always lands on heads, there is no surprise at all, because we already know the answer before the flip. Entropy measures how unpredictable an outcome is, so an outcome we can always guess has no uncertainty.

Why does the entropy formula use log base 2?

Because we measure information in bits. Each yes-or-no question cuts the possibilities in half, so the number of bits grows slowly as the number of outcomes grows. With log base 2, one fair coin flip with two equally likely outcomes gives exactly one bit.

When is entropy the highest?

Entropy is highest when every outcome is equally likely, because that is when we are the most unsure. The more balanced the outcomes, the higher the entropy. For example, a fair coin has 1 bit of entropy, while a coin that lands heads 90% of the time has only about 0.47 bits.

The minus sign makes the final number positive. The log2 of a probability is zero or negative, so without the minus sign the sum would never be positive. The minus sign turns log2(p) into the surprise -log2(p), which gets bigger as an outcome gets rarer.

Cross-entropy loss comes directly from the idea of entropy. Entropy is the average surprise across all possible outcomes. Cross-entropy, a very common loss function in machine learning, measures how surprised a model is by the true answer.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.