How does a confusion matrix work?
- Authors
- Name
- Amit Shekhar
- Published on
A confusion matrix is a small table that compares what a model predicted with the real answers, showing how often it was right and what kinds of mistakes it made.
In this blog, we will learn about how a confusion matrix works. We will also see why we need it, the four boxes TP, TN, FP, and FN, how to read it, how accuracy, precision, recall, and F1 score come from it, how it works for more than two classes, and how we build one in code.
We will cover the following:
- What is a confusion matrix?
- Why we need a confusion matrix
- The four boxes: TP, TN, FP, FN
- A real example: a spam filter
- How to read the matrix
- Accuracy and why it can fool us
- Precision: how trustworthy are the "yes" answers
- Recall: how many real cases did we catch
- F1 score: balancing precision and recall
- Confusion matrix for more than two classes
- How we build one in real tools
- Where a confusion matrix is used
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning at Outcome School.
Let's get started.
What is a confusion matrix?
A confusion matrix is a small table that compares what our model predicted with what the real answer actually was.
It shows us, in one picture, how many times the model was right and how many times it was wrong, and what kind of mistakes it made.
The word confusion is here because this table shows us where our model gets confused, which means where it makes mistakes.
It is used for a model that predicts categories, such as "spam" or "not spam", "cat" or "dog", "sick" or "healthy".
Why we need a confusion matrix
Suppose we build a model that looks at an email and predicts "spam" or "not spam".
After the model is ready, we test it on emails where we already know the true answer. Now we want to know one thing: how good is this model?
The simplest idea is to count how many predictions were correct. But this single number hides a lot.
For example, a model can be correct most of the time and still be useless. We will see exactly how this happens later in this blog.
So, we need something deeper than a single number. We need to see the different kinds of mistakes the model makes.
So, here comes the confusion matrix to the rescue.
It breaks down every prediction into clear groups, so we can see not just how often the model is wrong, but how it is wrong.
The four boxes: TP, TN, FP, FN
To read a confusion matrix, we must first understand four ideas.
In a yes-or-no problem, we pick one class as the positive class. The positive class is the thing we are trying to detect. For a spam filter, "spam" is the positive class.
Now, every prediction falls into one of four boxes.
True Positive (TP): The model said "yes", and the real answer was also "yes". The model said an email was spam, and it really was spam. This is correct.
True Negative (TN): The model said "no", and the real answer was also "no". The model said an email was not spam, and it really was not spam. This is correct.
False Positive (FP): The model said "yes", but the real answer was "no". The model said an email was spam, but it was actually a normal email. This is a mistake. It is also called a Type I error, which means a false alarm.
False Negative (FN): The model said "no", but the real answer was "yes". The model said an email was not spam, but it actually was spam. This is also a mistake. It is also called a Type II error, which means a missed catch.
Let's place all four boxes into one grid like below:
REAL ANSWER
Yes No
+----------------+----------------+
MODEL Yes | True Positive | False Positive |
SAID | (TP) | (FP) |
| correct | mistake |
+----------------+----------------+
No | False Negative | True Negative |
| (FN) | (TN) |
| mistake | correct |
+----------------+----------------+
Here, we can see how the model's prediction and the real answer meet in each box. When both agree, we land in a correct box (TP or TN). When they disagree, we land in a mistake box (FP or FN). The two correct boxes sit on the diagonal, and the two mistakes sit off the diagonal.
Here is a simple way to remember it. The first word tells us if the model was right or wrong. The second word tells us what the model predicted.
So, "False Positive" means the model predicted positive, and it was false, which means wrong.
Planning your move into AI?
No matter which tech domain you work in today, with your effort, you can transition to these roles:
- AI Engineer
- Forward Deployed Engineer
- AI Agent Engineer
- Harness Engineer
- Voice AI Engineer
- AI Architect
- LLM Engineer
- LLM Inference Engineer
- Gen AI Engineer
- MLOps Engineer
- ML Engineer
- Deep Learning Engineer
Our AI and Machine Learning Program at Outcome School takes you there.
A real example: a spam filter
The best way to learn this is by taking an example.
Let's say we test our spam filter on 100 emails. We already know the true answer for each one.
Out of these 100 emails, suppose 40 are really spam and 60 are really normal.
Now our model makes its predictions, and we count the results into the four boxes like below:
REAL ANSWER
Spam Not Spam
+---------------+---------------+
MODEL Spam | TP = 35 | FP = 5 |
SAID | (caught it) | (false alarm) |
+---------------+---------------+
Not | FN = 5 | TN = 55 |
Spam | (missed it) | (correct no) |
+---------------+---------------+
Here, we can see the four boxes filled with numbers. The model caught 35 real spam emails (TP). It wrongly flagged 5 normal emails as spam (FP). It missed 5 real spam emails and let them through (FN). And it correctly let 55 normal emails pass (TN).
Let's check the totals. The real spam emails are 35 + 5 = 40. The real normal emails are 5 + 55 = 60. This matches our 100 emails.
This table is our confusion matrix. Now let's learn how to read it.
How to read the matrix
A confusion matrix always has two directions.
One direction is the real answer, which is the truth we already know.
The other direction is the model's prediction, which is the guess the model made.
The boxes on the main diagonal, going from the top-left to the bottom-right, are the correct predictions. In our example, those are TP = 35 and TN = 55.
The other boxes, off the diagonal, are the mistakes. Those are FP = 5 and FN = 5.
Let's see this idea as below:
+---------------------+---------------------+
| CORRECT (TP) | MISTAKE (FP) |
| diagonal | off-diagonal |
+---------------------+---------------------+
| MISTAKE (FN) | CORRECT (TN) |
| off-diagonal | diagonal |
+---------------------+---------------------+
Diagonal = the model got it right
Off-diagonal = the model got it wrong
Here, we can see a simple rule. The numbers sitting on the diagonal are the correct predictions. The numbers sitting off the diagonal are the mistakes. A good model has big numbers on the diagonal and tiny numbers everywhere else.
This single rule works even when we have many classes, as we will see later.
A quick note for you
No matter which tech domain you work in, get familiar with these topics:
- LLM
- RAG
- MCP
- Agent
- Fine-tuning
- Quantization
We put it all together in one video:
AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization
No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.
Now, let's get back to the topic.
Accuracy and why it can fool us
Now, let's use these four numbers to measure the model.
The most common measure is accuracy.
Accuracy means the fraction of all predictions that were correct.
We calculate it like below:
Accuracy = (TP + TN) / (TP + TN + FP + FN)
= (35 + 55) / (35 + 55 + 5 + 5)
= 90 / 100
= 0.90 (which means 90%)
Here, we can see our model is correct 90% of the time. But accuracy alone can mislead us.
Suppose only 1 email out of 100 is spam, and 99 are normal. Now imagine a lazy model that simply says "not spam" for every single email.
This lazy model will be correct 99 times out of 100. So its accuracy is 99%. But it never catches a single spam email. It is completely useless.
This problem happens with imbalanced data, which means one class is far more common than the other.
So, accuracy alone is not enough. We need measures that look deeper.
Precision: how trustworthy are the "yes" answers
The first deeper measure is precision.
Precision answers one question: out of all the times the model said "yes", how many were actually correct?
When the model says "this is spam", how often should we believe it?
We calculate it using only the predicted-positive boxes, like below:
Precision = TP / (TP + FP)
= 35 / (35 + 5)
= 35 / 40
= 0.875 (which means 87.5%)
Here, we can see that when our model marks an email as spam, it is right about 87.5% of the time. The other 12.5% of the time, it wrongly flags a normal email.
Precision matters most when a false alarm is costly. For a spam filter, a false alarm means a real, important email gets sent to the spam folder, and the user may never see it. So we want precision to be high.
Recall: how many real cases did we catch
The next measure is recall.
Recall answers a different question: out of all the real "yes" cases, how many did the model actually catch?
Of all the spam emails that truly existed, how many did the model find?
We calculate it using only the real-positive boxes, like below:
Recall = TP / (TP + FN)
= 35 / (35 + 5)
= 35 / 40
= 0.875 (which means 87.5%)
Here, we can see our model caught 87.5% of all the real spam. The other 12.5% slipped through into the inbox.
Recall matters most when a miss is costly. Let's say the model detects a serious disease. Missing a sick patient (a false negative) can be very dangerous. So in that case, we want recall to be very high, even if it means a few more false alarms.
So, precision and recall look at the same model from two different angles. One asks "are your yes answers trustworthy", and the other asks "did you catch everything". We have a detailed blog on Precision vs Recall that explains when to use which one in depth.
Let's see exactly which boxes each one uses, like below:
PRECISION reads the ROW where the MODEL SAID "Yes":
+---------------+---------------+
MODEL Yes | TP = 35 | FP = 5 | ---> Precision = TP / (TP + FP)
SAID +---------------+---------------+
RECALL reads the COLUMN where the REAL ANSWER is "Yes":
REAL = Yes
+---------------+
MODEL Yes | TP = 35 |
SAID +---------------+ ---> Recall = TP / (TP + FN)
No | FN = 5 |
+---------------+
Here, we can see the difference at a glance. Precision moves across the row of everything the model called "yes", so it checks how many of those guesses were right. Recall moves down the column of everything that was truly "yes", so it checks how many of those real cases the model managed to catch. Both share the TP box, but each pairs it with a different mistake box.
To master Supervised and Unsupervised Learning, Logistic Regression, and Evaluation and Testing in depth, check out our AI and Machine Learning Program at Outcome School.
F1 score: balancing precision and recall
Often, when precision goes up, recall goes down, and the other way around.
If we make the model say "yes" very easily, it catches more real cases, so recall goes up. But it also raises more false alarms, so precision goes down.
If we make the model very strict, it raises fewer false alarms, so precision goes up. But it misses more real cases, so recall goes down.
So, we need a single number that balances both. That number is the F1 score.
The F1 score is the harmonic mean of precision and recall. A harmonic mean is a special kind of average that stays low if either of the two numbers is low. So, a model that is good at one but poor at the other gets a low F1 score.
We calculate it like below:
F1 = 2 * (Precision * Recall) / (Precision + Recall)
= 2 * (0.875 * 0.875) / (0.875 + 0.875)
= 1.53125 / 1.75
= 0.875 (which means 87.5%)
Here, we can see the F1 score is 87.5%. Because our precision and recall were equal, the F1 score lands at the same value. When they are different, the F1 score sits closer to the smaller one.
So, the F1 score gives us one balanced number to judge a model, especially when our data is imbalanced.
Confusion matrix for more than two classes
So far, we used a yes-or-no problem. But a confusion matrix also works when there are many classes.
Let's say we build a model that looks at a photo of an animal and predicts "Cat", "Dog", or "Bird".
Now the matrix becomes a bigger grid, with one row and one column for each class, like below:
REAL ANSWER
Cat Dog Bird
+-------+-------+-------+
MODEL Cat | 40 | 3 | 2 |
SAID +-------+-------+-------+
Dog | 4 | 35 | 1 |
+-------+-------+-------+
Bird | 2 | 2 | 38 |
+-------+-------+-------+
Here, we can see the same idea as before, just larger. The diagonal boxes (40, 35, 38) are the correct predictions. Every other box is a mistake.
Let's read one mistake. The number 3 in the top row means the model said "Cat" 3 times when the real animal was actually a Dog. This tells us the model sometimes confuses dogs with cats.
We can also measure precision and recall for each class on its own. For example, we can ask how good the model is at finding cats, then separately how good it is at finding dogs. The same four-box idea is applied to one class at a time.
A confusion matrix does not just say "the model was wrong". It tells us exactly which classes the model mixes up, so we know where to improve.
Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.
How we build one in real tools
In real machine learning work, we do not draw these tables by hand. We use ready-made tools.
A popular Python library for this is scikit-learn. It has a built-in function for this job.
We give it two lists. One list has the real answers. The other list has the model's predictions. The function then counts the four boxes for us.
We can write the code like below:
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
real_answers = ["spam", "spam", "not spam", "spam", "not spam"]
predictions = ["spam", "not spam", "not spam", "spam", "not spam"]
matrix = confusion_matrix(real_answers, predictions)
print(matrix)
# To see it as a picture:
ConfusionMatrixDisplay.from_predictions(real_answers, predictions)
This will print the following:
[[2 0]
[1 2]]
Here, we pass the real answers and the predictions, and confusion_matrix returns the grid of numbers. The grid above is the confusion matrix counted for us. In scikit-learn, the rows are the real answers and the columns are the predictions, and the classes are sorted alphabetically, so "not spam" comes first. The ConfusionMatrixDisplay part draws it as a colored picture, so we can read it at a glance. This same code works for two classes or for many classes.
Where a confusion matrix is used
Now, let's see where a confusion matrix is used.
A confusion matrix is used almost everywhere a model predicts categories. Here are a few examples:
- Email spam filters use it to balance catching spam against blocking real emails.
- Medical tests use it to make sure the model does not miss sick patients.
- Fraud detection in banks uses it to catch fraud while avoiding false alarms on honest customers.
- Image recognition uses it to see which objects a model confuses with each other.
- Text classification uses it when a model sorts text into categories, such as positive or negative reviews.
In every one of these cases, the confusion matrix helps us pick the right measure for the job, whether we care more about precision, recall, or a balance of both.
Now we must have understood how a confusion matrix works, how to read it, and how it helps us measure a model.
Frequently Asked Questions
What are Type I and Type II errors in a confusion matrix?
A Type I error is a False Positive, and a Type II error is a False Negative. A Type I error is a false alarm, where the model said "yes" but the real answer was "no". A Type II error is a missed catch, where the model said "no" but the real answer was "yes".
What does a good confusion matrix look like?
A good confusion matrix has big numbers on the main diagonal and tiny numbers everywhere else. The diagonal, going from the top-left to the bottom-right, holds the correct predictions. The boxes off the diagonal hold the mistakes. This rule works for two classes and for many classes.
Why does the F1 score use the harmonic mean?
The F1 score uses the harmonic mean because it stays low if either precision or recall is low. So, a model that is good at one but poor at the other gets a low F1 score. When precision and recall are different, the F1 score sits closer to the smaller one.
In scikit-learn, what do the rows and columns of a confusion matrix mean?
In scikit-learn, the rows are the real answers and the columns are the predictions. The classes are sorted alphabetically, so for a spam filter, "not spam" comes first. The library counts the boxes for us, and it can also draw the matrix as a colored picture.
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
You can connect with me on:
Follow Outcome School on:
