How does Semantic Segmentation work?
- Authors
- Name
- Amit Shekhar
- Published on
Semantic Segmentation labels every single pixel in an image with a class, using an encoder that understands the image and a decoder that paints a label on each pixel.
In this blog, we will learn about how Semantic Segmentation works. We will also see how it differs from Image Classification and Object Detection, why we need it, how an encoder and a decoder label every pixel in an image, how the model learns, and where it is used in the real world.
We will cover the following:
- What is Semantic Segmentation?
- Image Classification vs Object Detection vs Semantic Segmentation
- Why do we need Semantic Segmentation?
- How does Semantic Segmentation work step by step?
- What is an Encoder and a Decoder?
- A simple worked example with numbers
- How does the model learn?
- Real-world use cases
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning at Outcome School.
Let's get started.
What is Semantic Segmentation?
Semantic Segmentation is the task of labeling every single pixel in an image with a class.
An image is a grid of tiny dots called pixels, and for a computer, each pixel is just numbers for its color.
The word Semantic means meaning: what does this thing represent? Is it a cat, a road, or the sky? So, Semantic Segmentation divides the image into meaningful parts and gives each part a name.
The name we give to a pixel is called a class. A class is just a category. For example, the classes can be cat, road, sky, tree, and person. Giving a pixel its class is also called giving it a label. So, a label and a class mean the same thing here.
Let's say we have a photo of a street. After Semantic Segmentation, every pixel will get a label. The pixels of the road become road. The pixels of the sky become sky. The pixels of a person become person.
So, the model does not just say "there is a person in this image". It tells us exactly which pixels are the person.
Now, let's compare it with other tasks.
Image Classification vs Object Detection vs Semantic Segmentation
There are three common tasks in computer vision. Let's understand all three with one simple example.
Consider a photo that has a dog sitting on grass.
Image Classification answers one question: what is in this image? It gives one label for the whole image. The answer is dog. It does not tell us where the dog is.
Object Detection goes one step further. It draws a box around the dog. This box is called a bounding box. A bounding box is a rectangle that surrounds an object. So, now we know there is a dog and roughly where it is. But the box also covers some grass in the corners. So, it is not exact.
Semantic Segmentation is the most detailed. It does not draw a rough box. It colors every pixel that belongs to the dog as dog, and every pixel of the grass as grass. So, we get the exact shape of the dog, not a rough rectangle.
Let's visualize the three tasks side by side as below:
Image Classification Object Detection Semantic Segmentation
-------------------- ---------------- ---------------------
+----------------+ +----------------+ +----------------+
| | | +------+ | | dddd |
| a dog | | | dog | | | dddddd |
| | | +------+ | | dddd |
+----------------+ +----------------+ +----------------+
Output: "dog" Output: a box Output: every pixel
(one label for the around the dog labeled (d = dog,
whole image) (covers some grass) blank = grass)
Here, we can see that Image Classification gives only one label for the whole image, Object Detection gives a rough box that also covers some grass, and Semantic Segmentation labels each pixel so we get the exact shape of the dog.
Let me tabulate the differences between these three for your better understanding.
| Task | What it tells us | Output |
|---|---|---|
| Image Classification | What is in the image | One label for the whole image |
| Object Detection | What is in the image and roughly where | Boxes around objects |
| Semantic Segmentation | The exact shape of every object | A label for every pixel |
Semantic Segmentation gives us the most detailed understanding of an image.
Note: Semantic Segmentation does not separate two objects of the same class. If there are two dogs touching each other, all their pixels are labeled dog together. Telling dog 1 apart from dog 2 is a different task called Instance Segmentation. We will keep our focus on Semantic Segmentation in this blog.
Planning your move into AI?
No matter which tech domain you work in today, with your effort, you can transition to these roles:
- AI Engineer
- Forward Deployed Engineer
- AI Agent Engineer
- Harness Engineer
- Voice AI Engineer
- AI Architect
- LLM Engineer
- LLM Inference Engineer
- Gen AI Engineer
- MLOps Engineer
- ML Engineer
- Deep Learning Engineer
Our AI and Machine Learning Program at Outcome School takes you there.
Why do we need Semantic Segmentation?
Now, the question is, why do we even need this?
Let's say we are building a self-driving car. The car must know exactly where the road ends and where the footpath begins. A rough box is not enough here. The car needs the exact shape of the road, the exact shape of each person, and the exact shape of other cars.
So, here comes Semantic Segmentation to the rescue. It gives the car a pixel-perfect map of the scene.
Consider one more example. A doctor wants to measure the exact area of a tumor in a medical scan. A rough box around it is not useful. The doctor needs the exact shape. Semantic Segmentation gives exactly that.
Now that we know why we need it, it's time to learn how it actually works.
How does Semantic Segmentation work step by step?
The best way to learn this is by taking an example.
Assume that we give an image of size 4 by 4 to our model. This image is just a grid of numbers.
The job of the model is to produce another grid of the same size. But this new grid does not hold colors. It holds a class label for each pixel.
Let's say we have only two classes for now, cat and background. The background is everything that is not the cat.
So, the input is a grid of pixels, and the output is a grid of labels.
Here is a tiny example of what the output looks like for our 4 by 4 image, where C means cat and B means background:
B B B B
B C C B
B C C B
B B B B
Here, we can see that the four pixels in the middle are labeled cat. All the other pixels are labeled background. So, the model has found the exact shape of the cat, which is the middle square.
This output grid is called a segmentation map. We can show a segmentation map as a picture where each pixel is painted with the color of its class.
Now, the big question is, how does the model turn the input grid of pixels into this output grid of labels? The answer is the Encoder and the Decoder.
What is an Encoder and a Decoder?
A Semantic Segmentation model has two main parts. They work one after the other.
The Encoder is the first part. Its job is to understand the image.
The Encoder slowly shrinks the image and, at the same time, learns what objects are present.
Let's understand this with an analogy. Suppose we look at a big photo from very close. We see tiny details like single hairs, but we lose the full picture. Now, suppose we step back. We lose the tiny details, but we clearly understand "this is a cat".
The Encoder does the same thing. It steps back step by step. The image becomes smaller in size, but richer in meaning. After the Encoder, the model knows a cat is present, but it has lost the exact location of every pixel because the image is now very small.
We have a detailed blog on Convolutional Neural Networks that explains how a neural network reads an image and turns it into a small, meaningful summary.
Our final answer must be the same size as the original image. We must label every original pixel. A small grid is not enough.
So, here comes the Decoder into the picture.
The Decoder is the second part. Its job is to grow the small grid back to the full original size.
The Decoder takes the small, meaning-rich grid from the Encoder and slowly enlarges it. While enlarging, it places the correct label on each pixel. So, the Decoder rebuilds the full-size image, but now as a segmentation map of labels.
Let me tabulate the roles for your better understanding.
| Part | Job | Effect on image size |
|---|---|---|
| Encoder | Understand what is in the image | Makes it smaller |
| Decoder | Rebuild the full picture with labels | Makes it bigger again |
So, the Encoder asks "what is in this image?" and the Decoder asks "where exactly is each thing?". Together they give us a pixel-by-pixel answer.
This shape, where the image first shrinks and then grows back, looks like the letter U. This is why one well-known Semantic Segmentation model is called U-Net.
We can picture this shrink-then-grow flow in a diagram like below:
Input image (full size) Segmentation map (full size)
↓ ↑
+-----------+ +-----------+
| Encoder | | Decoder |
| shrinks | | grows |
+-----------+ +-----------+
↓ ↑
+-----------+ +-----------+
| Encoder | | Decoder |
| shrinks | | grows |
+-----------+ +-----------+
↓ ↑
+-----> small, meaning-rich grid ------+
("a cat is present")
Here, we can see that the left side is the Encoder shrinking the image step by step until it becomes a small grid that holds the meaning "a cat is present", and the right side is the Decoder growing that grid back to the full size as a segmentation map. The two sides together form the U shape.
A quick note for you
No matter which tech domain you work in, get familiar with these topics:
- LLM
- RAG
- MCP
- Agent
- Fine-tuning
- Quantization
We put it all together in one video:
AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization
No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.
Now, let's get back to the topic.
A simple worked example with numbers
Let's do a step-by-step numeric walkthrough so this becomes concrete.
Assume that for one single pixel, the model has to decide between two classes, cat and background. The model does not directly say "cat". Instead, it produces a score for each class.
Let's say for this pixel the model produces two scores like below:
cat = 8.0
background = 2.0
Here, we can see that the score for cat is higher than the score for background. But these raw scores are hard to read. We want them as probabilities, which means values between 0 and 1 that add up to 1.
So, here comes Softmax to the rescue. The model passes these scores through a step called Softmax. Softmax turns scores into probabilities, where a bigger score becomes a bigger probability.
After Softmax, our two scores become probabilities like below:
cat = 0.9975
background = 0.0025
Here, we can see that the probability of cat is 0.9975 and the probability of background is 0.0025. Both add up to 1. The model picks the class with the highest probability. So, this pixel is labeled cat.
The model does this for every single pixel in the image. Once every pixel has a label, our segmentation map is ready.
This is how the model decides, pixel by pixel, what each part of the image is.
Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.
How does the model learn?
Now, it's time to learn how the model became good at this in the first place. How does the model know that a cat pixel is a cat?
The model is not born with this knowledge. It learns from examples. This learning process is called training.
For training, we need labeled data. Labeled data means images where a human has already painted the correct label on every pixel. This correct answer is called the ground truth.
Let's understand the training loop with a simple flow.
First, we give an image to the model. Then, the model produces its own segmentation map, which is its guess. After that, we compare the model's guess with the ground truth. The difference between the guess and the correct answer is called the loss. A big loss means the model made a big mistake.
Finally, the model adjusts itself a little to reduce this loss. We repeat this for thousands of images, again and again.
We have a detailed blog on How Does Gradient Descent Work? that explains this adjustment step by step.
We can picture this training loop as below:
+-------------------------+
| Image |
+-------------------------+
↓
+-------------------------+
| Model | <-------+
+-------------------------+ |
↓ |
+-------------------------+ |
| Model's guess | |
+-------------------------+ |
↓ |
+-------------------------+ |
| Compare with ground | |
| truth to get the loss | |
+-------------------------+ |
↓ |
+-------------------------+ |
| Adjust the model to | --------+
| reduce the loss |
+-------------------------+
Here, we can see that the image goes into the model, the model produces its guess, we compare that guess with the ground truth to get the loss, and then the model adjusts itself to reduce the loss. This loop runs again and again until the loss becomes small.
Slowly, the loss becomes smaller and smaller. The model's guesses get closer and closer to the ground truth. After enough training, the model becomes good at labeling pixels it has never seen before.
This is how the model learns Semantic Segmentation, and that too without anyone writing a rule for every shape. It learns only by looking at examples.
To master Loss Functions, Gradient Descent, and Backpropagation, which are the core of how every such model learns, check out our AI and Machine Learning Program at Outcome School.
Real-world use cases
Now, let's see where Semantic Segmentation is used in the real world.
- Self-driving cars: to find the exact shape of the road, people, and other vehicles.
- Medical scans: to mark the exact area of a tumor or an organ in an X-ray, MRI, or CT scan.
- Satellite images: to mark exactly which area is water, forest, road, or building.
- Photo editing apps: to separate the person from the background so we can blur or change the background.
- Video calls: to find the exact shape of a person so the app can blur the room behind them.
- Farming: to find the exact area of crops, soil, and weeds in a field image.
Semantic Segmentation is useful anywhere we need the exact shape of things, not just a rough idea.
Now we must have understood how Semantic Segmentation works.
Frequently Asked Questions
What is the difference between Semantic Segmentation and Instance Segmentation?
Semantic Segmentation does not separate two objects of the same class. If two dogs touch each other, all their pixels are labeled dog together. Instance Segmentation is a different task that tells dog 1 apart from dog 2. So, Semantic Segmentation tells us what each pixel is, not which individual object it belongs to.
Why is it called U-Net?
U-Net is a well-known Semantic Segmentation model. Its Encoder first shrinks the image step by step, and its Decoder then grows it back to the full original size. When we draw this shrink-then-grow flow, it looks like the letter U, which is where the name comes from.
What is the output of a Semantic Segmentation model?
The output is a segmentation map. It is a grid of the same size as the input image, where every pixel holds a class label instead of a color. We can show it as a picture where each pixel is painted with the color of its class, so we see the exact shape of every object.
What training data does Semantic Segmentation need?
It needs labeled data, which means images where a human has already painted the correct label on every pixel. This correct answer is called the ground truth. The model compares its guess with the ground truth to get the loss, and adjusts itself to reduce that loss over thousands of images.
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
You can connect with me on:
Follow Outcome School on:
