How does Object Detection work?
- Authors
- Name
- Amit Shekhar
- Published on
Object Detection is the task of finding objects in an image, drawing a box around each one, and naming each object, so it answers both what and where.
In this blog, we will learn about how Object Detection works. We will also see why the naive approach fails, how modern detection uses bounding boxes, anchor boxes, and confidence scores, how Non-Maximum Suppression and IoU clean up the extra boxes, how the model learns through training, and where object detection is used in the real world.
We will cover the following:
- What is Object Detection?
- Image Classification vs Object Detection
- What is a bounding box?
- The naive approach and why it fails
- How modern Object Detection works
- Anchor boxes and the grid
- Confidence score and class score
- The problem of many boxes
- Non-Maximum Suppression to the rescue
- IoU: measuring how much two boxes match
- How the model learns: training
- A simple worked example
- Where Object Detection is used
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning at Outcome School.
Let's get started.
What is Object Detection?
Object Detection is the task of finding objects in an image and drawing a box around each one while also naming each object.
Object Detection answers two questions at the same time:
- What objects are in the image?
- Where exactly is each object?
Let's say we give the computer a photo of a street. The computer should tell us, "There is a car here, a person there, and a traffic light over there." And for each one, it should point to the exact location.
Consider a self-driving car. Knowing only "there is a pedestrian somewhere" is not enough. The car needs the exact location to drive safely.
This is why Object Detection always gives us two things together, the name and the location.
Image Classification vs Object Detection
Before we go deeper, we must clear one common confusion.
Image Classification tells us only what is in the image, as a whole. It gives one label for the entire picture. For example, "This is a photo of a cat."
But Image Classification does not tell us where the cat is. And it cannot handle many objects. If there are three cats and one dog, one label cannot describe all of them.
Object Detection is the bigger task. It finds every object, names each one, and gives the location of each one.
Let me tabulate the differences between Image Classification and Object Detection for your better understanding.
| Image Classification | Object Detection |
|---|---|
| One label for the whole image | Many labels, one per object |
| Tells what, not where | Tells what and where |
| Cannot count objects | Can find and count many objects |
| Gives no box | Gives a box around each object |
Object Detection does more, so it needs more work. Let's understand how that work is done.
What is a bounding box?
The location of an object is given using a bounding box. A bounding box is a rectangle drawn tightly around an object.
A box is described by four numbers. There are different ways to write these numbers, but a common and simple way is like below:
x = 120 // distance from the left edge to the box's left side
y = 80 // distance from the top edge to the box's top side
width = 60 // how wide the box is
height = 90 // how tall the box is
Here, we can see that with just four numbers, we can describe exactly where an object sits in the image. The computer's job is to predict these four numbers correctly for every object.
So, the final output of Object Detection for one object looks like below:
label = "dog"
confidence = 0.95
box = (x=120, y=80, width=60, height=90)
Here, we have three things. The label is the name of the object. The confidence is how sure the model is, from 0 to 1. The box is the location.
This is the goal. Now, the question is, how does the computer reach this goal?
The naive approach and why it fails
The best way to learn this is by taking an example.
Let's say we already have a model that can look at a small picture and tell us what single object is inside it. This is the Image Classification model we discussed.
Now we want to use it for Object Detection. So, here is a simple idea.
We take a small box, place it in the top-left corner, and ask the classifier, "Is there an object here?" Then we slide the box a little to the right and ask again. We keep sliding it across the whole image. This is called the sliding window approach.
Objects come in many sizes. A small box will miss a big car. So, we must also try big boxes, medium boxes, tall boxes, and wide boxes. And we must slide each of them across every position.
The issue with this approach is speed. We end up asking the classifier thousands or even millions of times for a single image. This is far too slow for real use, especially for a video, where we must handle many images every second.
We needed a faster solution for that, and modern Object Detection was introduced to solve this problem.
Planning your move into AI?
No matter which tech domain you work in today, with your effort, you can transition to these roles:
- AI Engineer
- Forward Deployed Engineer
- AI Agent Engineer
- Harness Engineer
- Voice AI Engineer
- AI Architect
- LLM Engineer
- LLM Inference Engineer
- Gen AI Engineer
- MLOps Engineer
- ML Engineer
- Deep Learning Engineer
Our AI and Machine Learning Program at Outcome School takes you there.
How modern Object Detection works
So, here comes the modern Object Detection model to the rescue.
The central idea is that instead of running a classifier millions of times, the model looks at the whole image only once and predicts all the boxes in one go.
The model is a neural network, which learns patterns from many photos along with the correct answers. The kind of neural network used for images is a Convolutional Neural Network, often called a CNN.
A CNN scans the image in small parts and learns to recognize useful patterns. The early parts of the network find simple things like edges and corners. The later parts combine these into bigger things like eyes, wheels, and faces.
So, the CNN turns the raw pixel numbers of the image into a smaller, meaningful summary. This summary holds clues like "there is something round here" or "there is a furry texture there."
Now, on top of this summary, the model predicts the boxes and the labels. Let's understand exactly how.
Anchor boxes and the grid
To predict boxes without sliding millions of times, many modern detectors, like the YOLO family, use two ideas, a grid and anchor boxes.
First, the model divides the image into a grid. Suppose we split the image into a 13 by 13 grid. That gives us 169 small cells.
Each cell is responsible for detecting objects whose center falls inside that cell. So, every part of the image has a cell looking after it.
Now, each cell faces one problem. An object inside it could be tall like a standing person, or wide like a car. One guess per cell is not enough.
So, here come anchor boxes into the picture.
An anchor box is a box shape decided in advance that the model uses as a starting guess, like a tall rectangle, a wide rectangle, or a square.
Let's say each cell uses 3 anchor boxes. The model does not predict a box from nothing. Instead, it starts from each anchor and predicts a small adjustment, like "make it a bit wider" or "shift it slightly up."
This is much easier for the model. Adjusting a good starting guess is simpler than guessing from scratch.
Let's visualize this as below:
The image is split into a grid of cells:
+-----+-----+-----+-----+
| | | | |
+-----+-----+-----+-----+
| | dog | | |
+-----+-----+-----+-----+
| | | car | |
+-----+-----+-----+-----+
Inside one cell, 3 anchor boxes act as starting guesses:
+---------+ +----+ +------------+
| | | | | |
| square | |tall| | wide |
| | | | +------------+
+---------+ | |
+----+
Here, we can see that the image is divided into cells, and the cell that holds an object's center, like the dog's center, is responsible for it. Inside that cell, the model starts from a few ready-made anchor shapes, a square one, a tall one, and a wide one, and then nudges the closest one to fit the object.
Confidence score and class score
For every anchor box in every cell, the model predicts a few values together.
First, it predicts the four box numbers, which are the adjusted x, y, width, and height.
Then, it predicts a confidence score. This score, from 0 to 1, tells how sure the model is that a real object is present in this box.
After that, it predicts a class score for each possible object type. If the model knows 80 object types, it gives 80 scores, one for "dog", one for "car", and etc. The highest one becomes the label.
So, for each anchor box, the output looks like below:
box = (x, y, width, height) // where the object is
confidence = 0.92 // how sure an object is here
class_scores = // what the object is, a score per type
dog: 0.88
cat: 0.05
car: 0.01
// and etc.
Here, we can see that one anchor box gives us a full guess. It tells where, how sure, and what. The highest class score wins, so this guess is labeled a dog. The model produces all of these for every anchor in every cell, in one single pass over the image.
A quick note for you
No matter which tech domain you work in, get familiar with these topics:
- LLM
- RAG
- MCP
- Agent
- Fine-tuning
- Quantization
We put it all together in one video:
AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization
No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.
Now, let's get back to the topic.
The problem of many boxes
Now we have a new problem.
Let's count. We had a 13 by 13 grid, which is 169 cells. Each cell has 3 anchor boxes. So, the model gives 169 times 3, which is 507 boxes.
That is 507 guesses for one image. Most of them are useless. Many will have a very low confidence score, like 0.02, which means there is almost nothing there.
Also, one real object often gets covered by many overlapping boxes. Suppose there is one dog. Five nearby boxes may all claim, "I found the dog." But we want only one clean box around the dog.
So, we must clean up these boxes. We must do this in two steps.
First, we throw away every box with a low confidence score. Let's say we keep only boxes above 0.5. This removes most of the useless guesses.
Then, we still have the problem of many boxes around the same object. We need a smarter step for that.
Non-Maximum Suppression to the rescue
So, here comes Non-Maximum Suppression to the rescue. It is often shortened to NMS.
Non-Maximum Suppression keeps the best box for each object and removes the weaker boxes that overlap it.
Let me explain the idea with a step-by-step walkthrough. Suppose we have five boxes left, all around the same dog, with these confidence scores:
Box A: 0.95
Box B: 0.91
Box C: 0.88
Box D: 0.60
Box E: 0.55
We follow these steps.
Step 1: We pick the box with the highest confidence. That is Box A with 0.95. We keep it as a winner.
Step 2: We compare every other box with Box A. If a box overlaps Box A heavily, it is most likely the same dog, so we remove it. Box B, C, D, and E all overlap Box A, so we remove all of them.
Step 3: If any boxes were left for a different object somewhere else, we repeat the process for them.
After Non-Maximum Suppression, only Box A remains for the dog. We now have one clean box per object.
But there is one word we used without explaining, "overlap heavily." How does the model measure overlap? We need a number for that.
IoU: measuring how much two boxes match
To measure overlap, we use Intersection over Union, often called IoU.
IoU tells us how much two boxes match. It is a number from 0 to 1. A value of 1 means the two boxes are exactly the same. A value of 0 means they do not touch at all.
The formula is as below:
IoU = Area of Overlap / Area of Union
Here, the Area of Overlap is the region where both boxes cover the same part. The Area of Union is the total region covered by both boxes together.
We can picture two overlapping boxes as below:
+-------------+
| Box 1 |
| +----+--------+
| |####| |
| |####| Box 2 |
+--------+----+ |
| |
+-------------+
#### = Area of Overlap (where both boxes share the same part)
Area of Union = Box 1 + Box 2 minus the shared part
Here, we can see that the #### region is where both boxes cover the same part of the image, which is the Area of Overlap. The Area of Union is everything both boxes cover together, counting the shared part only once. The more the two boxes sit on top of each other, the bigger the overlap, and the closer the IoU gets to 1.
Let's take a small numeric example. Suppose two boxes overlap in a region of 30 square units, and together they cover 100 square units.
IoU = 30 / 100 = 0.3
Here, we can see that an IoU of 0.3 means the shared part is only 30% of the total area that the two boxes cover. In Non-Maximum Suppression, if the IoU between two boxes is above a chosen value, like 0.5, we treat them as the same object and remove the weaker one.
This is how the model decides that two boxes point to the same object.
Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.
How the model learns: training
We have talked a lot about the model predicting boxes. But how does it learn to predict them correctly in the first place?
This step is called training.
To train, we need a dataset. A dataset is a large collection of images where humans have already drawn the correct boxes and written the correct labels. These human-made answers are called the ground truth. We have a detailed blog on Supervised vs Unsupervised Learning that explains learning from labeled data in depth.
Now, the training happens like below.
First, we show the model an image from the dataset. The model makes its guesses, which are boxes, confidence scores, and labels.
Then, we compare the model's guesses with the ground truth. We use IoU here too, to check how well a predicted box matches the correct box. We measure the total mistake using a number called the loss. The loss is high when the model is wrong and low when the model is right.
After that, the model adjusts its internal values a tiny bit to reduce the loss. This adjustment is done using a method called backpropagation, which is the standard way neural networks learn from their mistakes.
Finally, we repeat this for millions of images, again and again. Slowly, the loss goes down, and the model gets better at putting the right box around the right object.
This is how the model learns. After enough training, it can detect objects in new images it has never seen before.
To master Neural Networks, Loss Functions, and Backpropagation, and to build a Neural Network from scratch, check out our AI and Machine Learning Program at Outcome School.
A simple worked example
Let's put the whole flow together with one example so that everything connects.
Suppose we give the trained model a photo of a street with one dog and one car.
Step 1: The model sends the image through the CNN. The raw pixels become a meaningful summary of patterns.
Step 2: Using the grid and anchor boxes, the model produces 507 guesses. Each guess has a box, a confidence score, and class scores.
Step 3: We remove every box with a confidence score below 0.5. Now only a handful of boxes remain.
Step 4: We run Non-Maximum Suppression using IoU. The overlapping duplicate boxes are removed.
We can picture the whole flow as below:
Raw image (table of pixel numbers)
↓
+--------------------------+
| CNN |
| (reads the image once) |
+--------------------------+
↓
Grid + anchor boxes
produce 507 guesses
(box + confidence + class scores)
↓
Remove boxes with confidence < 0.5
↓
Non-Maximum Suppression using IoU
(keep one best box per object)
↓
Clean results: dog box + car box
Here, we can see the full journey from raw pixels to clean answers. The CNN reads the image once, the grid and anchor boxes produce many guesses, the low confidence guesses are dropped, and Non-Maximum Suppression removes the overlapping duplicates so that one clean box is left per object.
Finally, we are left with two clean results as below:
Result 1:
label = "dog"
confidence = 0.96
box = (x=120, y=200, width=80, height=70)
Result 2:
label = "car"
confidence = 0.93
box = (x=400, y=150, width=220, height=140)
Here, we can see the final answer. The model found the dog and the car, named each one, and gave the exact location of each. This is exactly the goal we set at the start.
Where Object Detection is used
Now, let's see where Object Detection is used in the real world.
- Self-driving cars detect pedestrians, vehicles, and traffic signs.
- Phone cameras detect faces to focus the lens correctly.
- Security cameras detect people and unusual movement.
- Shops use it to count products on shelves.
- Doctors use it to find tumors and other problems in medical scans.
- Factories use it to spot defective items on a conveyor belt.
Well-known Object Detection models like YOLO, which means "You Only Look Once", follow the core ideas we learned here: looking at the image, predicting many boxes, scoring them, and cleaning them up. Newer models change some parts, for example, many of them predict boxes directly without anchor boxes.
Now, we have understood how Object Detection works.
Frequently Asked Questions
What does YOLO stand for in Object Detection?
YOLO stands for "You Only Look Once". It is a well-known family of Object Detection models. As the name says, the model looks at the whole image only once and predicts all the boxes in one go, instead of running a classifier thousands or even millions of times like the sliding window approach.
Why do modern detectors use anchor boxes?
Anchor boxes give the model a good starting guess. An anchor box is a box shape decided in advance, like a tall rectangle, a wide rectangle, or a square. The model does not predict a box from nothing. It predicts a small adjustment to the anchor, which is much easier than guessing from scratch.
What IoU value is used in Non-Maximum Suppression?
A common choice is 0.5. If the IoU between two boxes is above this chosen value, the two boxes are treated as the same object, and the weaker box is removed. This leaves one clean box per object.
Do all Object Detection models use anchor boxes?
No. Many modern detectors, like the YOLO family, use a grid and anchor boxes. But newer models change some parts, and many of them predict boxes directly without anchor boxes. They still follow the core ideas of looking at the image, predicting many boxes, scoring them, and cleaning them up.
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
You can connect with me on:
Follow Outcome School on: