Convolutional Neural Networks

Convolutional Neural Networks (CNNs) are neural networks built to understand images. They slide small pattern detectors over a picture to find edges, shapes, and objects, step by step.

In this blog, we will learn about Convolutional Neural Networks. We will also see why a normal neural network struggles with images, how a filter slides over an image to find patterns, what feature maps, stride, padding, and pooling are, how many layers work together to go from edges to whole objects, how to write a small CNN in code, and where CNNs work well and where they fail.

We will cover the following:

• How does a computer see an image?
• Why does a normal neural network struggle with images?
• What is a Convolutional Neural Network?
• What is a filter and how does convolution work?
• What is a feature map?
• Stride and padding
• What is ReLU?
• What is pooling?
• Putting the layers together
• How does a CNN learn its filters?
• Why are CNNs so efficient?
• A simple CNN in code
• Famous CNNs
• Where CNNs work well and where they fail

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

How does a computer see an image?

Before jumping into Convolutional Neural Networks, we must know how a computer sees an image, because a CNN works on images.

When we look at a photo of a cat, we see a cat. But a computer does not see a cat. It sees only numbers.

An image is made of tiny colored squares called pixels. If we zoom into any photo on our phone, we will see these small squares.

In a black and white image, each pixel is one number from 0 to 255. Here, 0 means black, 255 means white, and the numbers in between are shades of gray.

Let's say we have a tiny 5 x 5 black and white image. The computer sees it like below:

  0    0    0  255  255
  0    0    0  255  255
  0    0    0  255  255
  0    0    0  255  255
  0    0    0  255  255

Here, the left part is black and the right part is white. There is a straight vertical line where black meets white. This line is called an edge.

In a color image, each pixel has three numbers: one for red, one for green, and one for blue. These three layers of numbers are called channels. So, a color photo of size 1000 x 1000 is actually 1000 x 1000 x 3 = 3,000,000 numbers.

So, for a computer, "understanding an image" means finding meaning inside a big grid of numbers.

Why does a normal neural network struggle with images?

In a normal neural network, every neuron in a layer is connected to every number from the previous layer, and each connection has its own importance value called a weight. This is called a fully connected layer.

Let's see what happens if we give our 1000 x 1000 color photo to a fully connected layer.

• The input has 3,000,000 numbers.
• Each neuron in the first layer needs one weight for each input number. That means 3,000,000 weights for just one neuron.
• If the first layer has 1000 neurons, we need 3,000,000 x 1000 = 3,000,000,000 weights.

Three billion weights in just the first layer. This is very slow, needs huge memory, and needs a huge amount of data to train.

But the following issues also arise:

• It ignores the neighbors. In an image, pixels that are close to each other belong together. A fully connected layer treats the image like a long flat list of numbers, so it loses the idea of "this pixel is next to that pixel".
• It does not reuse what it learns. Suppose the network learns to find a cat's eye in the top-left corner. If the eye appears in the bottom-right corner, it has to learn it again with different weights.

We needed a solution for that, and Convolutional Neural Networks were introduced to solve this problem.

What is a Convolutional Neural Network?

A Convolutional Neural Network (CNN) is a neural network that looks at an image through small windows and uses the same small pattern detector everywhere in the image.

Here, convolution is the name of the operation where we slide a small window over the image and calculate a value at every position. We will understand it in detail with numbers in a moment.

Think of how we search for our friend in a big crowd photo. We do not look at the whole photo at once. We move our eyes from one small area to the next and check, "Is my friend here?". We use the same idea of our friend's face in every area. A CNN does exactly this.

What is a filter and how does convolution work?

A filter is a small grid of numbers that acts as a pattern detector. It is also called a kernel.

A common filter size is 3 x 3. Let's take a filter that detects vertical edges:

 -1   0   1
 -1   0   1
 -1   0   1

Here, the left column is negative and the right column is positive. So, this filter gives a big number when the right side of a small area is brighter than the left side. That is exactly what a vertical edge looks like.

Now, let's apply this filter on our 5 x 5 image. For the sake of understanding, let's use 9 instead of 255 to keep the math simple:

 0  0  0  9  9
 0  0  0  9  9
 0  0  0  9  9
 0  0  0  9  9
 0  0  0  9  9

Step 1: Place the 3 x 3 filter on the top-left corner of the image. It covers the first three columns:

 0  0  0
 0  0  0
 0  0  0

Multiply each image number with the filter number at the same place and add everything. Every image number here is 0, so every product is 0.

The result is 0. Means, no edge here. It is all black.

Step 2: Slide the filter one step to the right. Now it covers columns 2, 3, and 4:

 0  0  9
 0  0  9
 0  0  9

Multiply and add:

(0 x -1) + (0 x 0) + (9 x 1)  = 9
(0 x -1) + (0 x 0) + (9 x 1)  = 9
(0 x -1) + (0 x 0) + (9 x 1)  = 9
Total = 27

The result is 27. A big number. Means, there is a vertical edge here.

Step 3: Slide one more step to the right. Now it covers columns 3, 4, and 5:

 0  9  9
 0  9  9
 0  9  9

Each row gives (0 x -1) + (9 x 0) + (9 x 1) = 9, so the total is again 27.

Then we move the filter down one row and repeat the same thing. Since every row of our image is the same, we get the same values again.

This sliding, multiplying, and adding is called convolution.

What is a feature map?

When we finish sliding the filter over the whole image, we get a new grid of numbers:

 0  27  27
 0  27  27
 0  27  27

This new grid is called a feature map. It shows where in the image the filter found its pattern.

Here, we can see that the big numbers appear exactly where the black part meets the white part. The filter has found the edge.

Note: The feature map is 3 x 3, smaller than the 5 x 5 image. This is because a 3 x 3 filter can fit in only 3 positions across a row of 5 pixels.

A real CNN layer does not use just one filter. It uses many filters, for example 32 or 64. One filter finds vertical edges, another finds horizontal edges, another finds a certain color, and so on. Each filter gives its own feature map. So, 32 filters give 32 feature maps stacked together.

Also, on a color image, each filter is actually 3 x 3 x 3, so that it looks at the red, green, and blue channels together.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Stride and padding

Now, let's understand two small settings of a convolution.

Stride is how many pixels the filter moves at each step. In our example, the stride was 1, because we moved one pixel at a time. If the stride is 2, the filter jumps two pixels at a time, and the feature map becomes roughly half the size.

Padding means adding a border of zeros around the image before sliding the filter.

 0  0  0  0  0  0  0
 0  0  0  0  9  9  0
 0  0  0  0  9  9  0
 0  0  0  0  9  9  0
 0  0  0  0  9  9  0
 0  0  0  0  9  9  0
 0  0  0  0  0  0  0

Here, our 5 x 5 image now has a border of zeros, so it is 7 x 7. A 3 x 3 filter can now fit in 5 positions across, so the feature map stays 5 x 5, the same size as the original image. Padding also lets the filter look properly at the pixels on the edges of the image.

The size of the feature map can be calculated as below:

output size = (input size + 2 x padding - filter size) / stride + 1

For our first example: (5 + 0 - 3) / 1 + 1 = 3. For the padded example: (5 + 2 - 3) / 1 + 1 = 5.

What is ReLU?

After convolution, a CNN passes every number of the feature map through a rule called ReLU, which stands for Rectified Linear Unit.

The rule is: if the number is negative, make it 0. If it is positive, keep it as it is.

Before ReLU:  -5   3   0  -2   8
After ReLU:    0   3   0   0   8

Here, we can see that only the positive values survive. So, ReLU keeps the places where the pattern was found and removes the rest.

ReLU also does something important. Without such a rule, stacking many layers would be no more powerful than a single layer, because adding and multiplying again and again only gives straight-line relationships. ReLU adds a bend, so the network can learn complex patterns.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

What is pooling?

Pooling is a step that shrinks a feature map by keeping only the most important value from each small area.

The most common type is max pooling. We take a 2 x 2 window, keep the biggest number, and move to the next 2 x 2 window.

Let's say we have this 4 x 4 feature map:

 1   3 |  2   0
 4   6 |  1   2
-------+-------
 0   2 |  9   5
 1   1 |  3   7

Each 2 x 2 box gives its maximum:

• Top-left box: 1, 3, 4, 6 -> maximum is 6
• Top-right box: 2, 0, 1, 2 -> maximum is 2
• Bottom-left box: 0, 2, 1, 1 -> maximum is 2
• Bottom-right box: 9, 5, 3, 7 -> maximum is 9

The result will be:

 6  2
 2  9

Here, the feature map went from 16 numbers to 4 numbers. We kept the strongest signal from each area.

Pooling helps in two ways:

• It makes the data smaller, so the next layers have less work.
• It makes the network less sensitive to small shifts. If the cat's eye moves by one pixel, the maximum in that area stays almost the same.

Putting the layers together

Now, it's time to see how all these pieces work together in a full CNN.

Image
  ↓
[ Convolution + ReLU ]   finds edges
[ Pooling ]
  ↓
[ Convolution + ReLU ]   combines edges into shapes like corners and circles
[ Pooling ]
  ↓
[ Convolution + ReLU ]   combines shapes into parts like eyes, ears, wheels
[ Pooling ]
  ↓
[ Flatten ]              turns the grid into one long list of numbers
  ↓
[ Fully connected ]      makes the final decision
  ↓
"cat: 92%, dog: 7%, car: 1%"

Here, we can see that each layer builds on the previous one.

• The first layer finds very simple things like edges and color changes.
• The second layer looks at the feature maps of the first layer, so it finds combinations of edges, like corners and curves.
• The deeper layers find bigger things like eyes, wheels, and faces.
• At the end, flatten turns the final small grids into one long list. Then a fully connected layer, the normal neural network layer we saw earlier, uses this list to decide the answer.

Here, the fully connected layer is now cheap, because by this point the image has been shrunk into a small summary of important features.

The final step usually turns the scores into percentages that add up to 100%, using a function called softmax. So, the network says something like "92% cat".

We have a detailed blog on How Do Image Embeddings Work? that explains how such a small summary of an image is used to compare and search images by meaning.

How does a CNN learn its filters?

In our example, we wrote the vertical edge filter by hand. In a real CNN, nobody writes the filters. The CNN learns the numbers inside its filters on its own during training.

The filter numbers are the weights of a CNN. Training works as below:

• We show the CNN a photo with the correct label, like "cat".
• The CNN makes a guess.
• We measure how wrong the guess is. This is called the loss.
• A method called backpropagation finds how each filter number must change to reduce the loss.
• Every filter number is adjusted a tiny bit.
• We repeat this for thousands or millions of photos.

After training, the filters in the first layer usually end up looking like edge and color detectors, very similar to what we made by hand. Nobody told the CNN to do that. It discovered them because they are useful.

If we want to go deep into Neural Networks, Backpropagation, and Loss Functions, and build a Neural Network from scratch, check out our AI and Machine Learning Program at Outcome School.

Why are CNNs so efficient?

Now, let's understand why a CNN needs far fewer weights than a fully connected network.

A 3 x 3 filter on a color image has 3 x 3 x 3 = 27 weights, plus 1 extra number called the bias. So, 28 numbers. With 64 filters, the layer has 64 x 28 = 1,792 numbers.

Earlier, the fully connected layer needed 3,000,000,000 weights. The convolution layer needs 1,792.

This is possible because of two ideas:

• Local connections: Each filter looks at only a small area at a time, not the whole image.
• Weight sharing: The same filter is used at every position in the image. A cat's eye detector works in the top-left corner and in the bottom-right corner with the same numbers.

Let me tabulate the differences between a fully connected network and a CNN for your better understanding.

A simple CNN in code

Now, let's see the code for a small CNN in PyTorch. This CNN takes a 28 x 28 black and white image of a handwritten digit and tells which digit from 0 to 9 it is.

import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, kernel_size=3, padding=1)   # 1 channel in, 16 filters
        self.conv2 = nn.Conv2d(16, 32, kernel_size=3, padding=1)  # 16 in, 32 filters
        self.pool = nn.MaxPool2d(2)                               # 2 x 2 max pooling
        self.relu = nn.ReLU()
        self.fc = nn.Linear(32 * 7 * 7, 10)                       # final decision: 10 digits

    def forward(self, x):
        x = self.pool(self.relu(self.conv1(x)))   # 28 x 28 -> 14 x 14
        x = self.pool(self.relu(self.conv2(x)))   # 14 x 14 -> 7 x 7
        x = x.flatten(1)                           # grid -> one long list
        return self.fc(x)                          # scores for 0 to 9

Here, we have built a CNN with two convolution layers.

• nn.Conv2d(1, 16, kernel_size=3, padding=1) creates 16 filters of size 3 x 3. The input has 1 channel because the image is black and white. padding=1 keeps the size the same.
• nn.Conv2d(16, 32, ...) takes the 16 feature maps from the first layer and creates 32 new feature maps.
• nn.MaxPool2d(2) does 2 x 2 max pooling, which halves the width and height.
• nn.ReLU() turns negative numbers into zero.
• In forward, the image goes from 28 x 28 to 14 x 14 after the first pooling, and to 7 x 7 after the second pooling.
• x.flatten(1) turns the 32 feature maps of size 7 x 7 into one list of 32 x 7 x 7 = 1,568 numbers.
• nn.Linear(32 * 7 * 7, 10) is the fully connected layer. It gives 10 scores, one for each digit.

The digit with the highest score is the answer. After training on handwritten digits, a small CNN like this one usually reaches very high accuracy.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Famous CNNs

Let's see a few well-known CNNs:

• LeNet (1998): Created by Yann LeCun and his team to read handwritten digits, for example on bank checks. It showed that CNNs work in the real world.
• AlexNet (2012): Won the ImageNet image recognition competition by a large margin, using GPUs for training. This result is widely seen as the start of the modern era of deep learning.
• VGG (2014): Showed that stacking many small 3 x 3 filters works very well.
• ResNet (2015): Added shortcut connections, which allowed training networks with more than 100 layers.

Where CNNs work well and where they fail

Where CNNs work well:

• Image classification: Is this photo a cat or a dog?
• Object detection: Where are the cars and people in this street photo? Used in self-driving cars.
• Face recognition: Unlocking our phone with our face.
• Medical imaging: Finding signs of disease in X-rays and scans.
• Other grid-like data: Sound turned into a picture-like grid called a spectrogram, and even some text tasks.

Where CNNs fail or struggle:

• Big-picture relationships: A filter sees only a small area. Connecting two far-apart parts of an image needs many layers.
• Rotations and big changes in size: A CNN trained on upright faces can struggle with upside-down faces unless it saw such examples during training.
• Data hungry: They still need many labeled images to learn well.
• Easy to fool: Tiny, carefully chosen changes to pixels, invisible to us, can make a CNN give a completely wrong answer.

Note: Today, a newer architecture called the Vision Transformer also works very well on images, especially with huge amounts of data. But CNNs are still widely used because they are efficient, simple, and work well even with less data.

Now we must have understood Convolutional Neural Networks.

Frequently Asked Questions

Is a CNN better than a Vision Transformer?

Not always. A Vision Transformer works very well on images, especially when we have huge amounts of data. But CNNs are still widely used because they are efficient, simple, and work well even with less data. So, the right choice depends on how much data we have for the task.

Can CNNs be used for data other than images?

Yes. CNNs work on any grid-like data, not only photos. For example, sound can be turned into a picture-like grid called a spectrogram, and a CNN can learn from it. CNNs are even used for some text tasks. The same idea of sliding small filters over a grid applies in all these cases.

What is the difference between a filter and a kernel in a CNN?

There is no difference. A filter and a kernel are two names for the same thing: a small grid of numbers that acts as a pattern detector. A common size is 3 x 3. The CNN slides it over the image and multiplies and adds at every position to build a feature map.

Why does a CNN still use a fully connected layer at the end?

The fully connected layer makes the final decision. After the convolution and pooling layers, flatten turns the final small grids into one long list of numbers, and the fully connected layer uses this list to give a score for each answer. It is cheap at this point because the image has already been shrunk into a small summary of important features.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.