How does OCR with Deep Learning work?

Authors
How does OCR with Deep Learning work?

OCR with Deep Learning reads text in an image by first finding where the text is and then recognizing it, using a model that learns from examples, not hand rules.

In this blog, we will learn about how OCR with Deep Learning works. We will also see why we need it, why the old way failed, how the full pipeline detects and recognizes text one character at a time, how the model is trained, and where OCR is used in the real world.

We will cover the following:

  • What is OCR?
  • Why do we need OCR?
  • The old way of doing OCR and why it failed
  • The full pipeline of OCR with Deep Learning
  • Step 1: Detecting the text in the image
  • Step 2: Recognizing the text
  • How the model reads characters one by one
  • Training the model
  • A simple code example
  • Where OCR is used in the real world

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is OCR?

OCR stands for Optical Character Recognition. It is a way for a computer to read the text inside an image and turn it into actual text that we can copy, search, and edit.

Let's decompose the term to make it simple.

OCR = Optical + Character + Recognition.

"Optical" means something related to seeing, "Character" means a single letter or number, and "Recognition" means identifying what something is. So, OCR means a computer looking at an image and identifying the letters and numbers inside it.

Let's say we click a photo of a book page. To us, that photo shows words. But to a computer, that photo is just a grid of tiny colored dots. The computer does not know that those dots form the word "hello". OCR is the technology that helps the computer understand that those dots actually mean "hello".

Why do we need OCR?

Consider a photo of a printed bill. We can read it with our eyes, but the computer cannot search inside it, because for the computer it is just a picture.

Now, suppose we have ten thousand such bills. Reading and typing each one by hand would take days. This is slow, boring, and full of mistakes.

So, here comes OCR to the rescue. OCR reads all those bills automatically and gives us clean, editable text in seconds.

A few real places where we need OCR:

  • Reading the amount and date from a scanned invoice.
  • Reading the number plate of a car at a toll gate.
  • Reading the text from a photo of a document on our phone.
  • Turning old printed books into digital text.

Now, let's understand how it used to work before Deep Learning.

The old way of doing OCR and why it failed

Before Deep Learning, OCR was done using fixed, hand-written rules.

The idea was simple. Engineers studied the exact shape of each letter. For the letter "A", they wrote rules like "there are two slanted lines meeting at the top with a small line in the middle". Then the computer matched every shape in the image against these rules.

This is called a rule-based approach, which means humans write down every rule by hand and the computer just follows them.

This worked fine for clean, printed text in one single font.

The real world is messy. People write in different fonts. Photos are blurry. The lighting is bad. Some letters are tilted. Some are handwritten.

The moment the text looked even slightly different from the rules, the old OCR failed badly.

The issue with this approach is that we cannot write a rule for every possible shape of every possible letter in the world. There are simply too many variations. Let's see how the next approach solves this issue.

The full pipeline of OCR with Deep Learning

Deep Learning is a type of machine learning where a computer learns patterns directly from examples, instead of being told the rules by hand. The tool that does this learning is called a neural network. We show the neural network thousands of images of each letter, and it learns to recognize them.

OCR with Deep Learning is not one single step. It is a pipeline of two main steps.

OCR = Text Detection + Text Recognition

First, the computer finds where the text is in the image, and then it reads what that text says.

We can see the flow in a diagram like below:

+--------------------------+
|       Input image        |
|  (grid of colored dots)  |
+--------------------------+
             ↓
+--------------------------+
|  Step 1: Text Detection  |
|  Find WHERE the text is  |
|  (draw bounding boxes)   |
+--------------------------+
             ↓
+--------------------------+
| Step 2: Text Recognition |
|  Read WHAT the text says |
|    (letter by letter)    |
+--------------------------+
             ↓
+--------------------------+
|      Editable text       |
|          "CAT"           |
+--------------------------+

Here, we can see that the image first goes through Text Detection, which finds where the text is, and then through Text Recognition, which reads what the text says, and finally we get clean editable text out the other end.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Now, let's move to the first step.

Step 1: Detecting the text in the image

The first step is called Text Detection. This is the part where the model finds where the text is located inside the image.

Let's say we have a photo of a street with many shops, signboards, and people. There is text on the signboards, but there is also a lot of other stuff like the sky, roads, and cars.

The detection model looks at the whole image and draws boxes around the regions that contain text. These boxes are called bounding boxes. A bounding box is simply a rectangle that surrounds a piece of text.

So, how does the model know where the text is? The answer is a special neural network called a Convolutional Neural Network, also written as CNN.

A CNN is a type of neural network that is very good at understanding images. It scans the image in small pieces and learns visual patterns like edges, curves, and shapes. Text has very specific patterns, like sharp edges packed close together, so the CNN learns to spot those patterns.

After this step, the model gives us a set of bounding boxes. Each box says "there is text here". We have a detailed blog on How does Object Detection work? that explains how a model finds objects and draws boxes around them step by step.

We can see it in a diagram like below:

+-------------------------------------------+
|  sky         +---------+                  |
|              | OUTCOME |       cloud      |
|              +---------+                  |
|    +--------+                             |
|    | SCHOOL |          car                |
|    +--------+                             |
|       road               people           |
+-------------------------------------------+

+---------+ = bounding box around text

Here, we can see that the detection model has drawn a bounding box around each piece of text, like "OUTCOME" and "SCHOOL", and has left the rest of the image such as the sky, road, car, and people untouched.

Now, our detection step is ready. We know where the text is. But we still do not know what the text says. Let's move to the next step.

Step 2: Recognizing the text

The second step is called Text Recognition. This is the part where the model reads the text inside each bounding box.

The best way to learn this is by taking an example. Suppose the detection step found a box around the word "CAT". The recognition model now takes only that small box and tries to read the letters inside it.

First, the recognition model passes this small image through a CNN again, to understand the visual features of the letters.

Then, here comes a problem. A word is a sequence of letters in a specific order. "CAT" and "ACT" use the same letters, but the order changes the meaning. So, the model must read the letters in the correct left-to-right order.

So, here comes a special tool to the rescue, called a Recurrent Neural Network, also written as RNN.

An RNN is a type of neural network that is good at handling sequences, which means data that comes in a specific order. It reads the image from left to right, a little slice at a time, and remembers what it has seen so far. This memory helps it understand the order of the letters.

This is how the model reads letters in the correct order to form a full word.

We have a detailed blog on How do LSTM and GRU work? that explains how these improved RNNs remember what they have seen over long sequences.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

How the model reads characters one by one

Now, let's understand the reading part with a simple step-by-step walkthrough, just for the sake of understanding.

Suppose the model is reading the word "CAT" from left to right. It looks at thin vertical slices of the image, one slice at a time.

Step 1: It looks at the first slices and predicts the letter C.

Step 2: It moves a little right and predicts the letter A.

Step 3: It moves further right and predicts the letter T.

So, the raw predictions, slice by slice, can come out like below:

C C C A A T T

Here, we can notice a problem. The same letter got repeated because the model looked at many slices that all belonged to the same letter. We do not want "CCCAATT". We want "CAT".

So, here comes a technique to the rescue, called CTC, which stands for Connectionist Temporal Classification.

CTC is a method that cleans up these repeated predictions and merges them into the correct final word. It removes the extra repeats and joins the slices into real letters. CTC also uses a special blank symbol between slices, so that real double letters like the "LL" in "HELLO" are not merged by mistake.

Let's see this in a diagram like below:

image slices:  C  C  C  A  A  T  T
                  ↓     ↓     ↓
merge repeats:    C     A     T
                  ↓     ↓     ↓
final word:          C A T

Here, we can see that the model first predicts a letter for every thin slice, which gives us many repeated letters, and then CTC merges the repeats together so that "C C C A A T T" becomes the final word "CAT".

This is how the model reads a word correctly even when it does not know in advance how wide each letter is.

Training the model

Now, the next big question is: how does the model learn all of this in the first place?

The answer is training. Training is the process of showing the model thousands of examples so that it can learn the patterns.

Let's say we have ten thousand small images. Each image has a word in it, and we also know the correct answer for each one. For example, one image shows "CAT" and we tell the model the answer is "CAT".

Here is how training works.

First, we show the model an image. The model makes a guess. In the beginning, the guess is mostly wrong, because the model has not learned anything yet.

Then, we compare the model's guess with the correct answer. The gap between the guess and the correct answer is called the loss. The loss is simply a number that tells us how wrong the model is. A big loss means a very wrong guess. A small loss means a good guess.

After that, the parameters of the model get updated a tiny bit, so that next time the guess is closer to the correct answer. This tiny updation is the actual learning.

Finally, we repeat this process millions of times with millions of examples. Slowly, the loss becomes smaller and smaller, and the model becomes very good at reading text.

This is how the model learns to read text from messy, real-world images without us writing a single rule by hand.

To learn Deep Learning, Neural Networks, Loss Functions, Backpropagation, and Gradient Descent, and to build a Neural Network from scratch, check out our AI and Machine Learning Program at Outcome School.

A simple code example

Let's see how to use OCR with a ready-made library. We will use a popular Python tool called easyocr. We do not need to build the neural network ourselves, because the library already has a trained model inside it.

We can write the code as below:

import easyocr

# Create a reader that can read English text
reader = easyocr.Reader(['en'])

# Read the text from an image file
result = reader.readtext('my_image.png')

# Print each piece of text the model found
for box, text, confidence in result:
    print(text)

Here, we have done a few things. First, we imported the easyocr library, which already contains a trained Deep Learning model. Then, we created a reader and told it that we want to read English by passing ['en']. After that, we called readtext and gave it our image file named my_image.png. Finally, we looped over the result and printed each piece of text that the model found in the image.

Here, the box is the bounding box location we learned about, the text is the actual word the model read, and the confidence is how sure the model is about its answer.

Suppose our image contains the words "Outcome School". This will print the following:

Outcome
School

It works perfectly. With just a few lines, we used both detection and recognition without writing any rules ourselves.

Note: Behind the scenes, this small piece of code is doing everything we learned, which means it is detecting the text boxes with a CNN and then reading them with an RNN and CTC.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Where OCR is used in the real world

Now that we have understood how OCR with Deep Learning works, let's see where it is used in the real world.

  • Banking: Reading the amount, account number, and date from cheques and invoices automatically.
  • Toll gates and parking: Reading the number plate of a vehicle as it passes.
  • Phone apps: Letting us point our camera at a document and copy the text from it.
  • Translation apps: Reading text in one language from an image and translating it.
  • Digitizing books: Turning scanned pages of old books and newspapers into searchable text.
  • Identity verification: Reading the details from documents like passports and ID cards.

Now we must have understood how OCR with Deep Learning works and where it is used.

Frequently Asked Questions

Why does OCR use both a CNN and an RNN?

OCR uses both because each one solves a different part of the problem. The CNN is good at understanding images, so it learns the visual features of the letters, like edges, curves, and shapes. The RNN is good at handling sequences, so it reads the image from left to right and keeps the letters in the correct order.

What does CTC do in OCR?

CTC, which stands for Connectionist Temporal Classification, cleans up the raw predictions of the recognition model. The model predicts a letter for every thin slice of the image, so the same letter gets repeated, like "CCCAATT". CTC merges these repeats into the final word "CAT", and it uses a special blank symbol so that real double letters like the "LL" in "HELLO" are not merged by mistake.

What is a bounding box in OCR?

A bounding box is simply a rectangle that surrounds a piece of text in the image. The text detection model draws these boxes around the regions that contain text, and each box says "there is text here". The recognition model then takes only the small image inside each box and reads the letters in it.

Do we need to train our own OCR model?

No. For many tasks, we can use a ready-made library like easyocr, which already has a trained Deep Learning model inside it. With just a few lines of code, it detects the text boxes and reads the text in an image, and it also gives a confidence score that tells how sure the model is about each answer.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.