How does Self-Supervised Learning work?

Self-Supervised Learning is a way of teaching a machine where the data creates its own labels, so the machine learns from raw data with no human labeling.

In this blog, we will learn about how Self-Supervised Learning works. We will also see why plain Supervised Learning runs into trouble, how Self-Supervised Learning teaches itself from raw data through pretraining and fine-tuning, and where it is used in the real world.

We will cover the following:

• What is Supervised Learning?
• What is the problem with Supervised Learning?
• What is Self-Supervised Learning?
• How does Self-Supervised Learning work?
• A step-by-step example with text
• A step-by-step example with images
• The two phases: Pretraining and Fine-tuning
• Why Self-Supervised Learning works so well
• Where Self-Supervised Learning is used

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is Supervised Learning?

Supervised Learning is a way of teaching a machine using examples where each example already has the correct answer attached to it.

A label is the correct answer for a piece of data. Let's say we have a photo of a dog. The label for that photo is the word "dog".

Consider a teacher sitting beside a student. The teacher shows a flashcard with a picture and says the correct name out loud. The student listens and learns. Here, the teacher is the supervisor, and the correct name is the label.

So, in Supervised Learning, a human first looks at each piece of data and writes down the correct answer. Then the machine learns from those answers.

What is the problem with Supervised Learning?

The problem is that labels are made by humans, and humans are slow and expensive.

Let's say we want to teach a machine using one million photos. A human has to look at each photo and write the correct label for it. This takes a huge amount of time and money.

For some tasks, it is even harder. To label medical images, we need doctors. To label legal documents, we need lawyers. These experts are costly and rare.

So, we have a problem. The world has a massive amount of data, but most of it has no labels. We have billions of web pages, photos, and videos, and almost none of them come with correct answers attached.

What is the solution to this problem? Answer: Self-Supervised Learning.

What is Self-Supervised Learning?

Self-Supervised Learning is a way of teaching a machine where the data creates its own labels, so no human is needed to label anything.

It is still learning from examples that have correct answers, but the supervisor is not a human. The data itself acts as the supervisor.

The core idea is simple. Hide a part of the data, and ask the machine to predict the hidden part. The hidden part is the correct answer, and we already know it because we are the ones who hid it.

Now, let's understand how this actually works.

How does Self-Supervised Learning work?

The idea is to create a puzzle out of the data itself.

Here is the recipe we follow:

• Step 1: Take a piece of data that has no label.
• Step 2: Hide a part of it.
• Step 3: Ask the machine to guess the hidden part.
• Step 4: Compare the machine's guess with the part we hid. The part we hid is the correct answer.
• Step 5: If the guess is wrong, adjust the machine a little so that next time it guesses better.

We can picture this loop as below:

   +------------------------+
   |  Unlabeled data piece  |
   +------------------------+
              |
              ↓
   +------------------------+
   |   Hide a part of it    |
   +------------------------+
              |
              ↓
   +------------------------+        +------------------------+
   |  Machine guesses the   | -----> |  Compare guess with    |
   |     hidden part        |        |  the part we hid       |
   +------------------------+        +------------------------+
              ^                                  |
              |                                  ↓
              |                       +------------------------+
              +---------------------- |  Wrong? Adjust the     |
            adjust and repeat         |  machine a little      |
                                      +------------------------+

Here, we can see that the part we hid is the correct answer, so the machine always has something to check its guess against. We never needed a human in this loop. The data piece goes in, we hide a part, the machine guesses, we compare, and we adjust. Then we repeat.

We repeat this millions of times with millions of pieces of data.

This is why we call it self-supervised. The supervision comes from the data, not from a person.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

The best way to learn this is by taking an example.

A step-by-step example with text

Let's take a simple sentence as our piece of data.

The cat is sitting on the mat.

Here, we have a normal sentence. It has no label. Nobody told us what this sentence means or what category it belongs to.

Now, let's create a puzzle. We hide one word like below:

The cat is sitting on the ____.

Here, we have hidden the word "mat". We ask the machine to guess the missing word.

Let's visualize this puzzle as below:

   Original sentence:
   The   cat   is   sitting   on   the   [mat]
                                          |
                                          ↓
                                    hide this word
                                          |
                                          ↓
   Puzzle given to the machine:
   The   cat   is   sitting   on   the   [____]
                                          |
                                          ↓
                            machine must guess this word

   Correct answer (kept by us): mat

Here, we can see that the word "mat" is removed from the sentence and kept aside by us. The machine only sees the blank and must fill it. Because we are the ones who hid the word, we already know the correct answer is "mat", so we can check the guess without any human writing a label.

The machine reads the rest of the sentence and tries to predict the hidden word. Let's say the machine guesses "chair".

Now, we compare. The correct answer was "mat", but the machine said "chair". The guess is wrong, so we adjust the machine a little.

We repeat this with the next sentence, and the next, for billions of sentences from the internet.

Slowly, the machine learns how words fit together. It learns that "sitting on the" is usually followed by something like a "mat", a "chair", or a "floor". It learns grammar, meaning, and facts about the world, all by playing this simple guessing game.

This is how LLMs (Large Language Models) are pretrained. Depending on the model, they learn to predict the next token or a hidden token again and again.

We did not label a single sentence by hand. The missing word was always the correct answer.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

A step-by-step example with images

Now, let's see the same idea with images.

Suppose we have a photo of a dog. There is no label on it. We do not know it is a dog yet, and that is fine.

We create a puzzle. We cover a part of the photo with a gray box. Let's say we hide the face of the dog.

Then, we ask the machine to guess what is behind the gray box. The machine looks at the rest of the photo, the body, the ears, the tail, and tries to draw the hidden face.

We compare the machine's drawing with the real face that we hid. If it is wrong, we adjust the machine.

Here, we can notice the same pattern again. The correct answer, which is the hidden face, was always inside the photo. We just hid it and used it to check the machine.

There is another popular image puzzle. We take one photo and create two slightly different versions of it. We crop it differently, change the colors a little, or rotate it. Then we ask the machine to understand that both versions are actually the same photo.

By solving this, the machine learns the deep features of objects, like shapes, edges, and textures, without anyone telling it what the object is. We have a detailed blog on Contrastive Learning that explains this step by step.

This is how Self-Supervised Learning works for images.

The two phases: Pretraining and Fine-tuning

Now that we have learned the core idea, it is time to learn how it is used in two phases.

Self-Supervised Learning usually happens in two steps.

Pretraining: This is the first phase. Here, the machine plays the guessing game on a huge amount of unlabeled data. It hides parts of the data and predicts them, millions of times. After this phase, the machine has learned a lot about language or images in general. We call this trained machine a pretrained model. A pretrained model is a model that already understands the basics very well.

Fine-tuning: This is the second phase. Now we take the pretrained model and teach it one specific task using a small amount of labeled data. Let's say we want it to detect spam emails. We give it a few thousand labeled emails, and it quickly becomes good at this task.

Here, we can see the main benefit. The hard and expensive part, which is understanding language or images, was done using unlabeled data with no human labeling. The easy part, which is the final task, needs only a little labeled data.

We can picture the flow of the two phases as below:

   Phase 1: Pretraining                Phase 2: Fine-tuning
   (no human labels)                   (small human labels)

   +---------------------+             +---------------------+
   |  Huge unlabeled     |             |  Small labeled      |
   |  data (the internet)|             |  data (few thousand)|
   +---------------------+             +---------------------+
              |                                   |
              ↓                                   ↓
   +---------------------+             +---------------------+
   |  Play the guessing  |   ----->    |  Teach one specific |
   |  game millions of   |  pretrained |  task, like spam    |
   |  times              |    model    |  detection          |
   +---------------------+             +---------------------+
              |                                   |
              ↓                                   ↓
      learns general                     becomes good at
      knowledge                          our exact task

Here, we can see that the pretrained model from the first phase is carried into the second phase. The expensive general learning happens once on cheap unlabeled data, and then a small amount of labeled data finishes the job for our exact task.

Let me tabulate the difference between the two phases for your better understanding.

So, the model first learns the world by itself, and then learns our exact task with a small push.

To master Self-supervised Learning and Fine-tuning, and to build a Large Language Model (LLM) from scratch, check out our AI and Machine Learning Program at Outcome School.

Why Self-Supervised Learning works so well

Now, let's understand why this idea works so well.

The first reason is data. The internet has a huge amount of text, images, and videos. Self-Supervised Learning can use this data because it does not need human labels.

The second reason is cost. We save a huge amount of money and time because humans do not label billions of examples.

The third reason is understanding. By guessing hidden parts again and again, the model learns deep patterns about the world. It learns grammar, facts, shapes, and meaning. This general understanding makes it good at many tasks, not just one.

The LLMs used by ChatGPT and many image models start by playing this guessing game on massive unlabeled data.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Where Self-Supervised Learning is used

Here are a few real use cases:

• Language models: Models that write, translate, and answer questions learn by predicting hidden or next words.
• Image models: Many image models learn by predicting hidden patches of photos or by matching two versions of the same photo.
• Speech models: Models that understand audio learn by predicting hidden parts of sound.
• Recommendation systems: Models learn what we may like next by predicting the next item in our history.
• Medical AI: Models first learn from huge unlabeled scans, then fine-tune on a few labeled scans from doctors.

In all of these, the pattern is the same. Hide a part of the data, guess it, and learn from the mistake.

Frequently Asked Questions

Do LLMs use Self-Supervised Learning?

Yes. LLMs (Large Language Models) are pretrained with Self-Supervised Learning. Depending on the model, they learn to predict the next token or a hidden token again and again, across billions of sentences from the internet. Nobody labels a single sentence by hand, because the missing word is always the correct answer.

Does Self-Supervised Learning need any labeled data at all?

Not for pretraining. In pretraining, the data creates its own labels, so no human labeling is needed. Labeled data comes in only during fine-tuning, and only a small amount of it. For example, to detect spam emails, we give the pretrained model a few thousand labeled emails.

What is a pretrained model?

A pretrained model is a model that has already played the guessing game on a huge amount of unlabeled data. It already understands the basics of language or images very well. We then fine-tune it on a small amount of labeled data to make it good at one specific task.

Why is it called self-supervised?

It is called self-supervised because the supervision comes from the data, not from a person. We hide a part of the data and ask the machine to guess it. Because we hid that part ourselves, we already know the correct answer, so the data itself acts as the supervisor.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.