How does Semi-Supervised Learning work?

Authors
How does Semi-Supervised Learning work?

Semi-supervised learning trains a better model by using a small amount of labeled data together with a large amount of unlabeled data, through methods like self-training.

In this blog, we will learn about how Semi-Supervised Learning works. We will also see the difference between labeled and unlabeled data, why labels are expensive, the key assumption behind it, how self-training and label propagation work, and where it is used.

We will cover the following:

  • Labeled data vs unlabeled data
  • The problem: labels are expensive
  • What is semi-supervised learning?
  • The key assumption behind it
  • How self-training works
  • How label propagation works
  • Where semi-supervised learning is used

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

Labeled data vs unlabeled data

Before jumping into semi-supervised learning, we must understand two kinds of data.

Labeled data is data where we already know the answer. For example, a photo with a tag that says "cat". The answer, "cat", is the label.

Unlabeled data is data with no answer attached. For example, a photo with no tag. We can see it, but we do not know what it shows.

   Labeled:    [photo]  ->  tag: "cat"      (we know the answer)
   Unlabeled:  [photo]  ->  no tag          (we do not know the answer)

Normal supervised learning needs labeled data. It learns from examples that already have answers.

The problem: labels are expensive

Here is the real-world problem. Unlabeled data is everywhere and cheap. The internet is full of photos, text, and audio with no labels. But labeled data is expensive and slow to get, because a human has to look at each item and write the answer.

Let's say we have one million photos. Labeling all of them by hand would cost a lot of money and take a very long time. But we can easily afford to label just a few hundred.

   1,000,000 photos with no labels   (cheap and easy to collect)
         500 photos with labels      (costly, a human tagged each one)

So, we have a little labeled data and a lot of unlabeled data. The question is: can we use both together? This is exactly what semi-supervised learning does.

What is semi-supervised learning?

Semi-supervised learning uses a small amount of labeled data together with a large amount of unlabeled data to train a better model.

It sits between two other methods.

  • Supervised learning uses only labeled data.
  • Unsupervised learning uses only unlabeled data.
  • Semi-supervised learning uses both, a few labels plus lots of unlabeled data.

The goal is to get a model that is often close to one trained on many more labels, but by using only a few hundred labels plus the cheap unlabeled data.

We also have a detailed blog on how Self-Supervised Learning works that explains another way to learn from unlabeled data, where the data creates its own labels.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

The key assumption behind it

For this to work, we rely on one assumption.

Points that are close to each other, or look similar, probably share the same label.

Means, if an unlabeled photo looks a lot like a labeled "cat" photo, it is probably also a cat. Similar things tend to have the same answer.

   [cat, labeled]   . . . .   [unlabeled]   . . . .   [unlabeled]
                              looks like cat         looks like cat
                              so probably cat        so probably cat

This assumption lets the labels "spread" from the few labeled points to the many nearby unlabeled points.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

How self-training works

One common method is called self-training, also known as pseudo-labeling. Let's see the steps.

Step 1: Train a first model using only the small labeled data.

Step 2: Use this model to predict labels for the unlabeled data. These guessed labels are called pseudo-labels.

Step 3: Keep only the guesses the model is most confident about. Add these confident pseudo-labels to the training data as if they were real labels.

Step 4: Train the model again on this bigger, combined data.

We can repeat these steps, and the model keeps improving.

   Small labeled data
           ↓
   Train a first model
           ↓
   Predict labels for the unlabeled data
           ↓
   Keep only the confident guesses (pseudo-labels)
           ↓
   Add them to the labeled data
           ↓
   Train again on the bigger data
           ↓
   A better model

Here, we can see the idea. The model teaches itself. It labels the easy, confident cases first, then uses them to learn the harder cases. This is why it is called self-training.

How label propagation works

Another method is called label propagation. Here, we imagine all the data points connected in a graph, where similar points are linked.

We start with labels only on a few points. Then we let each label spread to its neighbors, step by step, like a color spreading across a map.

   Start:   [CAT]  -  o  -  o  -  o  -  o  -  [DOG]
                 (unlabeled points in the middle)

   Spread:  [CAT]  - cat - cat - dog - dog -  [DOG]
             (labels flow from the labeled points to neighbors)

Here, we can see that the "cat" label spreads to points near the cat, and the "dog" label spreads to points near the dog. In the end, every point gets a label based on which labeled points it is most strongly connected to.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Where semi-supervised learning is used

Now, let's see where it helps us.

  • Medical images: Expert doctors are needed to label scans, which is costly. So we label a few and use many unlabeled scans.
  • Speech recognition: We have huge amounts of audio but few written transcripts, so labels are limited.
  • Web page classification: There are billions of pages online, but only a few are labeled by hand.
  • Fraud detection: Very few transactions are confirmed as fraud, but there are millions of unlabeled transactions.

Frequently Asked Questions

Is semi-supervised learning supervised or unsupervised?

It is neither, it sits between the two. Supervised learning uses only labeled data, and unsupervised learning uses only unlabeled data. Semi-supervised learning uses both, a small amount of labeled data plus a large amount of unlabeled data, to train a better model.

What is a pseudo-label?

A pseudo-label is a label that the model guesses for an unlabeled example. In self-training, we first train a model on the small labeled data, use it to predict labels for the unlabeled data, and keep only the most confident guesses. These confident pseudo-labels are then added to the training data as if they were real labels.

Why does semi-supervised learning work?

It works because of one assumption: points that are close to each other, or look similar, probably share the same label. If an unlabeled photo looks a lot like a labeled "cat" photo, it is probably also a cat. This lets the labels spread from the few labeled points to the many nearby unlabeled points.

When should we use semi-supervised learning?

We should use it when labels are expensive but unlabeled data is cheap and plentiful. For example, medical scans need expert doctors to label them, and fraud detection has very few confirmed fraud cases but millions of unlabeled transactions. In such cases, we label a few examples and use the many unlabeled ones too.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.