What is One-hot Encoding in Machine Learning?

One-hot Encoding is a way to turn a category, like a color or a word, into a list of numbers where exactly one position is 1 and all the other positions are 0. The position of the 1 tells the model which category it is.

In this blog, we will learn about One-hot Encoding in Machine Learning. We will also see why a model needs numbers, what goes wrong with simple label encoding, how One-hot Encoding works step by step with Python code, how it is used for the correct answer in classification, how LLMs use it for tokens and why embeddings replace it, and where it works well and where it fails.

We will cover the following:

• Why does a model need numbers?
• The problem with label encoding
• What is One-hot Encoding?
• One-hot Encoding step by step
• One-hot Encoding in Python
• One-hot Encoding for the correct answer in classification
• One-hot Encoding of tokens in LLMs
• Why embeddings replace one-hot vectors
• Disadvantages of One-hot Encoding

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

Why does a model need numbers?

A machine learning model only understands numbers. Inside, it multiplies numbers, adds numbers, and compares numbers. It cannot multiply the word "red".

But, a lot of real data is not numbers. It is categories.

• The color of a car: red, green, blue
• The city of a customer: Delhi, Mumbai, Bengaluru
• The type of an animal in a photo: cat, dog, horse
• The next word in a sentence

So, before we give such data to a model, we must convert every category into numbers. The question is how to do it in the right way. We have a detailed blog on Feature Engineering that explains how we prepare raw data so that a model can learn better.

The problem with label encoding

The first idea that comes to our mind is to give each category a number. This is called label encoding.

Let's say we have three colors. We can write them as below:

It looks simple. But, the model does not know that these numbers are just names. It treats them as real numbers.

So, the model now believes the following things:

• blue (3) is bigger than red (1)
• green (2) is exactly in the middle of red and blue
• red + green = 1 + 2 = 3 = blue

None of this is true. Colors have no order. Blue is not bigger than red. Red plus green is not blue.

We have added a fake order to our data. The model will try to learn from this fake order, and it will learn wrong patterns.

Note: Label encoding is fine when the categories really have an order, for example small, medium, large. The problem comes only when there is no natural order.

So, we need a way to convert categories into numbers without any fake order. Here comes the One-hot Encoding into the picture.

What is One-hot Encoding?

One-hot Encoding converts each category into a list of 0s and 1s, where only one position is 1 and all other positions are 0.

This list of numbers is called a vector. The length of the vector is equal to the number of categories.

The name tells the idea. Only one position is "hot", which means it is 1. All the other positions are "cold", which means they are 0.

For our three colors, we get the following:

Let's see this as a diagram:

┌─────────────┐
│   "green"   │
└──────┬──────┘
       │
       ↓
    look up its position
    red = 0, green = 1, blue = 2
       │
       ↓
┌─────────────┐
│  [0, 1, 0]  │
└─────────────┘

Here, we can see that "green" sits at position 1 in our list of colors. We start counting positions from 0, as most programming languages do. So, its vector has a 1 at position 1 and 0 everywhere else.

Now, there is no fake order. No color is bigger than another, and no color sits in the middle of two others. If we compare any two colors, their vectors differ in exactly two positions. So, every color is equally different from every other color. Each color gets its own column.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

One-hot Encoding step by step

The best way to learn this is by taking an example.

Suppose we have this data about cars:

Step 1: We find all the unique categories. Here, they are red, green, and blue. So, we have 3 categories.

Step 2: We give each category a fixed position. The color red is at position 0, green is at position 1, and blue is at position 2.

Step 3: For each row, we create a vector of 3 zeros. Then, we put a 1 at the position of its category.

Step 4: We replace the single "Color" column with the 3 new columns.

Our data now looks like below:

Here, we can notice two things:

• Every row has exactly one 1.
• Car A and Car D have the same color, so they have the same vector.

This is how one column of words becomes three columns of numbers that the model can use.

One-hot Encoding in Python

Let's see the code for One-hot Encoding in plain Python:

colors = ["red", "green", "blue"]

def one_hot(color):
    vector = [0] * len(colors)
    index = colors.index(color)
    vector[index] = 1
    return vector

for color in ["green", "red", "blue", "green"]:
    print(color, one_hot(color))

Here, we have:

• colors is the list of all unique categories. Its order fixes the position of each color.
• [0] * len(colors) creates a vector of 3 zeros.
• colors.index(color) finds the position of the color.
• vector[index] = 1 makes that one position hot.

This will print the following:

green [0, 1, 0]
red [1, 0, 0]
blue [0, 0, 1]
green [0, 1, 0]

It matches the table that we built by hand.

Note: In real projects, we do not write this by hand. Libraries do it for us, for example pd.get_dummies in pandas and OneHotEncoder in scikit-learn. They follow the same idea.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

One-hot Encoding for the correct answer in classification

So far, we have used One-hot Encoding for the input. It is also used for the correct answer, which is called the target.

Let's say we are building a model that looks at a photo and tells whether it is a cat, a dog, or a horse. This is called classification.

The model does not give one answer. It gives a probability for each class. For example, the model says: cat 0.7, dog 0.2, horse 0.1. These numbers add up to 1. The softmax function at the end of the model produces them.

Now, suppose the photo is really a cat. We do not write the correct answer as cat = 1, dog = 2, horse = 3, because that again adds a fake order. Instead, we write it as a one-hot vector as below:

Now, both the correct answer and the prediction are vectors of the same length. So, we can compare them position by position.

The most common way to compare them is cross-entropy loss. The loss is a number that tells how wrong the model is. Lower is better.

Cross-entropy loss = -(sum over all classes of target_i * log(prediction_i))

Here, target_i is the value of the correct answer at position i, prediction_i is the probability that the model gives at position i, and log is the natural log.

Now, the one-hot target makes this very simple. The target is 0 for dog and horse, so those terms become 0. Only the cat term is left, because its target is 1.

Cross-entropy loss = -log(prediction for the correct class)

Let's see this as a diagram:

┌──────────────────────────────┐
│ correct answer: [1, 0, 0]    │
│ prediction: [0.7, 0.2, 0.1]  │
└──────────────┬───────────────┘
               │
               ↓
     the 1 is at position 0,
     so keep only 0.7
               │
               ↓
┌──────────────────────────────┐
│ loss = -log(0.7) = 0.36      │
└──────────────────────────────┘

Here, we can see that the one-hot target picks the probability of the correct class, and the loss is computed only from that one number.

Now, let's compare two cases:

• If the model says cat = 0.7, the loss is -log(0.7) = 0.36.
• If the model says cat = 0.2, the loss is -log(0.2) = 1.61.

Here, we can see that when the model gives a low probability to the correct class, the loss is high. During training, the model updates its parameters to push this loss down. Means, it learns to give a higher probability to the correct class.

To learn Loss Functions, Logits, Cross-Entropy, and Embeddings in depth, check out our AI and Machine Learning Program at Outcome School.

One-hot Encoding of tokens in LLMs

Now, let's move to LLMs (Large Language Models).

An LLM reads text as tokens. A token is a word or a piece of a word. The LLM has a fixed list of all the tokens it knows. This list is called the vocabulary.

Let's say, for the sake of understanding, our vocabulary has only 4 tokens:

Then, each token can be written as a one-hot vector of length 4:

• "I" becomes [1, 0, 0, 0]
• "love" becomes [0, 1, 0, 0]
• "AI" becomes [0, 0, 1, 0]
• "books" becomes [0, 0, 0, 1]

The same idea is used on the output side. When the LLM is trained to predict the next token, the correct next token is a one-hot target over the whole vocabulary, and we use the same cross-entropy loss that we just learned.

But, a real vocabulary is not 4 tokens. It is often 50,000 to more than 100,000 tokens. Let's see what happens then.

Why embeddings replace one-hot vectors

Let's say the vocabulary has 50,000 tokens. Then, every token becomes a vector of 50,000 numbers, where 49,999 of them are 0 and only one of them is 1.

This creates two big problems:

• Too big: Each token needs 50,000 numbers, and almost all of them are 0. This is a huge waste of memory and computation.
• No meaning: Every one-hot vector is equally different from every other one-hot vector. So, "cat" is as different from "kitten" as it is from "car". The vector tells us nothing about meaning.

So, here comes the embedding to the rescue.

An embedding is a short list of numbers that represents a token, for example 768 numbers instead of 50,000 numbers. These numbers are learned during training, so tokens with similar meanings end up with similar embeddings.

All the embeddings are stored together in a table called the embedding matrix. It has one row for each token in the vocabulary.

If we multiply a one-hot vector with the embedding matrix, we get exactly one row of the matrix. The 1 picks its row, and all the 0s remove the other rows.

Let's see this as a diagram:

┌──────────────────────────────┐
│   one-hot vector for "AI"    │
│         [0, 0, 1, 0]         │
└──────────────┬───────────────┘
               │
               ↓
     multiply with
               │
               ↓
┌──────────────────────────────┐
│       embedding matrix       │
│    (4 tokens x 3 numbers)    │
│                              │
│ row 0: I       0.1  0.2  0.3 │
│ row 1: love    0.4  0.5  0.6 │
│ row 2: AI      0.7  0.8  0.9 │
│ row 3: books   1.0  1.1  1.2 │
└──────────────┬───────────────┘
               │
               ↓
     the 1 at position 2
     picks row 2
               │
               ↓
┌──────────────────────────────┐
│      embedding of "AI"       │
│       [0.7, 0.8, 0.9]        │
└──────────────────────────────┘

Here, we can see that the 1 at position 2 selects row 2 of the matrix. The result is the embedding of "AI".

We can write the code for this as below:

import numpy as np

one_hot = np.array([0, 0, 1, 0])

embedding_matrix = np.array([
    [0.1, 0.2, 0.3],
    [0.4, 0.5, 0.6],
    [0.7, 0.8, 0.9],
    [1.0, 1.1, 1.2],
])

print(one_hot @ embedding_matrix)
print(embedding_matrix[2])

Here, @ is matrix multiplication. The first line multiplies the one-hot vector with the matrix. The second line picks row 2 directly.

This will print the following:

[0.7 0.8 0.9]
[0.7 0.8 0.9]

Both give the same answer. But, the second way is much faster, because it does not multiply thousands of zeros.

This is exactly what an LLM does. It never builds the giant one-hot vector. It takes the position of the token, which is called the token ID, and directly picks that row from the embedding matrix. This is called an embedding lookup.

So, the one-hot idea is still there in the background, but the embedding replaces the giant one-hot vector with a short, meaningful vector.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Disadvantages of One-hot Encoding

One-hot Encoding works well when the number of categories is small, like colors, days of the week, or a few product types. But, it has some disadvantages:

• Very wide: Each category adds one new column. A column with 10,000 different cities becomes 10,000 columns.
• Sparse: Almost every value is 0. Data where most values are 0 is called sparse data. Storing and processing so many zeros wastes memory and time.
• Hard to learn with many categories: When there are thousands of categories, many columns have very few 1s. So, the model sees many categories only a few times, and it needs much more data to learn something useful about each of them.
• No similarity: Every category is equally different from every other category. The encoding cannot show that "Mumbai" is closer to "Pune" than to "London".
• New categories: If a new category appears after training, for example a new color "yellow", there is no column for it.

For a small number of categories without any order, One-hot Encoding is the right choice. For a very large number of categories, like the tokens of an LLM, embeddings are the better choice.

Now we must have understood what One-hot Encoding is, how it works, and where it is used in machine learning.

Frequently Asked Questions

When should we use label encoding instead of One-hot Encoding?

We should use label encoding when the categories really have an order, for example small, medium, large. When the categories have no natural order, like colors or cities, label encoding adds a fake order that the model learns wrong patterns from, so One-hot Encoding is the right choice there.

How do we do One-hot Encoding in pandas or scikit-learn?

In real projects, we do not write One-hot Encoding by hand. We use libraries like pd.get_dummies in pandas and OneHotEncoder in scikit-learn. They follow the same idea: find the unique categories, give each one a fixed position, and put a 1 at that position for each row.

Do LLMs actually build one-hot vectors for tokens?

No. An LLM never builds the giant one-hot vector. It takes the token ID and directly picks that row from the embedding matrix, which is called an embedding lookup. This gives exactly the same result as multiplying a one-hot vector with the embedding matrix, but it is much faster.

Why does One-hot Encoding not work well for a very large number of categories?

Because each category adds one new column, so the data becomes very wide and sparse, with almost every value being 0. With thousands of categories, the model also sees many of them only a few times. For a very large number of categories, like the tokens of an LLM, embeddings are the better choice.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.