How does an embedding model work?
- Authors
- Name
- Amit Shekhar
- Published on
An embedding model is a model that turns a piece of text into a list of numbers that captures its meaning. Texts with similar meaning get similar lists of numbers, so a computer can measure how close two meanings are.
In this blog, we will learn about how an embedding model works. We will also see what an embedding actually is, how the model turns text into a list of numbers step by step, how it is trained to place similar meanings close together, how we measure the closeness between two embeddings, where embedding models are used, and where they fail.
We will cover the following:
- What is an embedding?
- Understanding meaning as a position on a map
- What is an embedding model?
- How does an embedding model work, step by step?
- How is an embedding model trained?
- How do we measure the similarity between two embeddings?
- A simple example in code
- Embedding model vs LLM
- Where do we use embedding models?
- Limitations of embedding models
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning at Outcome School.
Let's get started.
What is an embedding?
An embedding is a list of numbers that represents the meaning of a piece of text.
For example, the sentence "I love my puppy" can become a list like below:
[0.12, -0.45, 0.88, 0.03, -0.27, ..., 0.61]
This list of numbers is also called a vector. A vector is just an ordered list of numbers. The number of items in the list is called its dimension. Real embedding models produce lists with a few hundred to a few thousand numbers. For example, a popular small model called all-MiniLM-L6-v2 produces a list of 384 numbers for every sentence.
Note: A single number in this list does not have a clear human meaning like "this number means animal". The meaning lives in the whole list together, not in any one number.
Now, the next big question is: how can a list of numbers hold meaning? Let's understand this with a map.
Understanding meaning as a position on a map
The best way to learn this is by taking an example.
Suppose we have a big map. On this map, we place words in such a way that words with similar meanings sit close to each other.
^
| puppy dog
| cat kitten
|
|
| car bus
| truck
|
| apple banana
| mango
+------------------------------------->
Here, we can see that:
- "puppy", "dog", "cat", and "kitten" sit together in one corner, because they are all pets.
- "car", "bus", and "truck" sit together in another corner, because they are all vehicles.
- "apple", "banana", and "mango" sit together in a third corner, because they are all fruits.
Every word on this map has a position. And a position on a flat map with two directions, which we call a 2D map, is just two numbers: how far to the right, and how far up. So, "dog" can be written as something like [2.0, 9.0] and "truck" as [8.5, 5.0].
Means, the position itself is the meaning. Two things are similar if their positions are close.
Now, a 2D map has only two directions, which is too few to capture all the shades of meaning in human language. So, a real embedding uses hundreds of directions instead of two. We cannot draw a map with 384 directions, but the idea stays exactly the same: similar meanings get nearby positions, and different meanings get faraway positions.
This map of meanings is called the embedding space.
We have a detailed blog on What are Embeddings? that explains this in depth.
What is an embedding model?
An embedding model is a trained model that takes a piece of text as input and gives its embedding as output.
The embedding model decides where each text goes on the map of meanings.
"I love my puppy" ---> [ Embedding Model ] ---> [0.12, -0.45, 0.88, ..., 0.61]
(text) (embedding / vector)
The input can be a word, a sentence, a paragraph, or even a full page. The output is always a list of the same fixed length. A short sentence and a long paragraph both become, for example, 384 numbers.
This is very useful, because now we can compare any two texts in the same way, no matter how long they are.
Now, let's open this box and see what happens inside.
How does an embedding model work, step by step?
Most modern embedding models are built on an architecture called the Transformer. The Transformer reads all the words of a text together and understands how each word relates to the other words. It is the same architecture that powers LLMs (Large Language Models).
We do not need the full details of the Transformer here. We will just follow the journey of one sentence through the model.
Let's take the sentence: "I love my puppy".
Step 1: We break the text into tokens.
First, the model breaks the text into small pieces called tokens. A token is a word or a part of a word.
"I love my puppy" ---> ["I", "love", "my", "puppy"]
A rare word can be split into smaller parts. For example, "unbelievable" can become ["un", "believ", "able"]. This way, the model can handle any word, even words it has never seen fully.
Each token is then replaced by its ID number from the model's fixed list of known tokens, called the vocabulary.
Step 2: We look up a starting vector for each token.
The model has a big table where every token in its vocabulary has its own starting vector. It looks up each token in this table.
"I" ---> [0.3, -0.1, ...]
"love" ---> [0.7, 0.5, ...]
"my" ---> [0.2, 0.0, ...]
"puppy" ---> [0.9, -0.4, ...]
At this stage, every token gets the same vector no matter where it appears. The word "bank" gets the same vector in "river bank" and in "bank account". The model has not yet looked at the surrounding words.
Step 3: We let the tokens look at each other.
Now, the Transformer layers come into the picture. Inside these layers, every token looks at every other token in the sentence and updates its own vector based on what it sees. This mechanism is called attention.
So, after the Transformer layers, "bank" in "river bank" gets a vector close to water and nature, and "bank" in "bank account" gets a vector close to money. Each token's vector now carries the meaning of the token in its context.
Before: "bank" ---> same vector everywhere
After: "river bank" ---> bank vector near "water", "shore"
"bank account" ---> bank vector near "money", "loan"
Planning your move into AI?
No matter which tech domain you work in today, with your effort, you can transition to these roles:
- AI Engineer
- Forward Deployed Engineer
- AI Agent Engineer
- Harness Engineer
- Voice AI Engineer
- AI Architect
- LLM Engineer
- LLM Inference Engineer
- Gen AI Engineer
- MLOps Engineer
- ML Engineer
- Deep Learning Engineer
Our AI and Machine Learning Program at Outcome School takes you there.
Step 4: We combine all token vectors into one vector.
Now, we have one vector for each token. But we want one single vector for the whole sentence. So, the model combines them. This step is called pooling.
The most common way is mean pooling, which means taking the average of all the token vectors, number by number.
"I" ---> [0.2, 0.6]
"love" ---> [0.8, 0.4]
"my" ---> [0.1, 0.5]
"puppy" ---> [0.9, 0.5]
-------------------------
Average ---> [0.5, 0.5] <--- one vector for the whole sentence
Here, for the sake of understanding, we have used only two numbers per token. The first numbers are 0.2, 0.8, 0.1, and 0.9. Their sum is 2.0, and the average is 2.0 / 4 = 0.5. We do the same for the second numbers. Real models do the same thing with hundreds of numbers.
Some models use a different way. They add a special token at the start of the text, often called [CLS], and use the final vector of only that special token as the summary of the whole sentence. Both ways give us one vector for the full text.
Step 5: We normalize the vector.
Finally, many embedding models scale the vector so that its length becomes exactly 1. The length of a vector is how far its point is from the center of the map. This is called normalization. It does not change the direction of the vector, it only changes its size.
Why do we do this? Because we care about the direction of the vector, which carries the meaning, and not about its size. Normalization makes comparing two vectors simple and fair.
The full journey looks like below:
"I love my puppy"
↓
[ Step 1: Tokens ]
("I", "love", "my", "puppy")
↓
[ Step 2: Lookup table ]
(one starting vector per token)
↓
[ Step 3: Transformer ]
(vectors updated using context)
↓
[ Step 4: Pooling ]
(average into one vector)
↓
[ Step 5: Normalize ]
(length set to 1)
↓
[0.12, -0.45, 0.88, ..., 0.61]
(the embedding)
This is how an embedding model turns text into an embedding.
How is an embedding model trained?
Now, the question is: how does the model learn to put "I love my puppy" near "I adore my dog", and far from "The stock market crashed today"?
At the very beginning, all the numbers inside the model are random, so it places texts on the map randomly. In practice, we usually start from a Transformer that has already read a huge amount of text and knows language well. But even that model does not yet place similar sentences close together on the map. Training fixes this.
The most common way to train an embedding model is called contrastive learning.
Here, the model learns by contrasting, which means comparing, things that are similar with things that are different.
Let's say we have a teacher who shows the model three sentences at a time:
- Anchor: "How do I bake a cake?"
- Positive: "Steps to make a cake at home" (similar meaning)
- Negative: "How do I change a car tire?" (different meaning)
The teacher gives the model one rule: pull the anchor and the positive closer together, and push the anchor and the negative further apart.
Before training: After training:
cake-question cake-question cake-steps
tire-question (close together)
cake-steps
tire-question
(far away)
The model creates embeddings for all three sentences and checks how close they are. If the positive is not close enough, or the negative is too close, the model gets a penalty. This penalty is called the loss. The model's parameters then get updated slightly to reduce the loss.
We repeat this for millions of such examples. Slowly, the model learns a map where similar meanings sit together.
Where do these similar pairs come from? They come from data that naturally has pairs of related text, for example:
- A question and its answer from a question-and-answer website.
- The title of an article and its first paragraph.
- A search query and the page that people clicked on.
Note: We often do not need a hand-picked negative for every example. The model trains on a group of pairs at once, and for each anchor, it treats the positives of all the other pairs in the group as negatives. This trick is called in-batch negatives, and it gives a lot of negatives for free.
So, the model is not told what each number means. It only learns from "these two are similar, and these two are not". The map of meanings appears on its own from this training.
A quick note for you
No matter which tech domain you work in, get familiar with these topics:
- LLM
- RAG
- MCP
- Agent
- Fine-tuning
- Quantization
We put it all together in one video:
AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization
No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.
Now, let's get back to the topic.
How do we measure the similarity between two embeddings?
Now, we have embeddings. The next big question is: how do we find how close two embeddings are?
The most popular way is cosine similarity. It measures the angle between two vectors.
Let's say each embedding is an arrow starting from the center of the map.
- If two arrows point in the same direction, the meanings are similar. The cosine similarity is close to 1.
- If two arrows point at a right angle, the meanings are unrelated. The cosine similarity is close to 0.
- If two arrows point in opposite directions, the cosine similarity is close to -1.
When the vectors are normalized to length 1, cosine similarity becomes the same as a simple calculation called the dot product. The dot product means multiplying the matching numbers of the two lists and adding the results.
Let's see a small example with two-number vectors:
A = [0.6, 0.8]
B = [0.8, 0.6]
Dot product = (0.6 x 0.8) + (0.8 x 0.6)
= 0.48 + 0.48
= 0.96
Both vectors have length 1, and the result is 0.96, which is very close to 1. So, A and B have very similar meanings.
A simple example in code
Now, let's see the code for using an embedding model. We will use the sentence-transformers library in Python.
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer("all-MiniLM-L6-v2")
sentences = [
"I love my puppy",
"I adore my dog",
"The stock market crashed today",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
scores = util.cos_sim(embeddings[0], embeddings[1:])
print(scores)
Here, we have done the following:
- We loaded a small embedding model called
all-MiniLM-L6-v2. - We gave it three sentences.
model.encodeturned each sentence into an embedding. The shape will be(3, 384), which means 3 sentences, each with 384 numbers.util.cos_simmeasured the cosine similarity between the first sentence and the other two.
When we run this, the score between "I love my puppy" and "I adore my dog" will be high, and the score between "I love my puppy" and "The stock market crashed today" will be much lower. The exact numbers depend on the model, but the pattern stays the same.
It works perfectly. The first two sentences get a high score, even though they use different words.
Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.
Embedding model vs LLM
Many people get confused between an embedding model and an LLM, because both are built on the Transformer. Let me tabulate the differences between an embedding model and an LLM for your better understanding.
| Point | Embedding Model | LLM |
|---|---|---|
| Input | Text | Text |
| Output | A fixed list of numbers | New text, one token at a time |
| Main job | Understand and compare meaning | Generate answers and content |
| Size | Usually small and fast | Usually large and slower |
| Cost per use | Very low | Higher |
| Example use | Search, matching, grouping | Chat, writing, reasoning |
They often work together, as we will see next.
Where do we use embedding models?
Let's see the most common use cases.
- Semantic search: Search by meaning instead of exact words. If we search "cheap phone", we can also find a product described as "budget smartphone".
- RAG (Retrieval-Augmented Generation): Before an LLM answers a question, we use embeddings to find the most relevant documents from our own data, and give them to the LLM. This way, the LLM can answer using our private documents.
- Recommendations: Suggest articles, songs, or products that are close in meaning to what a user already liked.
- Clustering: Group thousands of customer reviews into topics automatically, because similar reviews land close together on the map.
- Duplicate detection: Find two support tickets that ask the same thing in different words.
The common flow in semantic search and RAG looks like below:
Step A (once):
All documents ---> Embedding Model ---> Store vectors in a database
Step B (each query):
User question ---> Embedding Model ---> Query vector
↓
Find the closest stored vectors (highest similarity)
↓
Return the matching documents
Here, we embed all our documents only once and store the vectors in a vector database, which is a database built for storing and searching vectors. Then, for every question, we embed only the question and find the nearest documents. This is very fast.
If we want to go deep into Embeddings, RAG, and Vector Databases, and even build an AI Tutor from scratch, we cover all of it in our AI and Machine Learning Program at Outcome School.
Limitations of embedding models
Embedding models are powerful, but they have some limits that we must know.
- Fixed input length: Every model can read only a limited number of tokens at a time. Very long text gets cut off, so we usually split long documents into smaller chunks first.
- Different models, different maps: Embeddings from one model cannot be compared with embeddings from another model. Each model builds its own map. If we change the model, we must re-embed all our data.
- Similar is not always correct: "The movie was good" and "The movie was not good" can still land fairly close, because they share most words and the same topic. Embeddings capture topic very well, but they can miss small details like "not".
- Domain knowledge: A general model can perform poorly on very special text like medical or legal documents. In that case, we use a model trained on that kind of text, or we fine-tune one, which means we train a general model a little more on our own special text.
Now, we must have understood how an embedding model works.
Frequently Asked Questions
Can we compare embeddings from two different embedding models?
No. Embeddings from one model cannot be compared with embeddings from another model, because each model builds its own map of meanings. If we change the embedding model, we must re-embed all our data with the new model so that every vector lives on the same map.
Do a short sentence and a long paragraph get embeddings of the same size?
Yes. An embedding model always gives a list of the same fixed length, no matter how long the input is. For example, all-MiniLM-L6-v2 turns both a short sentence and a long paragraph into 384 numbers. This is why we can compare any two texts in the same way.
Does a single number in an embedding have a meaning?
No. A single number in an embedding does not have a clear human meaning like "this number means animal". The meaning lives in the whole list together. The model is never told what each number means; it only learns which texts are similar and which are not, and the map of meanings appears from that training.
Can an embedding model understand negation like "not"?
Not always. "The movie was good" and "The movie was not good" can still land fairly close, because they share most words and the same topic. Embeddings capture the topic very well, but they can miss small details like "not", so similar is not always correct.
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
You can connect with me on:
Follow Outcome School on:
