How does a Vision-Language Model (VLM) work?

A Vision-Language Model (VLM) is an AI model that can look at an image and read text together, and then answer in text. It works by turning the image into a list of numbers that look like word tokens to a language model, so the language model can read the image just like it reads our question.

In this blog, we will learn about how a Vision-Language Model (VLM) works. We will also see how a computer sees an image as numbers, how a vision encoder cuts an image into patches and understands them, how CLIP connected images and text, how a projector turns image patches into tokens that an LLM can read, how a VLM is trained, and where VLMs work well and where they fail.

We will cover the following:

• What is a Vision-Language Model?
• How does a computer see an image?
• The main problem: images are not words
• The three parts of a VLM
• Part 1: The vision encoder
• How did CLIP teach images and text the same language?
• Part 2: The projector
• Part 3: The language model
• VLM in code
• How is a VLM trained?
• Other ways to connect vision and language
• Where VLMs work well and where they fail

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is a Vision-Language Model?

A Vision-Language Model, or VLM, is an AI model that takes both an image and text as input, understands them together, and gives text as output.

Let's say we give a VLM a photo of a kitchen and ask, "What can I cook with the things on this table?". The VLM looks at the photo, sees eggs, tomatoes, and onions, and replies, "You can make an omelette with tomatoes and onions."

We use VLMs every day. When we upload a photo to ChatGPT, Claude, or Gemini and ask a question about it, a VLM is doing the work.

A VLM is one kind of model that works with more than one type of data. We have a detailed blog on Multimodal AI that covers this bigger family end to end.

Some common tasks of a VLM:

• Describing what is in a photo.
• Answering questions about an image.
• Reading text from a screenshot, a bill, or a document.
• Understanding charts and diagrams.

Now, to understand how a VLM works, we must first understand how a computer sees an image.

How does a computer see an image?

A computer does not see a cat or a tree. It sees only numbers.

An image is a grid of tiny colored dots called pixels. Each pixel has three numbers: how much Red, Green, and Blue light it has. Each number goes from 0 to 255.

For example, a pure red pixel is [255, 0, 0], and a white pixel is [255, 255, 255].

So, a photo of size 336 x 336 pixels is a big grid of 336 x 336 = 112,896 pixels, and each pixel has 3 numbers. That is 338,688 numbers in total.

A cat photo (what we see)
            ↓
What the computer sees:

[[ [120, 85, 60], [122, 88, 61], ... ],
 [ [118, 84, 59], [121, 86, 60], ... ],
 ... ]

These raw numbers do not directly say "cat". Something must turn these raw numbers into meaning.

The main problem: images are not words

An LLM breaks text into small pieces called tokens, and turns each token into a list of numbers called an embedding. The LLM only understands a sequence of these embeddings, and it predicts the answer one token at a time. An image is a grid of pixels. These are two completely different forms of data.

Let's understand this with an analogy. Suppose a writer knows only English. We show the writer a painting and ask for a story about it. But the writer is blind to paintings and can only read English text. What do we do?

We hire two helpers:

• A painting expert who looks at the painting and notes down its details in their own private notes.
• A translator who converts those private notes into English sentences that the writer can read.

Now, the writer reads the English notes plus our question, and writes the story.

This is exactly how a VLM works. The question is: how do we convert an image into something that looks like word tokens? The answer is the three parts of a VLM.

The three parts of a VLM

Most popular VLMs are made of three parts:

          Image                          Text question
            ↓                                  ↓
   [ Vision Encoder ]                          ↓
    (painting expert)                          ↓
            ↓                                  ↓
      patch vectors                            ↓
            ↓                                  ↓
      [ Projector ]                            ↓
      (translator)                             ↓
            ↓                                  ↓
      visual tokens                       text tokens
            ↓                                  ↓
            +-----------------+----------------+
                              ↓
                     [ Language Model ]
                          (writer)
                              ↓
                         Text answer

Here, we have:

• Vision encoder: Understands the image and turns it into a set of vectors, which are lists of numbers.
• Projector: Converts those vectors into the same "language" as the LLM's word embeddings.
• Language model: Reads the image tokens and the text tokens together and writes the answer.

Now, let's understand each part one by one.

Part 1: The vision encoder

The most common vision encoder is the Vision Transformer, also called ViT. The Transformer is the same architecture used inside LLMs. ViT uses it for images.

Step 1: The image is cut into small squares called patches. For example, a 336 x 336 image cut into 14 x 14 pixel patches gives 24 patches per side. That is 24 x 24 = 576 patches.

+----+----+----+----+
| P1 | P2 | P3 | P4 |
+----+----+----+----+
| P5 | P6 | P7 | P8 |
+----+----+----+----+
| ...               |
+-------------------+

Here, each square is one patch. The patches are read in order, row by row, just like words in a sentence.

Step 2: Each patch is turned into a vector. Each 14 x 14 patch has 14 x 14 x 3 = 588 numbers. These are laid out in one long list and passed through a learned layer that turns them into a vector. Now, each patch is like one "word" of the image.

Step 3: The model adds a position embedding to each patch, so that it knows where the patch was in the image. Without this, the model would not know if the sky patch was at the top or the bottom.

Step 4: The patches pass through many Transformer layers. In each layer, using a step called attention, every patch looks at every other patch. A patch showing an ear looks at the patch showing an eye and the patch showing fur, and together they understand "this is a cat's face".

At the end, we get one vector for each patch. Each vector now carries meaning, not just raw colors. These are our painting expert's private notes.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Now, the next big question is: how does the vision encoder learn what things are? Here comes CLIP into the picture.

How did CLIP teach images and text the same language?

Many VLMs use a vision encoder taken from a model called CLIP, which was released by OpenAI in 2021. CLIP stands for Contrastive Language-Image Pre-training.

CLIP was trained on about 400 million pairs of images and their captions collected from the internet. For example, a photo of a dog with the caption "a brown dog playing in the park".

CLIP has two parts: an image encoder and a text encoder. The training works as below:

• Take a batch of images and their captions.
• Turn every image into a vector and every caption into a vector.
• Make the vector of each image close to the vector of its own caption.
• Make it far from the vectors of all the other captions in the batch.

This is called contrastive learning, because the model learns by contrasting the correct match with the wrong ones.

After this training, the image encoder of CLIP produces vectors that carry real meaning about the image, and that meaning is already connected to language. This is why it works well as the painting expert for a VLM.

Part 2: The projector

Now, we have the patch vectors from the vision encoder. But they are still in the vision encoder's own private language. The LLM cannot read them directly.

For example, the vision encoder can produce vectors of size 1,024, while the LLM expects embeddings of size 4,096. Even the meaning of each number is different.

So, here comes the projector to the rescue.

The projector is a small neural network that converts each patch vector into a vector that looks like a word embedding to the LLM.

The projector can be very simple. In the popular model LLaVA, the first version used a single linear layer, which means a multiplication by one learned table of numbers. The next version, LLaVA-1.5, used a small two-layer network called an MLP, which stands for Multi-Layer Perceptron.

After the projector, each patch becomes a visual token. For the LLM, a visual token looks just like a word token, even though it came from an image. This is our translator at work.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Part 3: The language model

Now, the LLM receives one long sequence:

[visual token 1] [visual token 2] ... [visual token 576] [What] [is] [in] [this] [image] [?]

The LLM reads this sequence exactly like it reads normal text. Through attention, the text tokens look at the visual tokens. The word "this" looks at the visual tokens and finds the cat patches.

Then the LLM generates the answer, one token at a time:

"This is an orange cat sleeping on a blue sofa."

This is our writer at work. The LLM never sees pixels. It only sees tokens. Some tokens came from words, and some came from the image.

VLM in code

Let's see the code for the shapes of the data as it flows through a VLM, using Python and NumPy. We are using random numbers, just for the sake of understanding the flow.

import numpy as np

image = np.random.rand(336, 336, 3)      # height, width, 3 colors (RGB)
patch = 14                               # each patch is 14 x 14 pixels

patches_per_side = 336 // patch          # 24
num_patches = patches_per_side ** 2      # 576
print("Patches:", num_patches)

vision_dim, llm_dim = 1024, 4096
patch_vectors = np.random.rand(num_patches, vision_dim)    # output of the vision encoder
projector = np.random.rand(vision_dim, llm_dim)            # learned during training
visual_tokens = patch_vectors @ projector
print("Visual tokens:", visual_tokens.shape)

text_tokens = np.random.rand(8, llm_dim)                   # "What is in this image?"
llm_input = np.concatenate([visual_tokens, text_tokens])
print("LLM input:", llm_input.shape)

Here, we have:

• image is a 336 x 336 picture with 3 color numbers per pixel.
• We cut it into 14 x 14 patches, which gives 24 x 24 = 576 patches.
• patch_vectors stands in for the output of the vision encoder: one vector of size 1,024 for each patch.
• projector is a learned table. The @ symbol means matrix multiplication. It converts each 1,024-size vector into a 4,096-size vector, the size the LLM expects.
• text_tokens stands in for the embeddings of our 8 question tokens.
• np.concatenate joins the visual tokens and the text tokens into one sequence.

This will print the following:

Patches: 576
Visual tokens: (576, 4096)
LLM input: (584, 4096)

Here, we can see that the LLM receives 584 tokens: 576 from the image and 8 from the text. All of them have the same size, so the LLM can read them together.

Note: One image used 576 tokens here, far more than our 8-token question. This is why images make a VLM slower and more costly than plain text.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

How is a VLM trained?

Now, let's understand how all three parts learn to work together. LLaVA is a good example because its recipe is simple.

We start with two parts that are already trained: a vision encoder from CLIP and an LLM that already knows language. Only the projector is new.

Step 1: We teach the translator. The vision encoder and the LLM are frozen, which means their parameters are not updated. Only the projector is trained on a large set of image-caption pairs. The task is to look at the image and produce its caption. This teaches the projector to convert image vectors into the LLM's language.

Step 2: We teach the model to follow instructions. The model is trained on examples of images with questions and good answers. For example, an image of a street with the question "Is it safe to cross?" and a helpful answer. In this step, the projector and the LLM are both trained. The vision encoder usually stays frozen. This teaches the VLM to have useful conversations about images.

This is how a VLM is trained cheaply, by reusing parts that are already good at their jobs.

To master Multimodal AI, Fine-tuning, and Large Language Models (LLMs) in depth, check out our AI and Machine Learning Program at Outcome School.

Other ways to connect vision and language

The projector design is the most common, but it is not the only one. Let me tabulate the main designs for your better understanding.

Here, we can notice the tradeoff. The projector design is simple and keeps all the details, but it uses many tokens. The query-based design uses fewer tokens, but it can lose small details.

The Flamingo design is built on cross attention, where the text looks at the image from inside the LLM. We have a detailed blog on Cross Attention in Transformers that explains how it works.

Where VLMs work well and where they fail

Where they work well:

• Describing photos and answering general questions about them.
• Reading text in screenshots, receipts, and documents.
• Understanding charts, tables, and diagrams.
• Helping people with low vision understand what is around them.

Where they fail:

• Hallucination: A VLM can describe things that are not in the image. For example, it can say there is a cup on the table when there is none. One big reason is that the LLM has strong habits from text. Cups are usually on kitchen tables, so it fills the gap with what is usually there.
• Counting and exact positions: Questions like "How many birds are in the sky?" or "Is the key to the left of the phone?" are often answered wrongly.
• Tiny details: When a big image is shrunk to fit the vision encoder, small text and small objects get blurred. Many newer VLMs fix this by cutting a large image into several tiles and encoding each tile, but that uses even more tokens.
• Cost: Each image adds hundreds or even thousands of tokens, which makes the model slower and more expensive.

Now we must have understood how a Vision-Language Model (VLM) works.

Frequently Asked Questions

Does the language model inside a VLM see the pixels of the image?

No. The language model never sees pixels. It only sees tokens. The vision encoder turns the image patches into vectors, and the projector turns those vectors into visual tokens that look like word tokens. The language model then reads the visual tokens and the text tokens together as one sequence.

Why does sending an image to a VLM cost more than sending text?

An image adds hundreds or even thousands of tokens to the input. For example, a 336 x 336 image cut into 14 x 14 patches gives 576 visual tokens, far more than a short 8-token question. More tokens make the model slower and more expensive.

Why do VLMs struggle to read small text in a large image?

When a big image is shrunk to fit the vision encoder, small text and small objects get blurred. Many newer VLMs fix this by cutting a large image into several tiles and encoding each tile on its own. This keeps the small details, but it uses even more tokens.

Is the vision encoder trained again when we build a VLM?

Usually not. In a simple recipe like LLaVA, the vision encoder comes from CLIP and is already trained. It stays frozen while the projector is trained first, and then the projector and the LLM are trained together on images with questions and good answers. This reuse is what keeps VLM training cheap.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.