How does HyDE work in RAG?

In this blog, we will learn about how HyDE works in RAG, which is the clever trick of searching with a fake answer. We will also see why searching with the plain question is weak, why a fake answer searches better, how HyDE works step by step with a worked example, and when to use it in the real world.

We will cover the following:

What is RAG in simple words
What is the search problem in RAG
Why searching with the question is weak
What is HyDE
Why searching with a fake answer works better
How HyDE works step by step
A worked example of HyDE
A simple code example of HyDE
Advantages of HyDE
Disadvantages of HyDE
When to use HyDE
Summary

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is RAG in simple words

Before jumping into HyDE, we must know what RAG is.

RAG stands for Retrieval-Augmented Generation. It is a way to make an AI model answer using our own documents instead of only using what it already knows.

RAG = Retrieval + Augmented + Generation.

Let's break it down:

Retrieval means searching and fetching the right pieces of text from our documents.
Augmented means adding the retrieved information to the prompt (context) before sending it to the AI model.
Generation means the AI model writing an answer using those pieces of text.

Let's say we have a big company handbook with thousands of pages. A new employee asks, "How many holidays do I get in a year?" The AI model has never read our private handbook. So, it cannot answer correctly on its own.

So, here comes RAG to the rescue. First, RAG searches the handbook and finds the few lines that talk about holidays. Then, it gives those lines to the AI model. Finally, the model reads those lines and writes a clear answer.

This is how RAG lets the model answer from our own data. If we want a deeper look at the full RAG flow, we can read the blog on Agentic RAG.

What is the search problem in RAG

The whole power of RAG depends on one thing. The search step must find the right text.

If the search brings the wrong pages, the model gets wrong information. And then the answer will also be wrong.

So, the search step is the heart of RAG. Now, let's understand how this search actually happens.

The search in RAG does not work by matching exact words. It works by matching meaning.

To match meaning, we convert text into numbers. These numbers are called an embedding.

An embedding is a list of numbers that represents the meaning of a piece of text.

In simple words, an embedding is the meaning of text written in the language of numbers.

The important rule is simple. Two texts with similar meaning get similar numbers. Two texts with different meaning get different numbers.

We also cut our big documents into small pieces. Each small piece is called a chunk, which means a short block of text, like a single paragraph.

So, in RAG, we do the following. First, we convert every chunk of our document into an embedding and store it. Then, when a user asks a question, we convert the question into an embedding too. After that, we compare the question embedding with all the stored embeddings. Finally, we pick the chunks whose embeddings are closest to the question embedding.

This is how RAG finds the right text by comparing meaning.

The store that holds all these embeddings and finds the closest ones is called a vector database. We have a detailed blog on how a vector database works that explains this in depth.

Why searching with the question is weak

Now, here is the catch.

A question and an answer often do not look alike. They live in different worlds.

Let's say the user asks, "Why is my laptop battery draining so fast?"

The real answer inside our documents may say something like this:

Background applications, high screen brightness, and old battery health reduce battery life. Closing unused apps and lowering brightness improves battery duration.

Notice the problem. The question is short and uses words like "draining" and "fast". The answer is detailed and uses words like "background applications", "brightness", and "battery health".

A question and its answer are written in two different styles. So, their embeddings are not always close to each other.

Means, when we search using only the question, the meaning gap is large. The search may miss the perfect answer that is actually sitting in our documents.

This is the weakness of searching with the question. We are comparing a question to answers, but questions and answers are shaped differently.

What is HyDE

So, here comes HyDE into the picture.

HyDE stands for Hypothetical Document Embeddings. It is a technique where we first ask the AI model to write a fake answer to the question, and then we search using that fake answer instead of the question.

In simple words, HyDE = Hypothetical + Document + Embeddings.

Let's break it down:

Hypothetical means imaginary or made-up. It is a guess answer.
Document means a piece of text that looks like a real answer.
Embeddings means the meaning of that text written as numbers.

The core idea is one beautiful trick. An answer looks like an answer. So, if we want to find a real answer, we must search using something that looks like an answer.

The question does not look like an answer. But a fake answer does look like an answer.

So, we tell the AI model, "Pretend you know the answer and write it." This made-up answer is called the hypothetical document.

We do not show this fake answer to the user. We do not even trust that it is correct. We only use it as a better search key, which means the text we search with.

This is the heart of HyDE. We search with a fake answer to find the real answer.

Why searching with a fake answer works better

Now, the question is, why does a fake answer help when it may even contain wrong facts?

The answer is about shape, not facts.

When the model writes a fake answer about battery draining, it naturally uses words like "background apps", "screen brightness", and "battery health". These are the same kinds of words that the real answer in our documents uses.

So, the fake answer and the real answer now look alike. Their embeddings become close. And close embeddings mean a strong search match.

Let's picture this meaning space as below:

        Meaning space (similar meaning sits closer)

   [ Question ]                                  ----- far -----+
   "battery draining fast"                                      |
                                                                v
                                          +--------------------------+
   [ Fake answer ]  ----- close ------->  | [ Real answer chunk ]    |
   "background apps, brightness,          | "background apps, high   |
    battery health"                       |  brightness, battery     |
                                          |  health reduce life"     |
                                          +--------------------------+

Here, we can see that the question sits far from the real answer chunk, because it is shaped differently. The fake answer sits close to the real answer chunk, because both use the same answer-like words. So, searching with the fake answer lands much nearer to the real answer.

We are not searching with the question. We are searching with something that is shaped like the answer.

Even if the fake answer gets a fact wrong, it still points the search in the right direction. The wrong facts get thrown away later, because we only use the real documents to write the final answer.

This is how a fake answer becomes a powerful search key.

To learn how embeddings, Vector Databases, and RAG fit together, check out our AI and Machine Learning Program at Outcome School.

How HyDE works step by step

The best way to learn this is by taking an example. Let's walk through HyDE step by step.

Step 1: The user asks a question. For example, "Why is my laptop battery draining so fast?"

Step 2: We send this question to the AI model with a simple instruction. We say, "Write a short answer to this question, even if you are not sure." The model writes a fake answer, which is our hypothetical document.

Step 3: We convert this fake answer into an embedding, which means we turn it into a list of numbers that holds its meaning.

Step 4: We compare this embedding with all the stored embeddings of our real document chunks. We pick the chunks that are closest in meaning.

Step 5: We take these real chunks and send them to the AI model along with the original question.

Step 6: The model reads the real chunks and writes the final answer. This final answer is based on our real documents, not on the fake one.

So, the fake answer is used only in the middle, only to improve the search. The final answer comes from real data.

We can picture the full flow as below:

   [ User question ]
          |
          v
   +-------------------+
   | AI model writes   |   <- "even if not sure"
   | a fake answer     |
   +-------------------+
          |
          v
   +-------------------+
   | Turn fake answer  |
   | into embedding    |
   +-------------------+
          |
          v
   +-------------------+        +----------------------+
   | Search the stored | -----> | Pick closest real    |
   | document chunks   |        | document chunks      |
   +-------------------+        +----------------------+
                                          |
              [ User question ] --------> |
                                          v
                               +----------------------+
                               | AI model writes the  |
                               | final answer from    |
                               | real chunks only     |
                               +----------------------+
                                          |
                                          v
                                  [ Final answer ]

Here, we can see that the fake answer is used only in the middle to improve the search. The real chunks and the original question flow into the last step, and the final answer is written from the real documents only.

This is how HyDE works from start to finish.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

LLM
RAG
MCP
Agent
Fine-tuning
Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

A worked example of HyDE

Let me tabulate the difference between the normal RAG search and the HyDE search for your better understanding.

Stage	Normal RAG	HyDE
What we search with	The user's question	A fake answer written by the model
Shape of the search text	Short and question-like	Detailed and answer-like
Closeness to real answers	Often far in meaning	Often close in meaning
Final answer source	Real documents	Real documents

Here, we can see the key difference. Normal RAG searches with a question. HyDE searches with a fake answer. Both write the final answer from real documents, but HyDE usually finds better chunks to start with.

Now, let's see the meaning gap with simple numbers, just for the sake of understanding.

Suppose closeness is measured from 0 to 1, where 1 means identical meaning and 0 means completely different meaning.

Question embedding compared to the real answer chunk: closeness is 0.55.
Fake answer embedding compared to the real answer chunk: closeness is 0.88.

Here, we can notice that the fake answer is much closer to the real answer. So, the search picks the correct chunk more easily with HyDE.

Note: These numbers are just for the sake of understanding. The real values depend on the model and the data.

A simple code example of HyDE

Now, let's see the code for HyDE in a simple form. We will write it in python as below:

def hyde_search(question, model, vector_store):
    # Step 1: ask the model to write a fake answer
    prompt = "Write a short answer to this question: " + question
    fake_answer = model.generate(prompt)

    # Step 2: turn the fake answer into an embedding
    fake_answer_embedding = model.embed(fake_answer)

    # Step 3: search using the fake answer's embedding
    top_chunks = vector_store.search(fake_answer_embedding, top_k=5)

    return top_chunks

Here, we have done three simple things:

First, we ask the model to write a fake_answer to the question.
Then, we convert that fake_answer into a fake_answer_embedding, which is its meaning as numbers.
After that, we call vector_store.search with that embedding to get the top five closest chunks.

Notice that we never search with the original question. We search with the fake_answer. That is the whole trick of HyDE.

Now, let's see how we use these chunks to write the final answer as below:

def answer_with_hyde(question, model, vector_store):
    chunks = hyde_search(question, model, vector_store)

    # combine the real chunks into one block of context
    context = "\n".join(chunks)

    # ask the model to answer using only the real chunks
    final_prompt = (
        "Use the context below to answer the question.\n"
        "Context: " + context + "\n"
        "Question: " + question
    )
    return model.generate(final_prompt)

Here, we can see the full flow. First, we get the real chunks using hyde_search. Then, we join them into one context, which means one block of helpful text. After that, we send the real context and the original question to the model. Finally, the model writes the answer using only the real documents.

So, the fake answer helped us find the chunks, but it never touched the final answer. The problem is solved.

If we want to go deep into RAG and build an AI Tutor from scratch, we cover it in our AI and Machine Learning Program at Outcome School.

Advantages of HyDE

Now, let's understand why HyDE is useful.

Advantages:

It closes the gap between questions and answers, because we search with answer-shaped text.
It often finds better chunks, which leads to better final answers.
It works well even when the user's question is short or vague.
It needs no extra training of the model. We only change how we search.

Disadvantages of HyDE

But again, no technique is perfect. HyDE has some costs too.

Disadvantages:

It is slower, because we make an extra call to the model to write the fake answer.
It costs more, because that extra call uses extra computing power.
The fake answer can drift off topic, and then the search may go in the wrong direction.
For very simple questions, the question itself may already be enough, so HyDE adds cost without much gain.

So, HyDE makes our search smarter, but we pay with a little extra time and cost.

When to use HyDE

Now, the question is, when must we use HyDE.

We should use HyDE when our questions and answers look very different from each other. This happens a lot with short questions, support tickets, and search over technical documents.

But, for very short and direct lookups, plain RAG search may be enough. So, we must choose HyDE based on our use case.

A good practice is to test both. We can run plain RAG search and HyDE search on the same set of questions, and then we keep the one that finds better chunks for our data.

Summary

Let's quickly recap what we have learned.

RAG searches our documents by matching meaning using embeddings. But a question and its answer are shaped differently, so searching with the question alone is weak.

So, here comes HyDE. We ask the model to write a fake answer first. We search with that fake answer, because it is shaped like a real answer and lands close to the real chunks. Then, we write the final answer using only the real documents.

This way we can use HyDE to solve the search problem in RAG in a very simple way. We search with a fake answer to find the real one.

Prepare yourself for AI Engineering Interview: AI Engineering Interview Questions

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.

Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.