Outcome School

How does RAG work?

Authors
How does RAG work?

RAG, or Retrieval-Augmented Generation, first finds the right information for our question and then gives it to the LLM, so that the LLM can write an accurate answer.

In this blog, we will learn about how RAG works. We will also see why a plain LLM is not enough, how indexing, embeddings, retrieval, and generation work together, a complete walkthrough of the full flow, and why RAG works so well.

We will cover the following:

  • The problem with an LLM
  • What is RAG?
  • How does RAG work
  • Indexing: preparing the knowledge
  • Embeddings and the vector database
  • Retrieval: finding the right information
  • Generation: writing the final answer
  • A complete walkthrough
  • Why RAG works so well

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

The problem with an LLM

An LLM (Large Language Model) is powerful, but it has a big limitation.

An LLM only knows what it read during its training. Once the training is over, its knowledge is frozen.

Let's say the LLM was trained with data up to last year. Now, we ask it about something that happened today. The LLM has no idea about it.

So, the LLM gives us a wrong answer, or it simply makes up something that sounds correct.

When an LLM makes up an answer that sounds confident but is actually wrong, we call it as a hallucination.

There is one more problem. An LLM does not know about our private data.

Suppose we run a company. We have our own internal documents, our own product manuals, and our own customer records. The LLM never read any of this. So, it cannot answer questions about our company.

So, we have two big problems:

  • The LLM does not know recent information.
  • The LLM does not know our private or company-specific information.

Now, the question is: how can we give the LLM the right information so that it answers correctly? The answer is RAG.

What is RAG?

RAG stands for Retrieval-Augmented Generation. It is a technique where we first find the right information, and then we give that information to the LLM so that it can write an accurate answer.

Let's decompose the term to understand it better.

RAG = Retrieval + Augmented + Generation

  • Retrieval means finding and fetching the right information.
  • Augmented means adding that information to help the LLM.
  • Generation means the LLM generating the final answer using that information.

Let's understand this with an analogy.

Suppose a student is sitting in an open-book exam. The student is smart, but the student does not remember every fact. So, before answering, the student opens the right book, finds the right page, reads it, and then writes the answer.

Here, the student is the LLM. The act of opening the book and finding the right page is the retrieval. Writing the answer using that page is the generation.

This is exactly what RAG does. It turns a closed-book exam into an open-book exam for the LLM.

Now, we have understood what RAG is. It's time to learn how it actually works.

How does RAG work

RAG works in two main phases.

The first phase is Indexing. This is the preparation phase, where we organize our knowledge so that it can be searched quickly.

The second phase is the Query phase. This phase has two steps inside it: Retrieval and Generation.

We can picture the two phases as below:

   INDEXING PHASE (done once)
   --------------------------
   Documents --> Chunks --> Embeddings --> Vector Database


   QUERY PHASE (done for every question)
   -------------------------------------
   Question --> Retrieval --> Generation --> Answer
                   |
                   +--> reads from the Vector Database

Here, we can see that the Indexing phase prepares our knowledge once and stores it in the Vector Database. The Query phase then uses that stored knowledge every time a user asks a question, first doing Retrieval and then Generation.

We will learn about each of them in detail.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Let's start with the indexing phase, because everything depends on it.

Indexing: preparing the knowledge

Before we can search for information, we must first organize it.

Let's say we have a big company handbook of 500 pages. We cannot give all 500 pages to the LLM for every question. It is too much. It is slow, and it is costly.

So, we break the big document into small pieces. Each small piece is called a chunk.

A chunk is a small piece of text, usually a paragraph or a few sentences, taken from a larger document.

Why do we break it into chunks? Because when someone asks a question, we want to find only the small piece that answers it, not the whole book.

So, we follow these steps in indexing:

  • First, we collect all our documents.
  • Then, we break each document into small chunks.
  • After that, we convert each chunk into a special number form so that the computer can search it.

That special number form is called an embedding. Let's understand it now.

Embeddings and the vector database

An embedding is a list of numbers that represents the meaning of a piece of text.

A computer does not understand words like we do. So, we convert each chunk of text into a list of numbers. These numbers capture the meaning of the text.

Two pieces of text that have a similar meaning will have similar numbers. We can imagine these numbers as points on a map. Points with a similar meaning sit close together, and points with a different meaning sit far apart.

Let's take an example for the sake of understanding. Consider these three sentences:

  • "How do I reset my password?"
  • "I forgot my login details and want to change them."
  • "What is the price of the premium plan?"

The first two sentences mean almost the same thing. So, their embeddings will be close to each other. The third sentence is about a different topic. So, its embedding will be far from the other two.

Let's visualize this as below:

   meaning map
   -----------

      [reset my password]
            *
            * <- close together (similar meaning)
      [forgot my login details]



                                  [price of premium plan]
                                          *
                                  (far away, different meaning)

Here, we can see that the two sentences about login problems sit close together because their meaning is similar, while the sentence about pricing sits far away because its meaning is different. This closeness in the meaning map is exactly what lets the computer compare meaning instead of exact words.

Now, where do we store all these embeddings? We store them in a vector database.

A vector database is a special database that stores embeddings and can quickly find the ones that are most similar to a given embedding.

A list of numbers is also called a vector. That is why it is called a vector database.

So, at the end of indexing, our handbook is broken into chunks, every chunk is converted into an embedding, and all embeddings are stored in the vector database.

The indexing phase is ready now. We do this only once, or whenever our documents change.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Now that we have prepared our knowledge, it's time to learn how we answer a real question.

Retrieval: finding the right information

This is the first step of the query phase.

Let's say a user asks our system a question:

How do I reset my password?

Now, the system must find the chunks that can answer this question.

First, we take the user's question and convert it into an embedding too, using the same method we used for the chunks.

Then, we take this question embedding and search the vector database. The vector database compares the question embedding with all the stored chunk embeddings.

It finds the chunks whose embeddings are closest in meaning to the question. These closest chunks are the most relevant pieces of information.

This search by meaning is called a semantic search.

A semantic search is a search that finds results based on meaning, not based on matching the exact words.

So, even if the user typed "I cannot log in" instead of "reset my password", the semantic search can still find the right chunk, because the meaning is similar.

We usually pick the top few chunks. For example, the top three chunks that are closest in meaning. We pick a few instead of just one, because the full answer may be spread across more than one chunk.

To pick the best few chunks even more accurately, many systems add one more step that reorders them. We have a detailed blog on how a Reranker works that explains this step by step.

The retrieval step is done. We now have the right pieces of information. Now, let's move to the final step.

Generation: writing the final answer

This is the second step of the query phase.

Now, we have two things in our hand:

  • The user's original question.
  • The top relevant chunks that we retrieved.

We combine both of these into a single instruction for the LLM. This combined instruction is called a prompt.

A prompt is the full input text that we give to the LLM to tell it what to do.

The prompt looks like below:

You are a helpful assistant. Answer the question using only the
information given below.

Information:
"To reset your password, go to Settings, click Security, then click
Reset Password. You will receive an email with a reset link."

Question:
How do I reset my password?

Here, we can see that we did not just send the question alone. We also attached the retrieved information on top of the question. For the sake of understanding, we have shown only one chunk of information here, but in real use we would attach the top few chunks.

This is the "Augmented" part of RAG. The word augmented means we added the right facts to the question.

Now, the LLM reads this prompt. It already knows how to write in good human language. So, it uses the given information to write a clear and correct answer like below:

To reset your password, open Settings and click on Security. Then click
Reset Password. You will get an email with a link to set a new password.

Here, we can notice that the answer is accurate. The LLM did not make up anything. It used the real facts that we retrieved and provided.

The LLM now answers based on our own data.

This is how the generation step works.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

A complete walkthrough

Now, let's connect everything in one clean flow, step by step.

Step 1: We take our documents and break them into small chunks.

Step 2: We convert each chunk into an embedding and store all of them in the vector database. This was the indexing phase.

Step 3: The user asks a question.

Step 4: We convert the question into an embedding.

Step 5: We search the vector database and retrieve the top chunks that are closest in meaning to the question.

Step 6: We combine the question and the retrieved chunks into a single prompt.

Step 7: We send this prompt to the LLM, and the LLM generates the final answer using the facts we provided.

This is the complete RAG flow. The first two steps are the one-time preparation. The last five steps happen every time a user asks a question.

We can picture the complete flow as below:

   Documents
      |
      ↓
   Chunks ---> Embeddings ---> +-----------------+
                               | Vector Database |
                               +-----------------+
                                        ↑
                                        | search by meaning
                                        |
   Question ---> Embedding -------------+
      |                                 |
      |                                 ↓
      |                          Top relevant chunks
      |                                 |
      +-------------+-------------------+
                    |
                    ↓
                 Prompt  (question + retrieved chunks)
                    |
                    ↓
                  LLM
                    |
                    ↓
              Final Answer

Here, we can see that the question and the retrieved chunks are combined into a single Prompt, and the LLM uses that prompt to write the Final Answer. The top part of the diagram is the one-time indexing, and the bottom part is what happens for every question.

To master RAG, Vector Databases, and Design a RAG System (Chat with Your Documents), and to build an AI Tutor from scratch, check out our AI and Machine Learning Program at Outcome School.

Why RAG works so well

Now, let's understand why RAG is so useful.

It gives fresh and updated answers. We do not need to retrain the huge LLM every time our data changes. We just update the chunks in the vector database, and the answers become up to date.

It works on our private data. We can feed our own company documents into the vector database. So, the LLM can answer questions about our own products, our own policies, and our own customers.

It reduces hallucinations. Since we hand the real facts to the LLM, the LLM is much less likely to guess. It answers based on the information we gave.

It can show the source. Since we know exactly which chunks were used, we can tell the user where the answer came from. This builds trust.

It is cheaper and faster. Retraining a big LLM costs a lot of money and time. With RAG, we only update the vector database.

This way we can use RAG to solve the interesting problem of giving an LLM new and private knowledge.

Some questions need more than one search, or need to pick between many sources. We have a detailed blog on Agentic RAG that covers how an AI Agent drives the retrieval in a loop.

Now we must have understood how RAG works, why a plain LLM is not enough, and why RAG works so well.

Frequently Asked Questions

Do we need to retrain the LLM to use RAG?

No. We do not need to retrain the LLM. When our data changes, we just update the chunks in the vector database, and the answers become up to date. This is also why RAG is cheaper and faster than retraining, which costs a lot of money and time.

Is the indexing phase done for every question?

No. Indexing is done only once, or whenever our documents change. In indexing, we break our documents into chunks, convert every chunk into an embedding, and store all of them in the vector database. Only retrieval and generation happen every time a user asks a question.

Why does RAG retrieve more than one chunk?

RAG usually picks the top few chunks, like the top three, because the full answer may be spread across more than one chunk. Picking a few instead of just one gives the LLM all the pieces it needs to write a complete answer.

Can RAG show where the answer came from?

Yes. Since we know exactly which chunks were used to build the prompt, we can tell the user where the answer came from. Showing the source builds trust, because the user can check the original text behind the answer.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.