How does the Context Window work?

A Context Window is the total amount of text, measured in tokens, that a Large Language Model can read and remember at one time. It is the model's short-term memory.

In this blog, we will learn about how the Context Window works in Large Language Models. We will also see why we need it, what tokens are, how the model reads everything inside it, what happens when it gets full, how to manage it, and the common mistakes we must avoid.

We will cover the following:

• What is a Context Window?
• Why do we need a Context Window?
• What are tokens?
• How does the model read the Context Window?
• A step-by-step walkthrough of a conversation
• What happens when the Context Window gets full?
• How to manage the Context Window
• Common mistakes while using the Context Window

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is a Context Window?

A Context Window is the total amount of text a model can read and remember at one time while answering us.

We can think of it as the model's short-term memory.

Here, context means everything the model needs to know to answer well. This includes our question, our earlier messages, and any documents we paste.

So, a Context Window is a fixed-size space that holds all the text the model is currently looking at.

Let's say we are talking to an AI assistant. Everything we typed and everything the model replied sits inside this window. The model reads this whole window before writing the next word.

Why do we need a Context Window?

To understand why we need it, let's first understand a problem.

A model does not have permanent memory of our chat. Every time we send a message, the model starts fresh and reads the text we give it.

Suppose we tell the model, "My name is Amit." Then in the next message we ask, "What is my name?"

How does the model answer correctly? The answer is the Context Window.

When we send the second message, the whole conversation is packed and sent again to the model. So, the model sees both messages together inside the Context Window. It reads "My name is Amit" and then answers "Your name is Amit."

We can picture the flow as below:

      We send message 2
              ↓
+----------------------------+
| Context Window             |
| "My name is Amit."         |
| "What is my name?"         |
+----------------------------+
              ↓
         +---------+
         |  Model  |
         +---------+
              ↓
     "Your name is Amit."

Here, we can see that both the old message and the new question travel together inside the Context Window. The model reads the whole window, finds "My name is Amit", and answers correctly.

Note: The model itself stays the same. The memory of the conversation lives inside the Context Window that we send each time.

What are tokens?

Before we go deeper, we must know about tokens, because the Context Window is measured in tokens, not in words.

A token is a small piece of text. It can be a whole word, a part of a word, or even a single character.

The model does not read full sentences directly. It first breaks the text into small pieces called tokens, and then it reads those pieces.

Let's see an example. Consider the sentence below:

I love learning

This sentence can be broken into tokens like below:

["I", " love", " learn", "ing"]

Here, we can see that the word "learning" got split into two tokens, "learn" and "ing". So, three words became four tokens.

A rough rule for English is that one token is about four characters, and 100 tokens are about 75 words. This is just for the sake of understanding, the exact count depends on the words we use.

So, when we say a model has a Context Window of 8,000 tokens, it means the model can hold about 6,000 words of text at one time. This includes both our text and the model's reply.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

How does the model read the Context Window?

Now that we have learned about tokens, it is time to learn how the model uses the window.

The model reads all the tokens in the Context Window together. It looks at how each token relates to the tokens that come before it. This is how it understands the meaning of the whole conversation.

Then the model predicts the next token. After that, it adds this new token back into the window and predicts the next one. It keeps doing this until the answer is complete.

So, the reply is built one token at a time, and each new token is decided by reading the entire window again.

This is why the Context Window matters so much. If important information is inside the window, the model can use it. If it is outside the window, the model cannot see it at all.

Let me tabulate the parts that usually sit inside the Context Window for your better understanding.

Here, we can notice that all of these share the same fixed space. They all compete for room inside the window.

To learn Tokenization, Self-Attention, and how to build a Large Language Model (LLM) from scratch, check out our AI and Machine Learning Program at Outcome School.

A step-by-step walkthrough of a conversation

The best way to learn this is by taking an example. Let's say our model has a small Context Window of only 100 tokens, just for the sake of understanding.

Step 1: We send "My name is Amit." This takes around 5 tokens. The window now holds 5 tokens. We have 95 tokens of room left.

Step 2: The model replies "Nice to meet you, Amit." This reply also goes into the window. Now the window holds around 12 tokens.

Step 3: We ask "What is my name?" This message is added too. The full conversation so far is sent to the model. The model reads everything, sees "My name is Amit", and replies "Your name is Amit."

Step 4: We keep chatting. With every message, the window keeps filling up. The total keeps growing toward the 100-token limit.

So, the window grows as the conversation grows. Each new turn carries all the older turns along with it.

This is why the model seems to remember us, even though it has no permanent memory.

We have a detailed blog on how ChatGPT remembers you that explains how saved memories work on top of this.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

What happens when the Context Window gets full?

Now, the next big question is: what happens when the conversation becomes longer than the window?

The window has a hard limit. We cannot fit more tokens than the limit allows.

Let's say our window is 100 tokens, and our conversation has now grown to 130 tokens. The extra 30 tokens cannot fit.

So, something must be removed. In many chat applications, the oldest messages are pushed out to make room.

Let's visualize this as below:

Conversation = 130 tokens
+---------------------+
| "My name is Amit."  |
| older reply         |
| recent message      |
| recent reply        |
| newest question     |
+---------------------+
           ↓
Window limit = 100 tokens, so the oldest messages are pushed out
           ↓
+---------------------+
| recent message      |
| recent reply        |
| newest question     |
+---------------------+
Kept inside the window

Here, we can see that the newest messages stay inside the window, while the oldest messages, including "My name is Amit.", fall out because there is no room left for them.

This leads to a real problem. If we told the model our name at the very start, and that message gets pushed out, the model can no longer see it. So, when we ask "What is my name?" later, the model does not know the answer anymore.

This is why a very long chat sometimes makes the model "forget" things we said earlier. The information is no longer inside the Context Window.

Note: Some tools handle this differently. Instead of dropping old messages, they may create a short summary of the old chat and keep the summary inside the window. We will learn about this next.

How to manage the Context Window

Now, let's learn how we can use the Context Window well so the model gives better answers.

We must keep the most important information inside the window and avoid filling it with text the model does not need.

First, we must put the important details close to our question. Since the window has limited room, we must not waste it on text that does not help.

Then, for long chats, we can summarize the old part. We replace many old messages with one short summary. This frees up tokens while keeping the key facts.

After that, when we paste a large document, we must paste only the relevant part. If we paste a 50-page file but only need one page, we are wasting most of the window.

Finally, we must remember that the reply also takes tokens. If the window is almost full with our input, there is very less room left for the answer. So, we must leave enough space for the model to write a complete reply.

Let's see a small example of summarizing to save tokens. Suppose we have a long chat, and we compress it like below:

Old long chat (about 400 tokens):
"My name is Amit. I am from India. I teach AI and Machine Learning.
I am building a course. I work at Outcome School... (many more lines)"

Short summary (about 30 tokens):
"User is Amit from India. Teaches AI and ML at Outcome School.
Building a course."

Here, we have replaced 400 tokens of old chat with a 30-token summary. The key facts are still there, but now we have a lot more room in the window for new messages and the reply.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Common mistakes while using the Context Window

Most of the time, we do mistakes while using the Context Window, so let's learn about them.

Mistake 1: We paste a huge document and expect the model to read all of it perfectly. If the document is larger than the window, part of it gets cut off and is never read, or the request fails.

Mistake 2: We keep a single chat going forever for many different topics. The window fills with old, unrelated text. The model then mixes things up. For a new topic, it is better to start a fresh chat.

Mistake 3: We bury the real question at the very top and then add pages of text after it. The important part can get lost. We must keep our actual question clear and near the relevant context.

We have a detailed blog on the Lost in the Middle problem in LLMs that covers how position inside a long context changes what the model uses.

Now we must have understood how the Context Window works, what happens when it gets full, and how to manage it.

Frequently Asked Questions

Does the model's reply also use the Context Window?

Yes. The model's reply also takes tokens inside the Context Window. If the window is almost full with our input, there is very little room left for the answer. So, we must leave enough space for the model to write a complete reply.

How many words fit in a Context Window of 8,000 tokens?

About 6,000 words fit in a Context Window of 8,000 tokens. A rough rule for English is that one token is about four characters, and 100 tokens are about 75 words. The exact count depends on the words we use, and this space includes both our text and the model's reply.

Why does an LLM forget things we said earlier in a long chat?

An LLM forgets earlier things because they are no longer inside the Context Window. When a conversation grows longer than the window, many chat applications push out the oldest messages to make room. If our name was in one of those messages, the model can no longer see it.

Should we start a new chat for a new topic?

Yes. For a new topic, it is better to start a fresh chat. If we keep a single chat going for many different topics, the window fills with old, unrelated text, and the model then mixes things up.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.