How does Pretraining work?

Pretraining is the first and biggest stage of building an LLM, where the model reads a huge amount of text and learns to predict the next word again and again. By doing this billions of times, it learns language, facts, and patterns of reasoning.

In this blog, we will learn about how pretraining works. We will also see what simple game the model plays during pretraining, where the training text comes from and how it is cleaned, how the model learns from its mistakes, why it needs so much data and computing power, what we get at the end of pretraining, and how pretraining is different from the stages that come after it.

We will cover the following:

What is Pretraining?
Why is it called "pre" training?
The simple game: predict the next word
Why no human labels are needed
Step 1: Collecting the text
Step 2: Cleaning the text
Step 3: Breaking text into tokens
Step 4: The training loop
A tiny example of learning from a mistake
Why does pretraining need so much data and compute?
What do we get after pretraining?
What comes after pretraining?
Pretraining vs Fine-tuning

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is Pretraining?

Inside an LLM (Large Language Model), there are billions of numbers called weights. These weights store everything the model knows. At the very beginning, these numbers are random, so the model knows nothing. It writes only nonsense.

So, the question is: how does a model full of random numbers become a model that can write essays, answer questions, and write code?

The answer is: training. And the first and biggest part of this training is called pretraining.

Pretraining is the stage where a model reads a massive amount of general text and learns to predict the next word. This gives the model a broad understanding of language and the world.

Pretraining is like the model's school education. Before a person becomes a doctor or a lawyer, they first go to school and learn reading, writing, math, science, and general knowledge. Pretraining is that general schooling for the model.

Why is it called "pre" training?

"Pre" means "before". So, pretraining is the training that happens before everything else.

After pretraining, the model goes through more training stages to become a helpful assistant. Those later stages are smaller and more focused. Pretraining comes first and builds the foundation on which everything else stands.

   Random model
 (knows nothing)
        ↓
 [ Pretraining ]
        ↓
    Base model
 (knows language)
        ↓
[ Later training ]
        ↓
    Assistant
  (like ChatGPT)

The model we get right after pretraining is called a base model. We will learn about the base model in detail later in this blog.

The simple game: predict the next word

Now, let's understand what the model actually does during pretraining.

The model plays one very simple game again and again: guess the next word.

Let's say the model sees this sentence:

The cat sat on the ___

The model has to guess what comes next. A good guess is "mat". A bad guess is "banana".

To guess the next word correctly, again and again, across all kinds of text, the model is forced to learn many things:

To complete "The capital of France is ___", the model must learn facts.
To complete "She was very hungry, so she ___", the model must learn cause and effect.
To complete "2 + 3 = ___", the model must learn some math.
To complete a line of Python code, the model must learn programming.
To complete a sentence in Hindi, the model must learn Hindi.

So, by playing this one game at a huge scale, the model ends up learning grammar, facts, reasoning patterns, coding, and much more.

Note: The model actually predicts the next token, not the next word. A token is a small piece of text, like a word or a part of a word. We will see tokens in a moment. For now, we can think of a token as a word, just for the sake of understanding.

Why no human labels are needed

In the usual style of Machine Learning, humans give the model the correct answers. For example, to teach a model to spot cats in photos, humans have to label thousands of photos as "cat" or "not cat". This is slow and expensive.

But pretraining does not need any human to label anything. The correct answer is already inside the text itself.

Let's take the sentence: "The cat sat on the mat."

From this one sentence, we can make many practice questions automatically, like below:

Input                        Correct next word
"The"                   ---> "cat"
"The cat"               ---> "sat"
"The cat sat"           ---> "on"
"The cat sat on"        ---> "the"
"The cat sat on the"    ---> "mat"

Here, we can see that every word in the sentence becomes a correct answer for the words before it. Nobody had to write these answers. The text provides them.

We can write the code to create these practice questions as below:

tokens = ["The", "cat", "sat", "on", "the", "mat"]

for i in range(1, len(tokens)):
    context = tokens[:i]
    target = tokens[i]
    print(context, "->", target)

Here, we have a list of tokens. In each round of the loop, context holds all the tokens before position i, and target holds the token at position i, which is the correct next token. It will print the following:

['The'] -> cat
['The', 'cat'] -> sat
['The', 'cat', 'sat'] -> on
['The', 'cat', 'sat', 'on'] -> the
['The', 'cat', 'sat', 'on', 'the'] -> mat

Only five lines of code, and one sentence gave us five practice questions with correct answers.

This way of learning, where the data creates its own answers, is called self-supervised learning.

Means, any text can be used for training without anyone labeling it. That is why we can train on trillions of tokens.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

AI Engineer
Forward Deployed Engineer
AI Agent Engineer
Harness Engineer
Voice AI Engineer
AI Architect
LLM Engineer
LLM Inference Engineer
Gen AI Engineer
MLOps Engineer
ML Engineer
Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Step 1: Collecting the text

Now, let's see the pretraining process step by step.

First, we need a huge amount of text. This text comes from many sources:

Web pages collected from the public internet. A popular public collection is Common Crawl, which has been saving copies of web pages for many years.
Books.
Wikipedia and other encyclopedias.
Code from public code websites.
Research papers.
Text in many languages.

The amount is enormous. Modern LLMs are trained on trillions of tokens. For example, Meta has shared that its Llama 3 models were pretrained on more than 15 trillion tokens.

To understand how big this is, if a person reads one book every day, they will not finish even a tiny part of this text in their whole life.

Step 2: Cleaning the text

But the internet has a lot of low-quality text. If we train the model on low-quality text, it learns low-quality patterns.

So, before training, the text is cleaned. The common cleaning steps are:

Removing duplicates: The same page often appears many times. Repeated text makes the model memorize instead of learn, so we keep only one copy.
Removing low-quality text: Spam, broken pages, random lists of keywords, and pages with very little real content are filtered out.
Removing harmful content: Toxic and unsafe text is filtered out as much as possible.
Removing private information: Things like phone numbers and email addresses are removed or masked.
Balancing the mix: The team decides how much of the final data comes from web pages, books, code, math, and other languages.

Very important: The quality of the data matters as much as the quantity. A smaller amount of clean, high-quality text often teaches the model better than a larger amount of messy text.

Step 3: Breaking text into tokens

The model does not understand letters or words directly. It only understands numbers. So, the text is broken into tokens, and each token gets an ID number.

"Pretraining is powerful"
        ↓
["Pre", "training", " is", " powerful"]
        ↓
[ 4721,   8812,     318,   3665 ]

Here, for the sake of example, the ID numbers are made up. Each model has its own fixed list of tokens called its vocabulary, and every token in the vocabulary has its own ID.

Common words usually become one token. Rare or long words get split into smaller pieces. This way, the model can handle any word, even a word it has never seen. We have a detailed blog on Byte Pair Encoding (BPE) that explains how this splitting works step by step.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

LLM
RAG
MCP
Agent
Fine-tuning
Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Step 4: The training loop

The model is a big neural network, which is made of many layers of simple math steps and learns patterns from examples. It learns through a loop that repeats millions of times.

Step A: We take a chunk of text from our cleaned data, for example, "The cat sat on the".

Step B: The model makes a guess. It does not pick just one word. It gives a probability for every token in its vocabulary. All these probabilities together add up to 1. Let's say, at some point in training, the guesses look like below:

Next token     Probability
"mat"          0.10
"floor"        0.08
"banana"       0.05
...            ...

Step C: We check how wrong the guess was. The correct next word from the real text is "mat". So, we look at how much probability the model gave to "mat". If it gave a high probability, the model did well. If it gave a low probability, the model did badly.

This "how wrong" score is called the loss. A lower loss means a better guess.

Step D: We adjust the weights. The model slightly changes its billions of weights, so that next time it gives a little more probability to "mat" in this kind of sentence. The method that figures out which weights to change, and by how much, is called backpropagation. The method that actually makes the small changes is called gradient descent.

The idea is simple: find the mistake, then nudge every weight a tiny bit in the direction that reduces the mistake.

Step E: We repeat steps A to D with the next piece of text, and the next, and the next, across trillions of tokens.

        Take text
            ↓
    Guess next token
            ↓
      Measure loss
            ↓
     Adjust weights
            ↓
Repeat with the next text

In practice, the model does not learn from one sentence at a time. It learns from a large group of text chunks at once, called a batch. This makes training much faster on powerful chips.

A tiny example of learning from a mistake

The best way to learn this is by taking an example. Let's follow one sentence over a few rounds of training. The numbers are made up just for the sake of understanding.

Sentence: "The cat sat on the ___". Correct answer: "mat".

Round 1:   P("mat") = 0.01   ---> very wrong, high loss  ---> big adjustment
Round 2:   P("mat") = 0.05   ---> still wrong            ---> adjustment
Round 3:   P("mat") = 0.20   ---> getting better         ---> smaller adjustment
...
Round N:   P("mat") = 0.60   ---> good guess, low loss   ---> tiny adjustment

Here, we can notice that the probability for the correct word slowly goes up, and the loss slowly goes down.

But the model is not just memorizing this one sentence. The same weights are shared across all sentences. So, while learning that cats sit on mats, the model also learns that dogs sit on mats, that "on the" is often followed by a place or an object, and many such patterns. Every update helps with millions of other sentences too.

It also never gets to 1.0 for "mat", and that is correct. A cat can also sit on the floor, the sofa, or the bed. A good model learns a sensible spread of probabilities, not one fixed answer.

To learn Loss Functions, Backpropagation, and Gradient Descent in depth, and to build a Large Language Model (LLM) from scratch, check out our AI and Machine Learning Program at Outcome School.

Why does pretraining need so much data and compute?

Now, the next big question is: why does pretraining need such huge resources?

Reason 1: Language is huge. To know facts about the whole world, many languages, coding, and math, the model must see a very large amount of varied text.

Reason 2: The model is huge. A model with billions of weights needs a lot of examples to set all those numbers well.

Reason 3: Every token costs computation. For every token, the model does a huge amount of math, both for guessing and for adjusting the weights.

So, pretraining runs on thousands of GPUs, which are special chips that do this kind of math very fast. These GPUs work together for weeks or months. This costs a huge amount of money and electricity. That is why only a few large companies and labs pretrain big models from scratch.

What do we get after pretraining?

At the end of pretraining, we get a base model.

A base model is very knowledgeable, but it is not yet a good assistant. It is a text completer. It continues whatever text we give it.

Let's say we give a base model this input:

What is the capital of France?

A base model can reply with something like:

What is the capital of Germany? What is the capital of Italy?

Why? Because on the internet, a question like this often appears inside a list of quiz questions. The model is just continuing the text in the most likely way. It is not trying to help us.

So, the base model has the knowledge, but not the behavior of a helpful assistant.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

What comes after pretraining?

To turn the base model into a helpful assistant, we do more training stages. These are much smaller than pretraining.

Instruction tuning: We train the model on examples of instructions and good answers, so that it learns to follow what we ask.
Learning from human preferences: Humans compare the model's answers and pick the better ones. The model learns to give answers that people prefer. A popular method for this is called RLHF (Reinforcement Learning from Human Feedback).

                   Pretraining
     (learns knowledge, trillions of tokens)
                        ↓
                Instruction tuning
(learns to follow instructions, much smaller data)
                        ↓
               Preference training
(learns to be helpful and safe, much smaller data)

Here, we can see that pretraining builds the knowledge, and the later stages shape how the model behaves.

If we want to go deep into Fine-tuning, Parameter-Efficient Fine-Tuning (PEFT), and RLHF, we have a complete program - check out our AI and Machine Learning Program at Outcome School.

Pretraining vs Fine-tuning

Fine-tuning means taking an already trained model and training it a little more on a smaller, focused set of data. Instruction tuning is one type of fine-tuning.

Let me tabulate the differences between pretraining and fine-tuning for your better understanding.

Point	Pretraining	Fine-tuning
Starting point	Random weights	An already trained model
Data	Trillions of tokens of general text	Thousands to millions of focused examples
Labels	None needed, the text is the answer	Usually prepared examples
Goal	Learn language and world knowledge	Learn a behavior or a special skill
Cost	Huge, weeks or months on thousands of GPUs	Much smaller, hours to days
Who does it	A few large labs	Many companies and developers

In real life, almost nobody needs to pretrain a model from scratch. We take an already pretrained model and fine-tune it, or simply use it with good prompts.

Now, we must have understood how pretraining works.

Frequently Asked Questions

Can a base model be used as a chatbot?

Not well. A base model is very knowledgeable, but it is a text completer, not a helpful assistant. It continues whatever text we give it in the most likely way. So, if we ask it a question, it can reply with more questions, because questions like this often appear inside lists of quiz questions on the internet.

How much data is an LLM pretrained on?

Modern LLMs are pretrained on trillions of tokens. For example, Meta has shared that its Llama 3 models were pretrained on more than 15 trillion tokens. This text comes from web pages, books, Wikipedia, code, research papers, and text in many languages.

Does the model just memorize the text during pretraining?

No. The same weights are shared across all sentences, so every update helps with millions of other sentences too. The model learns general patterns, not one fixed answer. This is also why duplicate pages are removed before training, because repeated text makes the model memorize instead of learn.

Is more data always better for pretraining?

Not always. The quality of the data matters as much as the quantity. A smaller amount of clean, high-quality text often teaches the model better than a larger amount of messy text. That is why the text is cleaned before training by removing duplicates, low-quality pages, harmful content, and private information.

Do we need to pretrain our own LLM?

No. Almost nobody needs to pretrain a model from scratch. Pretraining runs on thousands of GPUs for weeks or months, so only a few large companies and labs do it. We take an already pretrained model and fine-tune it, or simply use it with good prompts.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.