What are the Chinchilla scaling laws?

Authors
What are the Chinchilla scaling laws?

The Chinchilla scaling laws are rules that tell us how to split a fixed training budget between the size of a language model and the amount of text it is trained on. Their main finding is that model size and training data must grow together, at roughly 20 training tokens for every parameter.

In this blog, we will learn about the Chinchilla scaling laws. We will also see what a scaling law is, what the older belief before Chinchilla was, how DeepMind ran the experiments, the "20 tokens per parameter" rule with a step-by-step calculation, how the smaller Chinchilla model beat much bigger models, and where teams go beyond the Chinchilla rule today.

We will cover the following:

  • The three ingredients: parameters, tokens, and compute
  • What is a scaling law?
  • The problem: we have a fixed budget
  • The belief before Chinchilla
  • What did the Chinchilla paper do?
  • The main finding: grow both equally
  • The 20 tokens per parameter rule
  • Chinchilla vs Gopher: the proof
  • A step-by-step calculation
  • The Chinchilla formula
  • Why were the earlier models undertrained?
  • What changed after Chinchilla?
  • Limitations of the Chinchilla scaling laws

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

The three ingredients: parameters, tokens, and compute

Before jumping into the Chinchilla scaling laws, we must know three terms.

Parameters: Parameters are the numbers inside the model that it learns during training. They store what the model knows. More parameters means a bigger model with more room to store knowledge. We write the number of parameters as N.

Tokens: A token is a small piece of text, like a word or a part of a word. The number of tokens the model reads during training is the size of its training data. We write this as D (for data).

Compute: Compute is the total amount of calculation done during training. It is measured in FLOPs, which means the number of basic math operations, like additions and multiplications. We write compute as C.

Let's say we compare the model to a student. The parameters are the size of the student's brain, the tokens are the number of books the student reads, and the compute is the total hours of study.

There is a simple and widely used estimate that connects all three:

C = 6 x N x D (approximately)

Here, the idea is simple. For every token the model reads, it does work for every parameter, both to make a guess and to learn from the guess. That work comes to roughly 6 operations per parameter per token: about 2 for making the guess, and about 4 for learning from it.

Means, if we make the model bigger, or if we give it more tokens, the compute cost goes up.

We have a detailed blog on Pretraining that explains how an LLM learns from trillions of tokens step by step.

What is a scaling law?

A scaling law is a rule that predicts how good a model will become when we make it bigger, give it more data, or spend more compute on it.

A scaling law works like a forecast. Before spending millions of dollars on a giant training run, we can train many small models cheaply, see the pattern, and predict how a big model will perform.

How do we measure "how good" a model is? We use the loss. The loss is a number that shows how wrong the model's guesses are. A lower loss means a better model.

Researchers found that the loss goes down in a smooth and predictable way as we increase size, data, or compute. This smooth pattern is what we call a scaling law.

The problem: we have a fixed budget

Compute is expensive. A big training run needs thousands of GPUs, which are special chips that do this math very fast, running for weeks. So, every team has a fixed compute budget.

Let's say we have a fixed budget C. From C = 6 x N x D, we can see that we have a choice:

  • Option A: Build a very big model and train it on fewer tokens.
  • Option B: Build a smaller model and train it on many more tokens.
  • Option C: Something in between.

All three options cost the same compute. But which one gives the lowest loss?

Going back to our student analogy: with a fixed number of study hours, is it better to have a giant brain that reads a few books, or a normal brain that reads many books?

This is exactly the question the Chinchilla scaling laws answer.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

The belief before Chinchilla

In 2020, a team at OpenAI, led by Jared Kaplan, published an important paper on scaling laws. Their results suggested that when the compute budget grows, most of the extra budget must go into making the model bigger, and only a smaller part into more data. For example, by their rule, a 10 times bigger budget meant a roughly 5 times bigger model, but only about 2 times more data.

The industry followed this idea. Models became huge very quickly, but the amount of training text did not grow as fast.

Model              Parameters    Training tokens
GPT-3              175 billion   300 billion
Gopher (DeepMind)  280 billion   300 billion

Here, we can see that these giant models read only around 300 billion tokens. That is about 1 to 2 tokens for every parameter.

But the following question arises: was this really the best use of the budget?

What did the Chinchilla paper do?

In 2022, a team at DeepMind, led by Jordan Hoffmann, published the paper "Training Compute-Optimal Large Language Models". This is the paper we now call the Chinchilla paper.

They asked one clear question: for a fixed compute budget, what is the best model size and the best number of training tokens?

To answer this, they did not guess. They trained more than 400 language models, with sizes from 70 million to more than 16 billion parameters, and with training data from 5 billion to 500 billion tokens.

Then, they studied the results in three different ways:

  • Approach 1: Fix the model size, vary the training length, and see which combination reaches the lowest loss for each budget.
  • Approach 2: Fix the compute budget, try many model sizes, and find the size with the lowest loss. The curves look like a valley, and the bottom of the valley is the best size. These curves are called IsoFLOP curves, because every point on one curve uses the same number of FLOPs.
  • Approach 3: Fit one mathematical formula to all the results, and use it to find the best size and data.

All three approaches pointed to the same answer.

The main finding: grow both equally

The main finding of the Chinchilla paper is that model size and training tokens must be scaled up in equal proportion.

For example, if we double our model size, we must also double our training data.

Let's say our compute budget becomes 4 times bigger. What do we do?

Old belief (roughly):   make the model much bigger, add a little more data
Chinchilla:             make the model 2x bigger AND give it 2x more data
                        (2 x 2 = 4x compute)

Here, we split the extra budget equally between model size and training data.

The 20 tokens per parameter rule

From the Chinchilla results, the community took a simple rule of thumb:

A compute-optimal model needs roughly 20 training tokens for every parameter.

Model size (N)        Tokens needed (D = 20 x N)
1 billion             20 billion
10 billion            200 billion
70 billion            1.4 trillion

Here, "compute-optimal" means the best loss we can get for that compute budget.

Let's compare this with the older models. GPT-3 and Gopher had only around 1 to 2 tokens per parameter. By the Chinchilla rule, they were too big for the amount of text they had read. Such a model is called undertrained.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Chinchilla vs Gopher: the proof

A rule is only useful if it works in practice. So, DeepMind tested it.

They took roughly the same compute budget used for their own Gopher model and trained a new model the Chinchilla way. They named this new model Chinchilla.

PointGopherChinchilla
Parameters280 billion70 billion
Training tokens300 billion1.4 trillion
Tokens per parameterAbout 120
Compute usedRoughly the sameRoughly the same
ResultGoodBetter on most tests

Chinchilla was 4 times smaller than Gopher, but it read more than 4 times more text. And it performed better than Gopher on a large range of tasks. It also beat other bigger models of that time, like GPT-3 (175 billion parameters) and Megatron-Turing NLG (530 billion parameters).

On a popular test of knowledge across many subjects, called MMLU, Chinchilla reached an average accuracy of 67.5%, which was more than 7 percentage points better than Gopher.

Also, because Chinchilla is 4 times smaller, it is much cheaper and faster to use after training. Every answer the model gives needs less computation.

A step-by-step calculation

The best way to learn this is by taking an example.

Let's say we have a compute budget of C = 1.2 x 10^22 FLOPs. We want to find the best model size and the best number of tokens.

Step 1: We write down the two rules.

Rule 1:  C = 6 x N x D        (compute cost)
Rule 2:  D = 20 x N           (Chinchilla rule)

Step 2: We put Rule 2 into Rule 1.

C = 6 x N x (20 x N)
C = 120 x N x N

Step 3: We solve for N.

N x N = C / 120
N x N = (1.2 x 10^22) / 120
N x N = 10^20
N     = 10^10   = 10 billion parameters

Step 4: We find D.

D = 20 x N
D = 20 x 10 billion
D = 200 billion tokens

So, for this budget, the Chinchilla rule says: build a 10 billion parameter model and train it on 200 billion tokens.

We can write the code for this calculation as below:

import math

def chinchilla_optimal(compute_flops):
    n = math.sqrt(compute_flops / 120)   # best number of parameters
    d = 20 * n                           # best number of tokens
    return n, d

n, d = chinchilla_optimal(1.2e22)
print(f"Parameters: {n:.2e}, Tokens: {d:.2e}")

Here, we have a function that takes the compute budget. It finds the best number of parameters using the square root from Step 3, and then multiplies it by 20 to get the number of tokens. It will print the following:

Parameters: 1.00e+10, Tokens: 2.00e+11

That is 10 billion parameters and 200 billion tokens. It works perfectly.

Let's also check this with the real Chinchilla model. It had 70 billion parameters and 1.4 trillion tokens. So, C = 6 x 70 billion x 1.4 trillion, which is about 5.9 x 10^23 FLOPs. And 1.4 trillion divided by 70 billion is exactly 20. The numbers match the rule.

To learn Optimization and Scaling Techniques and the Numbers Every AI Engineer Should Know, and to build a Large Language Model (LLM) from scratch, check out our AI and Machine Learning Program at Outcome School.

The Chinchilla formula

In Approach 3, the paper fitted a formula for the loss. It looks like below:

Loss(N, D) = E + A / N^alpha + B / D^beta

It says that the loss is made of three parts:

  • E: The best loss possible even with an infinite model and infinite data. This part comes from the natural randomness of language, which no model can remove. For example, after "I had a cup of", both "tea" and "coffee" are fine answers, so no model can be sure.
  • A / N^alpha: The extra loss because the model is not big enough. It shrinks as N grows.
  • B / D^beta: The extra loss because the model has not read enough data. It shrinks as D grows.

Here, A, B, alpha, and beta are numbers that the researchers found by fitting the formula to their 400 training runs.

Both the second and the third part matter. If we put all our budget into N, the third part stays large. If we put all our budget into D, the second part stays large. The lowest total loss comes from balancing both.

Why were the earlier models undertrained?

Now, the question is: why did the earlier study suggest bigger models?

The Chinchilla paper pointed to the way the earlier experiments were set up. One important difference was the learning rate schedule. The learning rate controls how big each learning step is during training. Usually, it slowly decreases over the training run, and the schedule must match the length of the run.

In the earlier study, the schedule was not matched to each shorter run. So, the models trained on less data looked worse than they really were. This made "more data" look less useful, and "bigger model" look more useful.

When the Chinchilla team matched the schedule properly for every run, the value of more data became clear.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

What changed after Chinchilla?

The Chinchilla paper changed how the whole industry trains models.

  • Teams stopped making giant models trained on too little text.
  • Collecting more and better training data became just as important as making models bigger.
  • Smaller models trained on much more data became popular, because they give good results and are cheaper to use.

But, the Chinchilla rule finds the best model for the training budget only. It does not care about what happens after training.

In the real world, a model is used by millions of people, again and again, for months. This usage is called inference. A smaller model makes every inference cheaper and faster.

So, many teams now train small models on far more than 20 tokens per parameter. For example, Meta trained its Llama 3 8 billion parameter model on more than 15 trillion tokens. That is nearly 1,900 tokens per parameter, far beyond the Chinchilla rule.

Means, this spends more compute on training than Chinchilla recommends, but it gives a small model that is cheaper to run after training. This trade-off makes sense for models that will be used heavily.

Chinchilla-optimal:   best model for a fixed TRAINING budget
Over-trained small:   a bit more training cost, much cheaper USAGE cost

If we want to go deep into SLMs and Model Distillation and LLM Inference Engineering, we have a complete program - check out our AI and Machine Learning Program at Outcome School.

Limitations of the Chinchilla scaling laws

The Chinchilla scaling laws are very useful, but we must know their limits.

  • Training cost only: They ignore the cost of using the model after training, as we have just seen.
  • The "20" is not a law of nature: It came from one set of experiments, one type of model, and one kind of data. Different data, a different way of splitting text into tokens, or a different model architecture can shift the best ratio.
  • Assumes data is unlimited: The rule assumes we can always get more fresh, good-quality text. In reality, high-quality text on the internet is limited.
  • Loss is not everything: A lower loss usually means a better model, but it does not tell us everything about how good the model is at a specific task.
  • The fits were checked again: Later, other researchers re-checked the formula from Approach 3 and found some errors in how it was fitted. After correcting it, their results agreed with Approaches 1 and 2, and the main idea of growing size and data together still held.

Now, we must have understood the Chinchilla scaling laws.

Frequently Asked Questions

Is a bigger language model always better?

No. A bigger model is not always better for the same compute budget. Chinchilla was 4 times smaller than Gopher, but it read more than 4 times more text, and it performed better than Gopher on a large range of tasks. It also beat bigger models like GPT-3 and Megatron-Turing NLG.

How do we estimate the compute needed to train an LLM?

We use the simple estimate C = 6 x N x D, where N is the number of parameters and D is the number of training tokens. For every token, the model does roughly 6 operations per parameter: about 2 for making the guess and about 4 for learning from it. For Chinchilla, this comes to about 5.9 x 10^23 FLOPs.

What is the difference between the Kaplan and Chinchilla scaling laws?

The Kaplan scaling laws from OpenAI in 2020 suggested putting most of a bigger budget into a bigger model, for example a roughly 5 times bigger model but only about 2 times more data for a 10 times bigger budget. The Chinchilla scaling laws from DeepMind in 2022 showed that model size and training data must grow in equal proportion.

Why do teams train small models on far more than 20 tokens per parameter?

Because the Chinchilla rule finds the best model for the training budget only, and ignores the cost of using the model after training. A smaller model makes every inference cheaper and faster. For example, Meta trained its Llama 3 8 billion parameter model on more than 15 trillion tokens, which is nearly 1,900 tokens per parameter.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.