How does Instruction Tuning work?
- Authors
- Name
- Amit Shekhar
- Published on
Instruction tuning is the training step that teaches a pretrained language model to follow instructions. We train it on many examples of an instruction paired with a good answer, so that it learns to respond helpfully instead of just continuing text.
In this blog, we will learn about how instruction tuning works. We will also see what problem a base model has, what an instruction dataset looks like and where it comes from, how the examples are formatted with a chat template, how the actual training happens step by step, why the model learns only from the answer part, and how it fits with the other training stages of an LLM.
We will cover the following:
- What is a base model?
- The problem with a base model
- What is Instruction Tuning?
- What does an instruction dataset look like?
- Where do the examples come from?
- Step 1: Formatting with a chat template
- Step 2: Training on the examples
- Step 3: Learning only from the answer
- A complete walkthrough with one example
- Why does instruction tuning work so well?
- Quality matters more than quantity
- Base model vs Instruction-tuned model
- Where instruction tuning fits in the full pipeline
- Limitations of instruction tuning
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning at Outcome School.
Let's get started.
What is a base model?
Before jumping into instruction tuning, we must know what a base model is.
An LLM (Large Language Model) is built in stages. The first and biggest stage is called pretraining.
During pretraining, the model reads a huge amount of text from the internet, books, and code. It plays one simple game again and again: predict the next token. A token is a small piece of text, like a word or a part of a word.
A base model is the model we get at the end of pretraining. It knows a lot about language and the world, but it is only good at continuing text.
The problem with a base model
Let's say we give a base model this input:
Write a short poem about the rain.
We expect a poem. But a base model can reply with something like:
Write a short poem about the sun.
Write a short poem about the sea.
Write a short poem about the wind.
Why does this happen? Because on the internet, a line like "Write a short poem about the rain." often appears inside a list of writing exercises. The base model just continues the text in the way that looks most likely. It does not understand that we are asking it to do something.
Similarly, if we ask "What is the capital of Japan?", it can reply with "What is the capital of China?". The knowledge is there. The model knows that the capital of Japan is Tokyo. But it does not know that it must answer the question.
So, the problem is: the base model has knowledge, but it does not know how to behave like a helpful assistant.
So, here comes instruction tuning to the rescue.
What is Instruction Tuning?
Instruction tuning is the process of training a base model on many examples of instructions and their ideal responses, so that it learns to follow instructions.
Here, tuning means training an already trained model a little more, to adjust its behavior. This is also called fine-tuning.
Instruction tuning is also commonly called Supervised Fine-Tuning, or SFT in short. "Supervised" means that we show the model the correct answer for every example, just like in supervised learning.
Let's take an analogy. Suppose a graduate joins a company. The graduate already knows a lot from years of study. But on the first day, they do not know how the company wants emails written or how customers must be answered. So, the manager shows them hundreds of examples: "When a customer asks this, we reply like this." After seeing enough examples, the graduate understands the pattern and handles new requests on their own.
The base model is the graduate. Instruction tuning is the on-the-job training.
To learn LLM Fundamentals and Fine-tuning, and to build a Large Language Model (LLM) from scratch, check out our AI and Machine Learning Program at Outcome School.
What does an instruction dataset look like?
The data used for instruction tuning is a collection of examples. Each example has an instruction and the ideal response. Some examples also have an extra input.
We can write an example as below:
{
"instruction": "Summarize the following text in one sentence.",
"input": "The meeting was moved from Monday to Wednesday because the manager was traveling. All team members must update their calendars.",
"output": "The meeting has moved to Wednesday due to the manager's travel, so everyone must update their calendars."
}
Here, we have three parts:
instruction: what the user asks the model to do.input: the extra content to work on. It can be empty.output: the ideal answer that we want the model to learn to give.
A good instruction dataset covers many different kinds of tasks, like below:
Task type Example instruction
Question answer "Why is the sky blue?"
Summarizing "Summarize this article in 3 points."
Translation "Translate this sentence into Hindi."
Writing "Write a polite leave request email."
Coding "Write a Python function to reverse a list."
Classification "Is this review positive or negative?"
Reasoning "If a train leaves at 3 PM and takes 2 hours, when does it arrive?"
Conversation Multi-turn chats between a user and an assistant
The variety is very important. We will see why later in this blog.
Where do the examples come from?
Now, the question is: who creates these examples? There are three common sources.
Source 1: Written by humans
People are hired to write instructions and high-quality answers. This gives the best quality, but it is slow and expensive.
Source 2: Converted from existing datasets
Researchers already have many datasets for tasks like translation, question answering, and sentiment detection. We can convert them into instruction format by wrapping them in templates. For example, a translation pair can become: "Translate the following sentence into French: ...". Google's FLAN work used this idea and converted a large number of existing tasks into instructions.
Source 3: Generated by another LLM
A strong LLM is asked to generate instructions and answers. This is fast and cheap. For example, Stanford's Alpaca project created 52,000 instruction examples using an OpenAI model, and then used them to instruction-tune a Llama model. The risk here is that mistakes from the generating model can get copied into our dataset.
In practice, many teams mix all three sources and then carefully filter the examples for quality.
Planning your move into AI?
No matter which tech domain you work in today, with your effort, you can transition to these roles:
- AI Engineer
- Forward Deployed Engineer
- AI Agent Engineer
- Harness Engineer
- Voice AI Engineer
- AI Architect
- LLM Engineer
- LLM Inference Engineer
- Gen AI Engineer
- MLOps Engineer
- ML Engineer
- Deep Learning Engineer
Our AI and Machine Learning Program at Outcome School takes you there.
Step 1: Formatting with a chat template
Now, let's see how instruction tuning actually works, step by step.
First, every example is turned into one piece of text in a fixed format. This format is called a chat template. It uses special markers to show where the user's message starts and where the assistant's answer starts.
For the sake of understanding, a simple chat template looks like below:
<|user|>
Summarize the following text in one sentence.
The meeting was moved from Monday to Wednesday because the manager was traveling. All team members must update their calendars.
<|assistant|>
The meeting has moved to Wednesday due to the manager's travel, so everyone must update their calendars.<|end|>
Here, we can see that:
<|user|>marks the start of the user's message.<|assistant|>marks the start of the model's answer.<|end|>marks the end of the answer, so that the model learns when to stop talking.
Every model family has its own exact template, but the idea is the same everywhere. These markers are a very important part of the learning. They teach the model the difference between "this is what someone asked" and "this is where I must answer".
Note: We must use the same chat template when we use the model later. If we train with one template and use another, the model gets confused.
Step 2: Training on the examples
Now, the formatted examples are used for training.
Instruction tuning uses the same learning method as pretraining: predict the next token. Only the data is different.
The training loop looks like below:
Formatted example
↓
Model predicts next tokens
↓
Compare with correct answer
↓
Measure the loss
↓
Adjust weights
↓
Repeat with the next example
Here, we have the following steps:
- We give a formatted example to the model.
- The model predicts the next token at each position.
- We compare each prediction with the real token in the ideal answer.
- We measure how wrong the predictions were. This number is called the loss.
- We slightly adjust the model's internal numbers, called weights, to reduce the loss.
- We repeat this for all the examples, usually going through the whole dataset only a few times.
Compared to pretraining, instruction tuning is very small:
- Pretraining uses trillions of tokens. Instruction tuning uses thousands to a few million examples.
- Pretraining takes weeks or months on thousands of chips. Instruction tuning often takes hours to days.
We also make only small changes to the weights during instruction tuning. This is because we want to keep all the knowledge from pretraining and only change the behavior.
A quick note for you
No matter which tech domain you work in, get familiar with these topics:
- LLM
- RAG
- MCP
- Agent
- Fine-tuning
- Quantization
We put it all together in one video:
AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization
No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.
Now, let's get back to the topic.
Step 3: Learning only from the answer
Now, the question is: should the model learn to predict the user's instruction too?
The answer is no. We do not want the model to learn to write questions. We want it to learn to write answers.
So, during instruction tuning, we calculate the loss only on the answer part. The instruction part is still given to the model as input, so that it can read it, but the model is not graded on it. This is called loss masking.
Part Graded?
<|user|> Summarize the following text ... No (read only)
<|assistant|> No (read only)
The meeting has moved ... <|end|> Yes (loss is calculated here)
Let's see a simple code for this idea:
prompt_tokens = tokenizer(prompt_text)["input_ids"]
answer_tokens = tokenizer(answer_text)["input_ids"]
tokens = prompt_tokens + answer_tokens
labels = [-100] * len(prompt_tokens) + answer_tokens
Here, we have used the tokenizer to turn the prompt text and the answer text into token IDs. Then, we have joined the prompt tokens and the answer tokens into one sequence. For the labels, which are the correct answers used for grading, we put -100 in place of every prompt token. In popular model-training libraries like PyTorch and Hugging Face Transformers, a label of -100 means "ignore this position while calculating the loss". So, the model is graded only on the answer tokens.
This way, the model learns that when it sees an instruction, it must produce a good answer.
Note: Some teams train on the whole text, including the instruction. But masking the instruction part is the common choice.
A complete walkthrough with one example
The best way to learn this is by taking an example. Let's follow one example through instruction tuning.
Step 1: We take the example.
Instruction: What is the capital of Japan?
Answer: The capital of Japan is Tokyo.
Step 2: We format it with the chat template.
<|user|> What is the capital of Japan? <|assistant|> The capital of Japan is Tokyo.<|end|>
Step 3: The model reads everything up to <|assistant|>. Now it must predict the first answer token.
Before training, the base model gives a high probability to something like "What", because it wants to continue with more questions. The correct token is "The". So, the loss is high, and the weights get adjusted.
Step 4: The model continues predicting each answer token: "capital", "of", "Japan", "is", "Tokyo", ".", and finally <|end|>. Each token that the model does not predict well adds to the loss, and the weights get adjusted.
Step 5: This repeats over thousands of different examples.
After training, when the model sees <|assistant|>, it has learned that it must start answering, and it must stop at the right time.
Before: What is the capital of Japan? ---> What is the capital of China? ...
After: What is the capital of Japan? ---> The capital of Japan is Tokyo.
Why does instruction tuning work so well?
Now, the next big question is: we train on a limited number of examples, so how does the model handle instructions that it has never seen?
The answer is: the model does not memorize the examples. It learns the general pattern of "someone asks, I help".
Google's FLAN paper in 2021 showed this. They instruction-tuned a model on many groups of tasks, and then tested it on completely new kinds of tasks that it had never seen during tuning. The instruction-tuned model did much better on these unseen tasks than the same model without instruction tuning. They also found that the more kinds of tasks they included, the better the model became at new tasks.
So, instruction tuning does not teach the model new knowledge. The knowledge already came from pretraining. Instruction tuning unlocks that knowledge and teaches the model how to use it to help us.
There is another famous result. In OpenAI's InstructGPT work, people preferred the answers of a 1.3 billion parameter model trained to follow instructions over the answers of the original 175 billion parameter GPT-3. That model was more than 100 times smaller. Note: InstructGPT used instruction tuning followed by one more stage of training on human preferences, which we will see shortly. But it showed how much the right training matters, not just the size.
Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.
Quality matters more than quantity
One more thing to notice. For instruction tuning, a small set of excellent examples can beat a large set of poor examples.
A research paper called LIMA showed this. The authors fine-tuned a large Llama model on only 1,000 carefully chosen, high-quality examples, and it gave strong results.
Their idea is called the Superficial Alignment Hypothesis. It says that almost all the knowledge comes from pretraining, and instruction tuning mainly teaches the style and format of answering. If that is true, we do not need millions of examples to teach style. We need good ones.
So, the lesson is clear:
- Clean, correct, and well-written answers are more important than a huge number of examples.
- A wide variety of tasks is more important than many copies of the same task.
- Wrong answers in the data teach the model wrong behavior.
Base model vs Instruction-tuned model
Let me tabulate the differences between a base model and an instruction-tuned model for your better understanding.
| Point | Base Model | Instruction-Tuned Model |
|---|---|---|
| Training | Pretraining only | Pretraining + instruction tuning |
| Behavior | Continues the text | Follows the instruction |
| Answering questions | Often continues with more questions | Gives a direct answer |
| Knows when to stop | Not reliably | Yes, stops at the end marker |
| Best for | Further training, research, text completion | Chat, assistants, apps |
| Ease of use | Needs careful prompts | Works with plain requests |
Many open model releases come in both versions. The instruction-tuned version usually has "Instruct" or "Chat" in its name. If we want to build an app, we use the instruction-tuned version. If we want to do our own training from a clean start, we use the base version.
Where instruction tuning fits in the full pipeline
Now, let's see the full journey of a modern assistant model.
[ Pretraining ]
Reads trillions of tokens
Result: base model
↓
[ Instruction Tuning ]
Learns to follow instructions
Result: instruct model
↓
[ Preference Training ]
Learns which answers people prefer
Result: helpful, safer assistant
After instruction tuning, many models go through one more stage called preference training. In this stage, the model learns from comparisons, like "answer A is better than answer B". Popular methods for this are RLHF (Reinforcement Learning from Human Feedback) and DPO (Direct Preference Optimization).
Why do we need this extra stage? Because instruction tuning only shows the model one good answer for each example. It does not show the model what a bad answer looks like. Preference training teaches the model to choose the better answer among many possible ones, for example, the more helpful, more honest, or safer one.
If we want to go deep into Fine-tuning, RLHF (Reinforcement Learning from Human Feedback), and Direct Preference Optimization (DPO), we have a complete program - check out our AI and Machine Learning Program at Outcome School to master them end to end.
Limitations of instruction tuning
Instruction tuning has some limits that we must know.
- It does not add much new knowledge: If the base model does not know something, instruction tuning will not reliably teach it. It mainly changes behavior.
- It copies the quality of the data: If the training answers have mistakes, the model learns those mistakes.
- It can teach the model to sound confident when it is wrong: If the examples always give confident answers, the model learns to answer confidently even when it does not know, which can lead to made-up answers.
- It sees only good answers: It does not learn what to avoid. That is why preference training comes after it.
- It can make the model forget: If we train too much or too hard on a narrow set of tasks, the model can lose some of its earlier skills. This is called catastrophic forgetting.
Now, we must have understood how instruction tuning works.
Frequently Asked Questions
Is instruction tuning the same as supervised fine-tuning?
Yes. Instruction tuning is also commonly called Supervised Fine-Tuning, or SFT in short. "Supervised" means that we show the model the correct answer for every example. It is a type of fine-tuning, where we train an already trained model a little more to adjust its behavior.
Does instruction tuning teach the model new knowledge?
No. Instruction tuning does not add much new knowledge. The knowledge already comes from pretraining, and instruction tuning mainly changes behavior. It unlocks that knowledge and teaches the model how to use it to help us. If the base model does not know something, instruction tuning will not reliably teach it.
How many examples do we need for instruction tuning?
Not as many as we might think. Instruction tuning uses thousands to a few million examples, which is very small compared to pretraining. The LIMA paper fine-tuned a large Llama model on only 1,000 carefully chosen, high-quality examples and got strong results. Clean answers and a wide variety of tasks matter more than a huge number of examples.
What happens if we use a different chat template after training?
The model gets confused. We must use the same chat template when we use the model later, because the markers in the template are a very important part of the learning. They teach the model the difference between "this is what someone asked" and "this is where I must answer". Every model family has its own exact template.
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
You can connect with me on:
Follow Outcome School on:
