Why do we split data into train, validation, and test sets?

We split data into train, validation, and test sets so that the model learns from one part, gets tuned on a second part, and gets judged fairly on a third part it has never seen before.

In this blog, we will learn about why we split data into train, validation, and test sets. We will also see what each of the three sets does, what goes wrong when we test a model on the data it has already seen, why one test set is not enough when we tune a model, how much data goes into each set, the common mistakes that leak information between the sets, and when to use which split for our use case.

We will cover the following:

• The problem of testing on seen data
• What is the training set?
• What is the test set?
• Why do we need a validation set?
• How the three sets work together
• How much data goes into each set?
• Common mistakes: data leakage
• When the simple split is not enough

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

The problem of testing on seen data

Let's say we want to build a model that looks at a house and predicts its price. Each example in our data is one house. The inputs are things like the size of the house, the number of rooms, and the location. The correct answer is the price at which that house was sold.

Here, we can see that each row is one house. The model looks at many such rows and learns the pattern between the inputs and the price.

We have a detailed blog on Supervised vs Unsupervised Learning that covers this kind of learning from correct answers in depth.

Now, the question is: once the model has learned, how do we know whether it learned well? The best way to learn this is by taking an example.

Suppose a teacher is preparing students for an exam. The teacher gives the students 100 practice questions along with the answers. The students study these 100 questions again and again.

Now, on the exam day, the teacher gives the exact same 100 questions.

Every student scores 100 out of 100.

Did the students really learn the subject, or did they just remember the answers? We cannot tell. A student who memorized the answers and a student who truly understood the subject both get the same score.

The same thing happens with a Machine Learning model. If we check the model on the same data that it learned from, it can remember the answers. It will look perfect, but it can fail badly on new data.

This problem of remembering instead of learning is called overfitting.

Overfitting means the model has learned the training examples so closely that it does not work well on new examples.

For example, suppose one house in our data sold at a very low price because the owner was in a hurry to move. That low price was an accident, not a pattern. An overfitted model remembers even this accident, and then wrongly predicts low prices for similar houses in the future.

So, the score on seen data tells us nothing useful. We need some data that the model has never seen. This is why we split our data.

What is the training set?

The training set is the part of the data from which the model learns.

The training set is like the practice book of the model.

The model looks at the examples in the training set, makes predictions, sees how wrong it is, and adjusts itself again and again. This adjusting process is called training.

We have a detailed blog on Gradient Descent that explains how this adjusting works step by step.

The training set is usually the biggest part of our data, because the model needs a lot of examples to learn the pattern well.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

What is the test set?

The test set is the part of the data that we keep locked away and use only once, at the very end, to measure how good the final model is.

The test set is like the final exam with questions the students have never seen.

Going back to our teacher example, the teacher now gives 100 practice questions for studying, but keeps 20 completely new questions for the exam. If a student scores well on these 20 new questions, we can trust that the student really understood the subject.

Similarly, if our model does well on the test set, we can trust that it will do well on real-world data that it has never seen.

Very important: The test set must stay untouched until the very end. If we keep looking at the test set while building the model, it slowly stops being a fair exam.

Now, at this point, we can think that two sets are enough: one to learn and one to test. Let's see why that is not true.

Why do we need a validation set?

While building a model, we have to make many choices. For example:

• How big is the model going to be?
• For how long do we train it?
• How fast does it learn from each mistake?

These choices are not learned by the model itself. We, the builders, set them before training. These choices are called hyperparameters.

Hyperparameters are the settings that we choose for the model, as opposed to the values that the model learns on its own.

Let's say we want to pick the best model size. We try a small model, a medium model, and a large model. We need some data to compare them.

Suppose we use the test set for this comparison. We check all three on the test set and pick the one with the best score.

The issue with this approach is that we have now used the test set to make a decision. We picked the model that happened to do best on those exact test examples. So, the test score is no longer a fair exam. It is a little too good, because we chose the winner by looking at it.

So, here comes the validation set to the rescue.

The validation set is a separate part of the data that we use to compare models and tune hyperparameters, so that the test set stays clean for the final check.

The validation set is like the mock exam. Students take mock exams many times to see where they stand and to adjust how they study. The real final exam happens only once.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

How the three sets work together

Now, let's put everything together. Let's say we have 10,000 houses in our data. We can split it as below:

   All data: 10,000 houses
   +--------------------------------------+-----------+-----------+
   |           Training set               | Validation|   Test    |
   |           7,000 houses               |   1,500   |   1,500   |
   +--------------------------------------+-----------+-----------+
     Model learns from here                Tune here   Final exam
                                                       (used once)

Here, we can see that the biggest part goes to training, and two smaller parts go to validation and test.

The full workflow looks like below:

Step 1: Train a model on the training set.

Step 2: Check the model on the validation set and note the score.

Step 3: Change a hyperparameter, such as the model size, train again on the training set, and check on the validation set again.

Step 4: Repeat Step 3 many times. Pick the settings that give the best validation score.

Step 5: Finally, check the chosen model on the test set exactly once. This number is the honest estimate of how the model will do in the real world.

Let me tabulate the role of each set for your better understanding.

Let's see the code for this split in Python as below:

from sklearn.model_selection import train_test_split

# First, keep 15% aside as the test set
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.15, random_state=42
)

# Then, split the rest into training and validation
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.176, random_state=42
)

Here, we have:

• X is the inputs, like size, rooms, and location. y is the correct answers, like the prices.
• The first call takes out 15% of the data as the test set and keeps it aside.
• The second call splits the remaining 85% into training and validation. Taking 17.6% of 85% gives us about 15% of the whole data for validation, so we end up close to 70%, 15%, and 15%.
• random_state=42 makes the shuffle repeatable, so we get the same split every time we run the code.

To learn Hyperparameter Tuning, Model Development and Training, and Evaluation and Testing in depth, check out our AI and Machine Learning Program at Outcome School.

How much data goes into each set?

The right split depends on how much data we have.

When we have a small or medium amount of data: A common choice is 70% for training, 15% for validation, and 15% for test. Another common choice is 80%, 10%, and 10%.

When we have a huge amount of data: Let's say we have 10 million examples. Even 1% of that is 100,000 examples, which is more than enough to get a reliable score. So, we can use something like 98% for training, 1% for validation, and 1% for test. And, more examples for training generally give us a better model.

The validation and test sets need to be big enough to give a trustworthy score. Everything else can go to training.

Let me tabulate this for your better understanding.

Common mistakes: data leakage

Most of the time, we do mistakes while splitting data, and the most dangerous mistake is called data leakage.

Data leakage means some information from the validation or test set sneaks into the training process, so the scores look better than they really are.

It is like a student who has secretly seen a few of the final exam questions.

Let's see the common ways leakage happens.

Mistake 1: Duplicate examples. If the same house appears twice in our data, one copy can land in training and the other in test. The model has already seen that exact house, so the test score is too good. We must remove duplicates before splitting.

Mistake 2: Preparing data before splitting. Before training, we often bring all the numbers to a similar range, for example by subtracting the average size from every house size. Suppose we calculate this average using the full data. Now the training data carries a small hint about the test data, because the average included the test examples. We must split first, then calculate such values only from the training set.

Mistake 3: Ignoring time. Let's say we are predicting tomorrow's sales. If we shuffle randomly, the model can train on next month's data and get tested on last month's data. That is like seeing the future. In real life, we never have future data. So, for time-based data, we must train on the past and test on the later period.

   Wrong for time data (random shuffle):
   Train: Jan, Mar, May, Jul ...   Test: Feb, Apr, Jun ...

   Right for time data (split by time):
   Train: Jan - Sep            Validation: Oct     Test: Nov - Dec

Here, we can see that in the right approach, the model always learns from the past and gets checked on the future, just like in the real world.

Mistake 4: Same person in two sets. Suppose we have many photos of each patient. If photos of the same patient go into both training and test, the model can recognize the patient instead of the disease. We must keep all the photos of one person inside the same set.

Mistake 5: Peeking at the test set again and again. If we check the test score after every change, we slowly tune the model to the test set. The test set becomes a second validation set. We must look at it only once.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

When the simple split is not enough

Sometimes we have very little data, say only 500 examples. Keeping 15% aside for validation gives us only 75 examples to judge with, and the score can change a lot depending on which examples landed in the validation set by luck.

Here comes cross-validation into the picture. In cross-validation, we split the data that is not in the test set into, say, 5 equal parts. We train 5 times, and each time a different part acts as the validation set. Then we take the average of the 5 scores. This gives a much more stable score.

Even with cross-validation, we still keep a separate test set aside for the final check.

Note: When some answers are rare, for example only 2% of emails are spam, a random split can put almost no spam in the test set. So, we use a stratified split, which means each set keeps the same mix of answers as the full data. In the code, we can pass stratify=y to train_test_split to do this.

Now we must have understood why we split data into train, validation, and test sets.

Frequently Asked Questions

What is overfitting in Machine Learning?

Overfitting means the model has learned the training examples so closely that it does not work well on new examples. It remembers the answers instead of learning the pattern. If we check such a model on the same data it learned from, it looks perfect, but it can fail badly on new data.

What is the difference between the validation set and the test set?

The validation set is used many times to compare models and tune hyperparameters, like a mock exam. The test set is kept locked away and used only once, at the very end, like the real final exam. Keeping them separate means the final test score stays an honest estimate.

Can we check the test set more than once?

No. We should look at the test set only once. If we check the test score after every change, we slowly tune the model to the test set, and it becomes a second validation set. Then the test score is a little too good and no longer a fair exam.

Should we shuffle time-based data before splitting?

No. For time-based data, a random shuffle lets the model train on later data and get tested on earlier data, which is like seeing the future. Instead, we split by time: we train on the past, validate on the next period, and test on the later period, just like in the real world.

What is a stratified split?

A stratified split means each set keeps the same mix of answers as the full data. We use it when some answers are rare, for example when only 2% of emails are spam, because a random split can put almost no spam in the test set. In the code, we pass stratify=y to train_test_split.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.