How does Cross-Validation work?
- Authors
- Name
- Amit Shekhar
- Published on
Cross-Validation is a way to check how well a Machine Learning model will work on new data by splitting the data into several parts, training and testing the model several times so that every part gets a turn to be the test part, and then taking the average of all the scores. It gives a more reliable score than testing the model only once.
In this blog, we will learn about how Cross-Validation works. We will also see why testing a model on just one split can fool us, how K-Fold Cross-Validation works step by step and how to pick the value of K, the popular types like Stratified, Leave-One-Out, Group, and Time Series Cross-Validation, how we use it to pick the best model, the common mistakes that leak information, and when to use which one.
We will cover the following:
- The problem with one single split
- What is Cross-Validation?
- How does K-Fold Cross-Validation work?
- A step-by-step example with numbers
- Seeing it in code
- Using Cross-Validation to pick the best model
- How to choose the value of K?
- Types of Cross-Validation
- Common mistakes while using Cross-Validation
- Where Cross-Validation works well and where it fails
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning at Outcome School.
Let's get started.
The problem with one single split
To know how good a model really is, we must check it on data that it has never seen during training. The simplest way is to split our data once. Let's say we have 100 emails. We use 80 for training and keep 20 aside for testing.
| <------------- 80 for training -------------> | <-- 20 for testing --> |
We train the model on the 80 and check it on the 20. Suppose it gets 18 out of 20 correct. The score, called accuracy, which means the fraction of examples for which the model gave the correct answer, is 90%.
Looks good. But the following question arises: what if those 20 test emails were very easy by luck? Or very hard by luck?
Let's say we shuffle the data and split again. This time, the score is 80%. We shuffle again, and the score is 95%. Which one is the real score?
This is the problem. With one single split, the score depends a lot on which examples landed in the test part by luck. When the data is small, this luck matters even more.
There is one more problem. The 20 test emails are never used for learning. When we have only 100 emails, throwing away 20 of them from training hurts.
We need a solution that solves both problems. So, here comes Cross-Validation to the rescue.
What is Cross-Validation?
Cross-Validation is a method to measure how well a model works on unseen data by training and testing it many times on different splits of the same data, and then averaging the scores.
Here, Cross means the parts take turns. The part used for testing in one round is used for training in the other rounds. The roles cross over.
Let's take an example from school. Suppose a teacher wants to know how good a student really is. Judging the student on only one test is risky. The student can have a bad day. So, the teacher gives five tests on five different chapters and takes the average. The average is a much fairer picture of the student.
Cross-Validation does exactly this for a model.
How does K-Fold Cross-Validation work?
The most common type of Cross-Validation is K-Fold Cross-Validation. Here, a fold means one part of the data, and K is the number of parts.
It works in these steps:
Step 1: Shuffle the data. Mix the examples randomly, so that each part gets a fair mix.
Step 2: Split the data into K equal folds. For example, with K = 5, we get 5 parts.
Step 3: Run K rounds. In each round, one fold acts as the test part, and the remaining K - 1 folds act as the training part. We train a fresh model on the training part and check it on the test part. We note the score.
Step 4: Rotate. In the next round, a different fold becomes the test part. We repeat until every fold has been the test part exactly once.
Step 5: Average the K scores. This average is our final Cross-Validation score.
Let's see it with K = 5:
Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
Round 1: [ TEST ] [train ] [train ] [train ] [train ] -> Score 1
Round 2: [train ] [ TEST ] [train ] [train ] [train ] -> Score 2
Round 3: [train ] [train ] [ TEST ] [train ] [train ] -> Score 3
Round 4: [train ] [train ] [train ] [ TEST ] [train ] -> Score 4
Round 5: [train ] [train ] [train ] [train ] [ TEST ] -> Score 5
Final score = average of Score 1 to Score 5
Here, we can notice two important things:
- Every example is used for testing exactly once. So, no example is left out of the judging.
- Every example is used for training K - 1 times. So, no example is wasted.
This solves both of our problems. The luck of one split gets averaged out, and all the data gets used.
Note: In each round, we train a brand-new model from scratch. The model from Round 1 is not carried over to Round 2. Otherwise, the model will have already seen the test fold, and the score will be dishonest.
Planning your move into AI?
No matter which tech domain you work in today, with your effort, you can transition to these roles:
- AI Engineer
- Forward Deployed Engineer
- AI Agent Engineer
- Harness Engineer
- Voice AI Engineer
- AI Architect
- LLM Engineer
- LLM Inference Engineer
- Gen AI Engineer
- MLOps Engineer
- ML Engineer
- Deep Learning Engineer
Our AI and Machine Learning Program at Outcome School takes you there.
A step-by-step example with numbers
The best way to learn this is by taking an example. Suppose we have 100 emails and we use K = 5. So, each fold has 20 emails.
| Round | Train on | Test on | Correct out of 20 | Accuracy |
|---|---|---|---|---|
| 1 | Folds 2, 3, 4, 5 (80 emails) | Fold 1 | 18 | 90% |
| 2 | Folds 1, 3, 4, 5 (80 emails) | Fold 2 | 16 | 80% |
| 3 | Folds 1, 2, 4, 5 (80 emails) | Fold 3 | 19 | 95% |
| 4 | Folds 1, 2, 3, 5 (80 emails) | Fold 4 | 17 | 85% |
| 5 | Folds 1, 2, 3, 4 (80 emails) | Fold 5 | 18 | 90% |
Average accuracy = (90 + 80 + 95 + 85 + 90) / 5 = 440 / 5 = 88%
Here, we can see that if we had used only one split, we could have reported anything from 80% to 95%, depending on luck. Cross-Validation tells us that the model is about 88% accurate, and that the score on a single split can land anywhere between 80% and 95%.
This second part is very useful. The spread of the scores tells us how stable the model is. If the five scores were 88%, 87%, 89%, 88%, and 88%, we would trust the model a lot. If they were 60%, 95%, 70%, 99%, and 100%, the average is still 84.8%, but the model is very unstable. We cannot trust it on new data.
For the sake of understanding, these numbers are made up. Now, let's see real numbers in code.
Seeing it in code
Now, let's run 5-Fold Cross-Validation in Python using the scikit-learn library. We will use a built-in dataset of 569 patients, where each patient has 30 measurements of a tumor, and the answer is whether the tumor is harmful or not. Our model is a small Decision Tree, which is a model that predicts by asking a series of yes-or-no questions.
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score, KFold
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
model = DecisionTreeClassifier(max_depth=3, random_state=0)
folds = KFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(model, X, y, cv=folds)
print("Score of each fold:", scores.round(3))
print("Average score:", round(scores.mean(), 3))
print("Spread (std):", round(scores.std(), 3))
Here, we have done the following:
- We have loaded the data.
Xholds the measurements, andyholds the correct answers. - We have created a Decision Tree that asks at most 3 levels of questions (
max_depth=3). KFold(n_splits=5, shuffle=True)shuffles the data and splits it into 5 folds.random_state=0fixes the random choices, so that we get the same result every time.cross_val_scoreruns the 5 rounds, trains a fresh model each time, and returns the 5 accuracy scores.- We print each score, the average, and the standard deviation (std), which is a number that tells us how spread out the scores are. Smaller means more stable.
This will print the following:
Score of each fold: [0.965 0.93 0.912 0.939 0.938]
Average score: 0.937
Spread (std): 0.017
Here, we can see that the scores of the 5 rounds range from 91.2% to 96.5%. If we had used only one split, we could have reported any of these numbers. Cross-Validation tells us that the model is about 93.7% accurate, and the scores are fairly stable, with a small spread of 0.017.
A quick note for you
No matter which tech domain you work in, get familiar with these topics:
- LLM
- RAG
- MCP
- Agent
- Fine-tuning
- Quantization
We put it all together in one video:
AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization
No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.
Now, let's get back to the topic.
Using Cross-Validation to pick the best model
Now, it's time to learn the most common real use of Cross-Validation: comparing models.
Let's say we are not sure how deep our Decision Tree must be. A very shallow tree is too simple. A very deep tree can overfit, which means it remembers the training data and fails on new data. Settings like this, which we choose before training, are called hyperparameters.
We can try a few values and compare their Cross-Validation scores as below:
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score, KFold
from sklearn.tree import DecisionTreeClassifier
X, y = load_breast_cancer(return_X_y=True)
folds = KFold(n_splits=5, shuffle=True, random_state=0)
for depth in [1, 3, 10]:
model = DecisionTreeClassifier(max_depth=depth, random_state=0)
scores = cross_val_score(model, X, y, cv=folds)
print(f"max_depth={depth:2d} average={scores.mean():.3f}")
Here, we run 5-Fold Cross-Validation for three different tree depths, using the same folds for all three, so that the comparison is fair.
This will print the following:
max_depth= 1 average=0.884
max_depth= 3 average=0.937
max_depth=10 average=0.924
Here, we can see that a tree with depth 1 is too simple, and a tree with depth 10 starts to overfit. A depth of 3 gives the best average score. So, we pick max_depth=3.
After picking the best setting, we train one final model on all of the training data with that setting.
Note: scikit-learn also has GridSearchCV, which tries many combinations of settings and runs Cross-Validation for each one automatically.
Very important: When we use Cross-Validation to pick settings, the best Cross-Validation score is a little too optimistic, because we picked the winner by looking at those very scores. So, we must still keep a separate test set aside, which is never touched during Cross-Validation, and check the final model on it only once at the end.
| <------------ used for Cross-Validation ------------> | <- final test set -> |
(train and compare models here, fold by fold) (touch only once)
To master Hyperparameter Tuning, Model Development and Training, and Evaluation and Testing from the ground up, check out our AI and Machine Learning Program at Outcome School.
How to choose the value of K?
Now, the next big question is: how many folds do we use?
- Small K (like 3): Each round trains on less data (only 2 out of 3 parts). So, the model is a bit weaker than it would be with all the data, and the score comes out a little lower than the truth. But it is fast.
- Large K (like 20): Each round trains on almost all the data, so the score is closer to the truth. But we must train 20 models, which is slow.
In practice, K = 5 or K = 10 is the most common choice. It gives a good balance between a reliable score and the time it takes.
Let me tabulate this for your better understanding.
| K | Models trained | Training data per round | Speed |
|---|---|---|---|
| 3 | 3 | 67% | Fast |
| 5 | 5 | 80% | Good balance |
| 10 | 10 | 90% | Good balance, slower |
| N (number of examples) | N | All but one | Very slow |
Types of Cross-Validation
K-Fold is the basic version. But real data comes in different shapes, so we have a few special types.
Stratified K-Fold
Suppose only 10% of our emails are spam. If we split randomly, one fold can get almost no spam by luck, and that round will give a misleading score.
Stratified K-Fold makes sure that every fold keeps the same mix of answers as the full data. If the full data is 10% spam, every fold is about 10% spam. For classification problems, which means problems where the answer is a category like "spam" or "not spam", this is the safer default. In scikit-learn, when we pass a plain number like cv=5 for a classifier, it already uses Stratified K-Fold.
Leave-One-Out Cross-Validation (LOOCV)
This is K-Fold where K is equal to the number of examples. Each round, we keep just one example for testing and train on all the others. If we have 50 examples, we train 50 models.
It uses the data to the maximum, so it is useful when the data is very small. But it is very slow for large data.
Group K-Fold
Suppose we have 10 X-ray photos of each patient. If photos of the same patient land in both the training part and the test part, the model can recognize the patient instead of learning the disease. The score will look better than it really is.
Group K-Fold makes sure that all the examples of one group, like one patient, stay together in the same fold.
Time Series Cross-Validation
When data has a time order, like daily sales, we must never train on the future to predict the past. In real life, we only know the past.
So, we always train on earlier data and test on the data that comes right after it:
Round 1: [train ] [ TEST ]
Round 2: [train ] [train ] [ TEST ]
Round 3: [train ] [train ] [train ] [ TEST ]
Round 4: [train ] [train ] [train ] [train ] [ TEST ]
---------------- time moves forward ---------------->
Here, we can see that the training part grows with each round, and the test part is always in the future. In scikit-learn, this is TimeSeriesSplit.
Let me tabulate the differences between these types for your better understanding so that you can decide which one to use based on your use case.
| Type | How it splits | When to use |
|---|---|---|
| K-Fold | Random, equal folds | General use, numbers to predict |
| Stratified K-Fold | Each fold keeps the same mix of answers | Categories, especially when some are rare |
| Leave-One-Out | One example tests, all others train | Very small data |
| Group K-Fold | One group stays in one fold | Many examples from the same person or source |
| Time Series Split | Train on past, test on future | Data with a time order |
Common mistakes while using Cross-Validation
Most of the time, we do mistakes while using Cross-Validation in the following ways.
Mistake 1: Preparing the data before splitting. Suppose we scale all the data, which means bringing all the numbers into a similar range using the average of the full data, and then run Cross-Validation. Now, the average included the test fold too. A bit of information from the test fold has leaked into training. This is called data leakage.
Solution: Put the preparation steps and the model together in a Pipeline in scikit-learn. Then, in every round, the preparation is learned only from the training folds. We can write the code as below:
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
model = make_pipeline(StandardScaler(), LogisticRegression())
scores = cross_val_score(model, X, y, cv=5)
Here, StandardScaler is the scaling step, and LogisticRegression is a simple model for yes-or-no answers. Since both are joined by make_pipeline, cross_val_score learns the scaling from the training folds only, in every round. The test fold stays fully unseen.
Mistake 2: Shuffling time-based data. Using normal K-Fold on daily sales mixes the future into training. Use Time Series Cross-Validation instead.
Mistake 3: Same person or source in two folds. Use Group K-Fold.
Mistake 4: Reporting the best Cross-Validation score as the final score after tuning. Keep a separate test set for the final check.
Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.
Where Cross-Validation works well and where it fails
Cross-Validation works well when:
- The data is small or medium, and we cannot afford to waste examples.
- We want to compare models or settings fairly.
- We want to know not just the average score, but also how stable the model is.
Cross-Validation is not the best fit when:
- The model is very slow to train. Training a large deep learning model 5 or 10 times can take days or weeks. In such cases, people usually use one fixed validation split.
- The data is huge. With millions of examples, one split already has a very large test part, so luck matters very little.
Now we must have understood how Cross-Validation works.
Frequently Asked Questions
What is the difference between Cross-Validation and a single train-test split?
A single split trains the model once and checks it once on one test part, so the score depends a lot on which examples landed in the test part by luck. Cross-Validation trains and tests the model several times so that every part gets a turn to be the test part, and then averages the scores. It also uses every example for both training and testing.
Do we still need a separate test set when we use Cross-Validation?
Yes. When we use Cross-Validation to pick settings, the best Cross-Validation score is a little too optimistic, because we picked the winner by looking at those very scores. So, we keep a separate test set aside, never touch it during Cross-Validation, and check the final model on it only once at the end.
Why do we train a new model in every round of Cross-Validation?
We train a brand-new model from scratch in each round so that the model has never seen the test fold of that round. If we carried the model from one round to the next, it would have already seen the test fold, and the score would be dishonest.
What does a large spread in Cross-Validation scores mean?
A large spread means the model is unstable. Even if the average looks fine, the scores change a lot from one fold to another, so we cannot trust the model on new data. A small spread, which we can measure with the standard deviation, means the model is stable.
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
You can connect with me on:
Follow Outcome School on:
