How does early stopping work?
- Authors
- Name
- Amit Shekhar
- Published on
Early stopping means we watch the validation loss during training and stop the moment the model starts to overfit, instead of training for all the planned epochs.
In this blog, we will learn about how early stopping works. We will also see what overfitting is, how the validation loss tells us when to stop, what a validation-loss plateau is, why early stopping acts as implicit regularization, how patience works in code, and when to use early stopping.
We will cover the following:
- What is overfitting?
- What is early stopping?
- How does early stopping work?
- What is a validation-loss plateau?
- Early stopping as implicit regularization
- The algorithm with patience
- A simple code example
- When to use early stopping
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning at Outcome School.
Let's get started.
What is overfitting?
During training, the model learns in rounds. Each full pass over the training data is called an epoch. After each epoch, we can measure the loss, which tells us how far the model's answers are from the correct answers. A lower loss means the model is doing better.
Now, here is a problem that appears during training.
Overfitting means the model memorizes the training examples instead of learning the general pattern.
Means, the model becomes very good at the exact examples we showed it, but very bad at new examples it has never seen.
Let's say a student is preparing for an exam. Suppose the student memorizes the answers to last year's question paper word for word, without understanding the subject. In the practice paper, the student scores full marks. But in the real exam, where the questions are new, the student fails.
That student has overfit to the old paper.
Our model can do the same thing. If we train for too long, the model stops learning the real pattern and starts memorizing the training data.
To catch this, we split our data into two parts, as below:
+-----------------------------------------------------+
| All our data |
+-----------------------------+-----------------------+
| Training data | Validation data |
| (model learns from this) | (model is tested on |
| | this, but never |
| | learns from it) |
+-----------------------------+-----------------------+
Here, we can see that we keep two separate parts. The model learns only from the training data. The validation data is kept aside and is never used for learning. We have a detailed blog on why we split data into train, validation, and test sets that explains this in depth.
So, we measure two losses. The training loss tells us how the model does on data it has seen. The validation loss tells us how the model does on data it has never seen.
What is early stopping?
Now that we understand overfitting, it is time to learn about the solution.
Early stopping means we stop training the moment the model starts to overfit, instead of training for all the planned epochs.
Means, we watch the validation loss during training, and we stop at the point where the model is doing its best on new data.
Let's say we planned to train for 100 epochs. But the model reaches its best performance on the validation data at epoch 30. After epoch 30, it starts memorizing the training data. Early stopping tells us to stop at epoch 30 and keep that version of the model.
How does early stopping work?
To understand this, let's look at what happens to both losses as training goes on, as below:
Loss
^
| \
| \ Validation loss
| \ ____/
| \ ____/
| \____ ____/
| \____ ____/
| \/ <- lowest validation loss (stop here)
|
| \_____
| \_______
| \_____________________ Training loss
+-------------------------------------------> Epochs
Here, we can see two lines.
The training loss (the bottom line) keeps going down. The model keeps getting better on the data it has seen.
The validation loss (the top line) goes down at first, reaches a lowest point, and then starts going up again.
That lowest point is the best point. Before it, the model is still learning the real pattern. After it, the model starts memorizing the training data, so it does worse on new data and the validation loss rises.
Early stopping tells us to stop right at that lowest point.
Planning your move into AI?
No matter which tech domain you work in today, with your effort, you can transition to these roles:
- AI Engineer
- Forward Deployed Engineer
- AI Agent Engineer
- Harness Engineer
- Voice AI Engineer
- AI Architect
- LLM Engineer
- LLM Inference Engineer
- Gen AI Engineer
- MLOps Engineer
- ML Engineer
- Deep Learning Engineer
Our AI and Machine Learning Program at Outcome School takes you there.
What is a validation-loss plateau?
Now, there is one more detail we must understand.
The validation loss does not always drop smoothly. Sometimes it goes down a little, then up a little, then down again.
A plateau is when the validation loss stops improving for a while, staying almost flat or slowly getting worse. It means the model is no longer getting better on new data.
Let's look at this as below:
Validation loss
^
| \
| \
| \
| \___
| \___
| \____ ___ ___ <- plateau (no more real improvement)
| \/ \/ \
|
+-----------------------------------> Epochs
|<-- flat -->|
no improvement here
Here, we can see that the loss drops at first. Then it flattens out and moves up and down a little. This flat part is the plateau.
When we see this plateau, it is a signal that more training is unlikely to help on new data, and it often leads to overfitting.
So, early stopping halts on a validation-loss plateau. When the validation loss stops improving, we stop.
Early stopping as implicit regularization
Now, let's learn about a deeper idea.
First, what is regularization?
Regularization means any technique that stops the model from overfitting, so it works better on new data.
There are two kinds. Explicit regularization is when we add a special rule or penalty into the training on purpose, for example a penalty that keeps the numbers inside the model small.
Implicit regularization is when a technique reduces overfitting as a side effect, without us adding any special penalty.
Early stopping is implicit regularization.
Here is why. When we train, the numbers inside the model (called weights) often start small and tend to grow larger as the model fits the data more and more tightly. Very large weights are often a sign of a complex, overfit model.
By stopping early, we stop the weights from growing too large. We keep the model in a simpler state.
Let's see this idea as below:
Weight size
^
| _______ large weights
| _____/ (complex, overfit)
| _____/
| _____/
| _____/ <- we STOP here (early stopping)
| ___/ weights still moderate
| __/ (simpler, general)
| _/
+----------------------------------------> Epochs
Here, we can see that the weights keep growing the longer we train. Early stopping cuts the training off while the weights are still moderate. This keeps the model simple.
So, early stopping does a similar job to a regularization penalty, but we never added any penalty. It happens as a side effect of stopping at the right time. That is why we call it implicit regularization.
A quick note for you
No matter which tech domain you work in, get familiar with these topics:
- LLM
- RAG
- MCP
- Agent
- Fine-tuning
- Quantization
We put it all together in one video:
AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization
No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.
Now, let's get back to the topic.
The algorithm with patience
Now, let's understand how a computer actually decides when to stop.
We do not want to stop the very first time the validation loss goes up, because it may just be a small bump and then improve again. So, we use an idea called patience.
Patience means the number of epochs we are willing to wait for improvement before we finally stop.
We give the model a few more chances. If it still does not improve after those chances, we stop.
Here is how the logic works, as below:
+------------------------------------------------------+
| Keep the best validation loss seen so far |
| Keep a counter, starting at 0 |
+------------------------------------------------------+
|
↓
+-------------------------------+
| After each epoch, check the |
| new validation loss |
+-------------------------------+
|
+-------------+-------------+
| |
Loss improved? Loss did NOT improve?
| |
↓ ↓
+------------------+ +----------------------+
| Save this model | | counter = counter+1 |
| as the best. | +----------------------+
| counter = 0 | |
+------------------+ ↓
+-------------------------+
| counter >= patience? |
+-------------------------+
| |
Yes No
| |
↓ ↓
+------------+ keep training
| STOP |
| Restore |
| best model |
+------------+
Here, we can see the full logic.
After every epoch, we check the validation loss. If it improved, we save that model as the best one so far, and we reset the counter to zero. If it did not improve, we add one to the counter.
When the counter reaches our patience value, we stop training. Then we restore the best model we had saved, not the last one.
So, patience lets us wait through small bumps but still stop once the plateau is real.
A simple code example
Now, let's see this idea in a short piece of code, as below:
best_loss = float("inf") # best validation loss so far, start very high
patience = 5 # wait 5 epochs without improvement
counter = 0 # counts epochs with no improvement
for epoch in range(100):
train_one_epoch() # model learns from training data
val_loss = check_validation() # measure loss on validation data
if val_loss < best_loss:
best_loss = val_loss # we found a new best
save_model() # save this best version
counter = 0 # reset the counter
else:
counter = counter + 1 # no improvement this epoch
if counter >= patience:
print("Early stopping now")
break # stop the training loop
restore_best_model() # load the best saved version
Here, we can see the whole idea. The functions train_one_epoch, check_validation, save_model, and restore_best_model stand for our own training code.
We start best_loss very high, so the first epoch will always beat it. After each epoch, we measure val_loss on the validation data. If it is lower than best_loss, we found a better model, so we save it and reset the counter. If not, we add one to the counter.
When the counter reaches patience, which is 5 here, we break out of the loop. Finally, we call restore_best_model to load the best version we saved, not the last one.
To master Loss Functions, Hyperparameter Tuning, and the full training loop with epochs and batches, check out our AI and Machine Learning Program at Outcome School.
When to use early stopping
Now, let's understand when early stopping helps us.
Early stopping is useful in most training setups where we have a separate validation set. It saves us time, because we do not train longer than needed. It saves us money, because training uses expensive computing power. And it gives us a better model, because we keep the version that does best on new data.
Let me tabulate the difference between training without and with early stopping for your better understanding.
| Point | Without early stopping | With early stopping |
|---|---|---|
| Training length | All planned epochs | Stops at the best point |
| Overfitting risk | Higher | Lower |
| Final model kept | The last epoch's model | The best epoch's model |
| Time and compute | Wasted after the best point | Saved |
| New-data accuracy | Often worse | Often better |
Here, we can see that early stopping trains for less time, lowers overfitting, keeps the best model, and often does better on new data.
Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.
Frequently Asked Questions
Why do we restore the best model instead of keeping the last one?
Because the last model is usually worse on new data. After the best point, the model starts memorizing the training data, so the validation loss rises. That is why we save the model every time the validation loss improves, and when training stops, we load that best saved version, not the one from the last epoch.
Why don't we stop the first time the validation loss goes up?
Because it may just be a small bump. The validation loss does not always drop smoothly, and it can go up a little and then improve again. So, we use patience, which is the number of epochs we are willing to wait for improvement. We stop only when the loss still has not improved after those epochs.
Does early stopping need a validation set?
Yes. Early stopping works by watching the validation loss, which is measured on validation data that the model never learns from. Without that separate data, we cannot tell when the model stops getting better on new data. That is why early stopping is useful in most training setups where we have a separate validation set.
Why does the training loss keep going down while the validation loss goes up?
Because the model starts memorizing the training data. It keeps getting better on the examples it has seen, so the training loss keeps dropping. But it gets worse on new examples it has never seen, so the validation loss rises. This gap is the sign of overfitting, and it is exactly where early stopping tells us to stop.
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
You can connect with me on:
Follow Outcome School on:
