How does the Bias-Variance Tradeoff work?

The Bias-Variance Tradeoff is the balance between a model that is too simple to learn the real pattern (high bias) and a model that is so sensitive that it learns the random accidents in its training data (high variance). A good model sits in the middle, where the total error on new data is the lowest.

In this blog, we will learn about how the Bias-Variance Tradeoff works. We will also see what error means in Machine Learning, what bias and variance are, how they connect to underfitting and overfitting, why reducing one usually increases the other, how we can spot each problem by comparing training and validation errors, and what we can do to fix each one for our use case.

We will cover the following:

• What is error?
• What is bias?
• What is variance?
• Bias and variance on a dartboard
• Underfitting and overfitting
• Why is it a tradeoff?
• The three parts of the total error
• Seeing it in code
• How to detect high bias and high variance
• How to fix high bias and high variance

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is error?

Before jumping into bias and variance, we must know what error is.

Let's say we want to predict the price of a house from its size. We collect 100 houses. For each house, we know the size and the price at which it was sold. The model looks at these 100 houses and learns how price changes with size. Then, when a new house comes, the model guesses its price.

The examples that the model learns from are called the training data. The new examples that the model has never seen are called new data or unseen data.

Error is the difference between the model's guess and the correct answer.

If the real price of a house is 50 lakh and our model guesses 45 lakh, the error is 5 lakh. Smaller error means a better model.

We do not care much about the error on the training data. We care about the error on new data. A model is useful only if it makes good guesses on houses that it has never seen.

Bias and variance are the two main reasons why a model makes errors on new data. Let's understand each of them.

What is bias?

Bias is the error that comes from wrong or too simple assumptions in the model. A model with high bias misses the real pattern, no matter how much data we give it.

Let's say the real relationship between house size and price is a curve. Small houses are cheap, and the price grows faster and faster as the size grows. But our model can only draw a straight line. It assumes that the pattern is a straight line.

Price
  |                              *
  |                          *
  |                      *  /
  |                  *   /
  |              *    /          <- straight line (model)
  |         *      /
  |    *        /                * = real houses
  | *        /
  |______/________________________ Size

Here, we can see that the straight line cannot bend to follow the curve. It is wrong at the small houses, wrong in the middle, and wrong at the big houses. Even if we give it 1 million houses, a straight line will still be a straight line. It will still miss the curve.

This is high bias. The model is too simple for the problem.

What is variance?

Variance is the error that comes from the model being too sensitive to the exact training data it saw. A model with high variance changes a lot if we train it on a slightly different set of examples.

Here, we must understand the word noise. Real data is never perfect. One house sold cheaper because the owner was in a hurry. Another sold higher because the buyer loved the garden. These random effects are noise. They are not part of the real pattern between size and price.

Now, suppose our model is very flexible. It can draw any wiggly line it wants. So, it draws a line that passes exactly through every single house in the training data.

Price
  |                          .-*-.      *
  |                *-.      /     \    /
  |               /   \    /       *--'
  |        .-*   /     *--'
  |       /   \ /                  <- wiggly line (model)
  |  *---'     *
  |                                 * = real houses
  |________________________________ Size

Here, we can see that the wiggly line hits every training house perfectly. The training error is zero. But the line has also learned the noise. Between the points, it goes up and down for no real reason. When a new house comes, the guess can be far off.

And, if we collect a different set of 100 houses and train again, we will get a completely different wiggly line. That is why we call it variance: the model varies a lot depending on which data it got.

This is high variance. The model is too sensitive for the problem.

Bias and variance on a dartboard

We can see bias and variance together on a dartboard.

Assume that the center of the dartboard is the correct answer. Each time we train the model on a different set of data and make a guess, we throw one dart.

   Low Bias, Low Variance         Low Bias, High Variance
        _________                      _________
       /         \                    /  x      \
      |    xxx    |                  |       x   |
      |    xox    |                  | x   o     |
      |    xxx    |                  |   x     x |
       \_________/                    \___x_____/

   High Bias, Low Variance        High Bias, High Variance
        _________                      _________
       /     xxx \                    /   x  x  \
      |      xxx  |                  |  x        |
      |    o      |                  |     o   x |
      |           |                  |  x        |
       \_________/                    \_________/

   o = center (correct answer)    x = model's guesses

Let's read each board:

• Low Bias, Low Variance: All the darts are close together and close to the center. This is the ideal model.
• Low Bias, High Variance: The darts are spread all over, but on average they are around the center. The model is right on average, but each single guess is shaky.
• High Bias, Low Variance: The darts are close together, but far from the center. The model is consistent, but consistently wrong.
• High Bias, High Variance: The darts are spread out and also away from the center. This is the worst case.

So, bias is about how far the average guess is from the center. Variance is about how spread out the guesses are.

Underfitting and overfitting

Now, let's connect bias and variance to two words that we hear a lot in Machine Learning.

Underfitting happens when the model is too simple to learn the real pattern. It does badly on the training data and also badly on new data. Underfitting comes from high bias.

Overfitting happens when the model learns the training data too closely, including the noise. It does very well on the training data but badly on new data. Overfitting comes from high variance.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

Why is it a tradeoff?

A tradeoff means giving up a bit of one thing to get more of another thing. Here, lowering bias usually costs us higher variance, and lowering variance usually costs us higher bias.

Now, the question is: why can we not just make both bias and variance zero?

The answer is: when we make a model more flexible to reduce bias, it also becomes more sensitive to noise, which increases variance. And when we make a model simpler to reduce variance, it becomes less able to learn the real pattern, which increases bias.

The flexibility of a model is called its complexity. A straight line has low complexity. A very wiggly curve has high complexity.

Let's see how the total error changes as we increase the complexity:

Total Error
  ^
  |  \                                 /
  |   \                               /
  |    \                             /
  |     '.                         .'
  |       '-.                   .-'
  |          '--..._______...--'
  |                   ^
  |  high bias here                 high variance here
  +-------------------------------------------> Model Complexity
     simple          sweet spot          complex
   (underfit)                           (overfit)

And let me tabulate what happens to each part of the error:

Here, we can notice that on the left side, the model underfits, and on the right side, it overfits. Somewhere in the middle, the total error is the lowest. This is the sweet spot.

This is the Bias-Variance Tradeoff. We cannot push one down for free. We trade a bit of one to get less of the other, and we look for the point where the total error on new data is the lowest.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

The three parts of the total error

Now, it's time to see how bias and variance add up. For the sake of understanding, we can break the total error of a model on new data into three parts as below:

Total Error = Bias^2 + Variance + Irreducible Error

Here, we have:

• Bias^2 (bias squared): The error from the model being too simple. We square it because, in the math, errors are measured as squared differences, so that positive and negative mistakes do not cancel each other.
• Variance: The error from the model being too sensitive to the training data.
• Irreducible Error: The error that comes from the noise in the data itself. No model can remove it. If the owner sold the house cheap because he was in a hurry, no model can know that from the size alone.

Note: This exact formula is true when we measure error as the squared difference between the guess and the correct answer, which is the most common way for predicting numbers. For other kinds of problems, the idea stays the same even if the math looks different.

The irreducible error is a floor. We can never go below it. So, our job is to reduce Bias^2 + Variance together, and that is exactly where the tradeoff comes into the picture.

Seeing it in code

Now, let's see the tradeoff with a small example in Python. We will create some data that follows a curve, add some noise, and try three models: a very simple one, a balanced one, and a very complex one.

import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

# Create data that follows a curve, plus some noise
rng = np.random.RandomState(1)
X = np.sort(rng.uniform(0, 1, 30)).reshape(-1, 1)
y = np.sin(2 * np.pi * X).ravel() + rng.normal(0, 0.2, 30)

X_train, X_val, y_train, y_val = train_test_split(
    X, y, test_size=0.3, random_state=0)

for degree in [1, 4, 12]:
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    model.fit(X_train, y_train)
    train_err = mean_squared_error(y_train, model.predict(X_train))
    val_err = mean_squared_error(y_val, model.predict(X_val))
    print(f"degree={degree:2d}  train={train_err:.3f}  val={val_err:.3f}")

Here, we have done the following:

• We have created 30 points that follow a wave shape (a sine curve) and added random noise to each point.
• We have kept 70% of the points for training and 30% as a validation set, which means data that the model does not learn from, so that we can check it on unseen points.
• We have trained three models. The degree controls how much the line can bend. degree=1 is a straight line. degree=4 can bend a few times. degree=12 can bend a lot.
• PolynomialFeatures together with LinearRegression lets the model draw a curve instead of only a straight line, and make_pipeline joins these two steps into one model.
• For each model, we print the training error and the validation error. We measure both with mean_squared_error, which is the average of the squared differences between the guesses and the correct answers. Lower is better.

This will print the following:

degree= 1  train=0.217  val=0.181
degree= 4  train=0.019  val=0.036
degree=12  train=0.010  val=9.616

Here, we can see that:

• The straight line (degree=1) has a high error on both training and validation data. It cannot follow the wave. This is high bias.
• The degree=12 model has the lowest training error of all three, but its validation error explodes to 9.616. It has bent itself to pass through the noisy training points, and it goes wild between and around them. This is high variance.
• The degree=4 model has a low error on both. This is the balance we want.

Note: The exact numbers will change if we change the random data, but the pattern stays the same.

To learn Machine Learning Fundamentals like Linear Regression, Loss Functions, and Evaluation and Testing in depth, check out our AI and Machine Learning Program at Outcome School.

How to detect high bias and high variance

The next big question is: how do we know which problem our model has?

Most of the time, we do mistakes by looking only at the training error. A low training error alone tells us nothing about new data. The answer is to always compare two numbers:

• Training error: the error on the data the model learned from.
• Validation error: the error on data the model has never seen.

Let me tabulate the cases for your better understanding.

So, the gap between the two errors tells us about variance. And the level of the training error tells us about bias. In our code example, degree=1 had a high training error (high bias), and degree=12 had a huge gap (high variance).

A big gap between training error and validation error means high variance. A high training error means high bias.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

How to fix high bias and high variance

Now that we know how to detect the problem, it's time to learn how to fix it.

If the model has high bias (underfitting):

• Use a more complex model. For example, move from a straight line to a curve, or from a small neural network to a bigger one.
• Add more useful inputs, which are called features. For a house, adding the number of rooms and the location along with the size helps the model see more of the picture.
• Reduce regularization. Regularization is a technique that puts a penalty on the model for being too complex. If the penalty is too strong, the model becomes too simple.
• Train for longer, if the model has not finished learning.

Note: Adding more data does not fix high bias. A straight line stays a straight line even with a million houses.

If the model has high variance (overfitting):

• Get more training data. With more examples, the noise of any single example matters less, and the model is forced to learn the real pattern.
• Use a simpler model, or reduce the number of features.
• Increase regularization, so that the model is punished for being too wiggly.
• Use ensembles, which means training many models and averaging their guesses. Averaging cancels out a lot of the random shakiness. This is the idea behind Random Forests.
• In neural networks, use techniques like dropout (randomly switching off some parts during training) and early stopping (stopping the training when the validation error starts going up).

We have a detailed blog on Data Augmentation that explains how to create more varied training examples from the data we already have.

Let me tabulate this for your better understanding.

Here, we can see the tradeoff again. Most fixes reduce one and increase the other. Only a few, like more training data, help one side without hurting the other.

Note: In modern deep learning, very large neural networks trained on huge data sometimes keep improving on new data even after they are big enough to memorize the training data. Researchers call this double descent. But the core idea of the tradeoff still guides us: we always watch the training error and the validation error together.

Now we must have understood how the Bias-Variance Tradeoff works.

Frequently Asked Questions

Does adding more training data fix underfitting?

No. Adding more data does not fix high bias. A model that is too simple, like a straight line, stays a straight line even with a million examples, so it still misses the real pattern. To fix underfitting, we use a more complex model, add more useful features, reduce regularization, or train for longer.

Can a model have both high bias and high variance?

Yes. A model can be too simple and also unstable at the same time. On a dartboard, its guesses are spread out and also away from the center, which is the worst case. We can spot it when the training error is high and the validation error is much higher.

Can we ever reach zero error on new data?

No. Even a perfect balance of bias and variance cannot remove the irreducible error, which comes from the noise in the data itself. For example, if a house sold cheap because the owner was in a hurry, no model can know that from the size alone. This error is a floor that we can never go below.

Why can a training error of zero be a bad sign?

A training error of zero often means the model has memorized the training data, including its noise. Such a model has high variance and can make guesses that are far off on new data. This is why we always compare the training error with the validation error instead of looking at the training error alone.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.