How does Gradient Boosting work?

Authors
How does Gradient Boosting work?

Gradient Boosting is a Machine Learning method that builds many small, weak models one after another, where each new model learns to fix the mistakes left by all the models before it. When we add up all these small fixes, we get one strong and accurate model.

In this blog, we will learn about how Gradient Boosting works. We will also see what boosting, weak learners, and residuals are, how each new tree learns from the mistakes of the previous trees with a step-by-step example, what the learning rate does, where the word "gradient" comes from, how it differs from a Random Forest, and where it works well and where it fails.

We will cover the following:

  • The problem with one model
  • What is boosting?
  • What is a weak learner?
  • What are residuals?
  • How does Gradient Boosting work? A step-by-step example
  • What is the learning rate?
  • Why is it called "Gradient" Boosting?
  • Seeing it in code
  • Overfitting and the important settings
  • Gradient Boosting vs Random Forest
  • Popular libraries and where Gradient Boosting works well

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

The problem with one model

Before jumping into Gradient Boosting, we must understand the problem it solves.

Let's say we want to predict the price of a house. We can train one model, for example one Decision Tree. A Decision Tree is a model that predicts by asking a series of yes-or-no questions, like "Is the size more than 1500 square feet?", and then gives an answer at the end.

If we keep the tree small, it is too simple and misses a lot of the pattern. If we make the tree very big, it starts memorizing the training data, including its random accidents, and then fails on new houses. This memorizing problem is called overfitting.

So, one model alone is either too weak or too fragile. The following question arises: can we combine many simple models to get one strong model?

The answer is yes. One of the ways to do this is boosting.

What is boosting?

Boosting is a method in which we train many simple models one after another, and each new model focuses on the mistakes made by the models before it.

The best way to learn this is by taking an example.

Suppose a golfer wants to put the ball into the hole, which is 100 meters away.

  • Shot 1: The golfer hits the ball. It goes 80 meters. The ball is still 20 meters short.
  • Shot 2: The golfer does not start again from the beginning. He hits from where the ball is, and only tries to cover the remaining 20 meters. The ball goes 15 meters. Now, it is 5 meters short.
  • Shot 3: He tries to cover the remaining 5 meters. The ball goes 4 meters. Now, it is 1 meter short.
  • Shot 4: A small tap, and the ball is in the hole.

Here, we can notice that each shot only fixes what is left from the previous shots. No single shot was perfect, but all the shots added together reached the target.

Boosting works exactly like this. The first model makes a rough guess. The second model learns to fix what the first one got wrong. The third model fixes what is still wrong after the first two. And so on.

What is a weak learner?

In boosting, each single model is intentionally kept simple. Such a model is called a weak learner.

A weak learner is a simple model that is only a little better than random guessing on its own.

In Gradient Boosting, the weak learner is almost always a small Decision Tree. It asks only a few questions, for example only 2 or 3 levels deep. A tree with only one question is called a decision stump.

Why do we keep them weak? Because a small tree cannot memorize the data. Each one learns only a small, safe piece of the pattern. Many small, safe pieces added together give a strong model that does not overfit easily.

What are residuals?

Now, we must learn one important word before we see the full process.

A residual is the difference between the correct answer and the current prediction. It is the mistake that is still left.

Residual = Correct Answer - Current Prediction

Let's say a house was sold for 50 lakh, and our model predicted 60 lakh.

Residual = 50 - 60 = -10

Here, the residual is -10. Means, our prediction is 10 lakh too high, and we must go down by 10 to reach the correct answer.

If the residual is positive, our prediction is too low. If the residual is negative, our prediction is too high. If the residual is 0, our prediction is perfect.

In the golf example, the residual is the remaining distance to the hole after each shot.

How does Gradient Boosting work? A step-by-step example

Now, it's time to see the full process with real numbers. For the sake of understanding, we will use only 3 houses.

HouseSizeCorrect Price (lakh)
ASmall40
BSmall50
CBig90

Step 1: Start with a simple first guess.

We do not use any tree yet. We predict the average price for every house.

Average = (40 + 50 + 90) / 3 = 60

So, our first prediction is 60 for every house.

Step 2: Calculate the residuals.

HouseCorrectPredictionResidual
A4060-20
B5060-10
C9060+30

Here, we can see that we are too high for the small houses and too low for the big house.

Step 3: Train a small tree to predict the residuals.

The new tree does not learn the prices. It learns the residuals, which means the mistakes.

Our small tree asks just one question: "Is the house small?"

            [Is the house small?]
              /             \
            Yes              No
            /                  \
  Average residual of A, B    Residual of C
   = (-20 + -10) / 2           = +30
   = -15

So, Tree 1 says: "For small houses, go down by 15. For big houses, go up by 30."

Step 4: Update the predictions, but only by a small step.

We do not add the full correction. We multiply it by a small number called the learning rate. Let's use a learning rate of 0.5 for this example.

New Prediction = Old Prediction + Learning Rate * Tree's Output

HouseOldTree 1 outputNew Prediction
A60-1560 + 0.5 x -15 = 52.5
B60-1560 + 0.5 x -15 = 52.5
C60+3060 + 0.5 x 30 = 75

Step 5: Calculate the new residuals and repeat.

HouseCorrectPredictionNew Residual
A4052.5-12.5
B5052.5-2.5
C9075+15

Here, we can see that every mistake got smaller. Now, Tree 2 learns these new residuals. It asks the same question and outputs -7.5 for small houses (the average of -12.5 and -2.5) and +15 for the big house.

HouseOldTree 2 outputNew Prediction
A52.5-7.552.5 + 0.5 x -7.5 = 48.75
B52.5-7.552.5 + 0.5 x -7.5 = 48.75
C75+1575 + 0.5 x 15 = 82.5

Let's see how the prediction for House C moves closer to the correct answer of 90:

Start:        60
After Tree 1: 75
After Tree 2: 82.5
After Tree 3: 86.25
After Tree 4: 88.1
...           (keeps getting closer to 90)

This is exactly like the golfer. Each tree covers part of the remaining distance.

Note: Houses A and B have the same size, so our tiny trees cannot tell them apart. They will both settle around 45, the average of 40 and 50. With more features, like the number of rooms or the location, the trees can ask more questions and separate them too.

The final model is the first guess plus all the small corrections added together:

Final Prediction = First Guess + Learning Rate * Tree 1 + Learning Rate * Tree 2 + Learning Rate * Tree 3 + ...

This is how Gradient Boosting works. In real use, we build hundreds or thousands of such small trees.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

  • AI Engineer
  • Forward Deployed Engineer
  • AI Agent Engineer
  • Harness Engineer
  • Voice AI Engineer
  • AI Architect
  • LLM Engineer
  • LLM Inference Engineer
  • Gen AI Engineer
  • MLOps Engineer
  • ML Engineer
  • Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

What is the learning rate?

The learning rate is a small number, usually between 0.01 and 0.3, that controls how much of each tree's correction we actually apply.

Why not apply the full correction at once? Let's go back to the golfer. If the golfer hits very hard every time, he will keep overshooting the hole, going past it, then coming back, and so on. Small, careful shots near the hole work better.

In the same way:

  • A big learning rate learns fast, but each tree can over-correct, and the model can overfit.
  • A small learning rate learns slowly and carefully. It needs more trees, but usually gives a better model on new data.

So, there is a balance between the learning rate and the number of trees. If we make the learning rate smaller, we need more trees.

Why is it called "Gradient" Boosting?

Now, the question is: where does the word "gradient" come from?

We already know Boosting. Now, let's understand Gradient.

First, we need the idea of a loss. A loss is a single number that tells us how bad our predictions are. A common loss for numbers is the average of the squared residuals. Bigger mistakes give a bigger loss.

Our goal is to make the loss as small as possible.

Now, think of the loss as a hill, and we are standing somewhere on it. We want to reach the bottom, where the loss is lowest. The gradient tells us which direction is uphill and how steep it is. So, if we walk in the opposite direction of the gradient, we go downhill. This idea of walking downhill step by step is called gradient descent.

When the loss is the squared error, the downhill direction for each prediction is exactly the residual (up to a constant number that does not change the direction). Means, when each new tree learns the residuals, it is actually learning the downhill direction, and adding it takes one step down the hill.

Let's check this with House C. Its prediction was 60 and the correct price was 90. The residual is +30, which says "move up". And moving up is exactly what reduces the loss for House C. Also, the bigger the mistake, the steeper the hill, and the bigger the residual. So, the residual tells us both the direction and how hard to push.

So:

  • Each tree points in the direction that reduces the loss.
  • The learning rate decides how big a step we take.
  • Adding trees one by one is walking downhill one step at a time.

This is why it is called Gradient Boosting.

The best thing about this view is that it works for any loss, not just squared error. For a yes-or-no problem, like "Will this customer leave or not?", we use a different loss, and each tree learns the downhill direction of that loss. These directions are called pseudo-residuals. The full process stays the same.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

Seeing it in code

Now, let's see Gradient Boosting in Python using the scikit-learn library. We will use a built-in dataset of 442 diabetes patients. Each patient has 10 measurements, like age and blood pressure, and the answer is a number that shows how much the disease progressed after one year.

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_absolute_error

X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42)

model = GradientBoostingRegressor(
    n_estimators=200, learning_rate=0.05, max_depth=2, random_state=42)
model.fit(X_train, y_train)

# Check the error after 1, 10, 50, 100, and 200 trees
for i, pred in enumerate(model.staged_predict(X_test), start=1):
    if i in [1, 10, 50, 100, 200]:
        print(f"trees={i:3d}  test error={mean_absolute_error(y_test, pred):.1f}")

Here, we have done the following:

  • We have loaded the data and kept 30% of the patients aside as a test set, which means patients that the model never sees while learning.
  • We have created a GradientBoostingRegressor with n_estimators=200, which means 200 trees, learning_rate=0.05, and max_depth=2, which means each tree asks at most 2 levels of questions. These are our weak learners.
  • random_state=42 fixes the random choices, so that we get the same result every time.
  • staged_predict gives us the predictions after each tree is added, so that we can watch the model improve.
  • mean_absolute_error is the average size of the mistake on the test patients. Lower is better.

This will print the following:

trees=  1  test error=63.0
trees= 10  test error=53.7
trees= 50  test error=43.2
trees=100  test error=41.9
trees=200  test error=42.1

Here, we can see that the error drops quickly as we add trees, from 63.0 with one tree to 41.9 with 100 trees. But, after 100 trees, the error on new patients starts going slightly up again, to 42.1. The extra trees have started learning the random accidents in the training data. This is overfitting, and it leads us to the next section.

Overfitting and the important settings

Gradient Boosting keeps fixing mistakes on the training data. If we let it run for too long, it starts fixing even the random accidents in the training data, which do not repeat in new data.

These are the main settings that we use to control this:

  • Number of trees (n_estimators): Too few trees means underfitting, which means the model is too simple. Too many trees means overfitting.
  • Learning rate (learning_rate): Smaller is safer, but needs more trees.
  • Tree depth (max_depth): Keep trees small, usually 2 to 8 levels. Bigger trees learn faster but overfit faster.
  • Subsampling (subsample): Train each tree on a random part of the data, like 80%. This adds some randomness and helps the model work better on new data.
  • Early stopping: Keep some data aside, watch the error on it after each tree, and stop adding trees when that error stops improving.

Note: Early stopping is the simplest way to pick the number of trees. It automatically stops near the best point, like the 100 trees in our example.

To master Hyperparameter Tuning along with Evaluation and Testing from the ground up, check out our AI and Machine Learning Program at Outcome School.

Gradient Boosting vs Random Forest

Both Gradient Boosting and Random Forest combine many Decision Trees. But they combine them in very different ways.

A Random Forest builds many big trees at the same time, each on a random sample of the data, and then takes a vote or an average. Each tree works alone, without knowing about the others. This way of combining is called bagging.

Gradient Boosting builds small trees one after another, and each tree learns from the mistakes of the trees before it. The trees work as a team.

Let me tabulate the differences between Gradient Boosting and Random Forest for your better understanding so that you can decide which one to use based on your use case.

PointRandom ForestGradient Boosting
How trees are builtAll at the same time, independentlyOne after another, in sequence
What each tree learnsThe original answersThe mistakes of previous trees
Size of each treeBig, deep treesSmall, shallow trees
How answers are combinedVote or averageSum of small corrections
Main thing it reducesVariance (shakiness)Bias (being too simple)
AccuracyVery goodOften the best on table-like data
Tuning effortWorks well with default settingsNeeds careful tuning
Risk of overfittingLowHigher, if trees are too many
Training speedCan train trees in parallelTrees must wait for each other

So, if we want a strong model quickly with little tuning, a Random Forest is a great start. If we want the highest accuracy on table-like data and can spend time on tuning, Gradient Boosting usually does better.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

In real projects, we mostly use one of these libraries:

  • XGBoost: One of the most popular libraries. It adds extra controls to prevent overfitting and is very fast.
  • LightGBM: Built by Microsoft. It is very fast on large data and uses less memory.
  • CatBoost: Built by Yandex. It handles text-like category columns, such as "City" or "Color", very well without extra work.

All of them follow the same core idea that we learned in this blog: start with a simple guess, then keep adding small trees that fix the remaining mistakes.

Gradient Boosting works very well on table-like data, which means data in rows and columns, like a spreadsheet of customers, sales, or loans. It is used for tasks like fraud detection, predicting which customers will leave, and ranking search results.

But, it is not the best choice for images, audio, or long text. For those, neural networks work much better. It also struggles when the data is very small and noisy, because it keeps chasing the mistakes and ends up learning the random accidents in the data. In such cases, a simpler model or a Random Forest is safer.

Now we must have understood how Gradient Boosting works.

Frequently Asked Questions

Can Gradient Boosting be used for classification?

Yes. Gradient Boosting also works for yes-or-no problems, like "Will this customer leave or not?". We use a different loss for such problems, and each new tree learns the downhill direction of that loss. These directions are called pseudo-residuals. The full process stays the same: start with a simple guess and keep adding small trees.

How many trees should we use in Gradient Boosting?

There is no fixed number. Too few trees means the model is too simple, and too many trees means it overfits. A smaller learning rate needs more trees. The simplest way to pick the number is early stopping: watch the error on data kept aside after each tree, and stop adding trees when that error stops improving.

What is the difference between bagging and boosting?

Bagging trains many models at the same time, each on a random sample of the data, and then takes a vote or an average. Each model works alone. Boosting trains small models one after another, and each new model learns from the mistakes of the models before it. Random Forest uses bagging, and Gradient Boosting uses boosting.

Is Gradient Boosting good for images and text?

No. Gradient Boosting is not the best choice for images, audio, or long text. For those, neural networks work much better. Gradient Boosting shines on table-like data, which means data in rows and columns, like a spreadsheet of customers, sales, or loans, and is used for tasks like fraud detection and ranking search results.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

Follow Outcome School on:

Read all of our high-quality blogs here.