How do Decision Trees and Random Forests work?

Decision Trees and Random Forests are Machine Learning models that make a decision by asking a series of simple yes-or-no questions about the data. A Decision Tree is one such flowchart of questions, and a Random Forest is a team of many slightly different Decision Trees that vote together to give a more reliable answer.

In this blog, we will learn about how Decision Trees and Random Forests work. We will also see what a Decision Tree looks like, how it picks the best question to ask at each step, why a single tree overfits so easily, how a Random Forest fixes that by building many trees on random samples of data, how the trees vote, and when to use which one.

We will cover the following:

• What is a Decision Tree?
• Parts of a Decision Tree
• How does a Decision Tree make a prediction?
• How does a Decision Tree learn which question to ask?
• Measuring mixedness with Gini Impurity
• When does the tree stop growing?
• The problem with a single Decision Tree
• What is a Random Forest?
• How does a Random Forest add randomness?
• How does a Random Forest make a prediction?
• Seeing it in code
• Decision Tree vs Random Forest

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is a Decision Tree?

We all have played the game of 20 questions. One person thinks of an animal, and the other person asks yes-or-no questions to guess it.

• Does it live in water? No.
• Does it have four legs? Yes.
• Does it bark? Yes.
• It is a dog.

Each question cuts down the possible answers, and after a few questions, we reach the answer.

A Decision Tree works exactly like this.

A Decision Tree is a Machine Learning model that makes a prediction by asking a series of yes-or-no questions about the input, one after another, until it reaches an answer.

Why do we call it a tree? Because when we draw these questions, the drawing branches out like a tree turned upside down.

Let's say we want to predict whether a friend will go out to play cricket today. We can draw it like below:

                  [Is it raining?]
                   /           \
                 Yes            No
                 /                \
          [Stay Home]       [Is it a holiday?]
                               /         \
                             Yes          No
                             /              \
                       [Play Cricket]  [Is homework done?]
                                          /         \
                                        Yes          No
                                        /              \
                                 [Play Cricket]    [Stay Home]

Here, we can see that to answer, we start at the top and follow the arrows. The best thing about a Decision Tree is that a human can read it and understand exactly why it gave an answer.

Parts of a Decision Tree

Before jumping into how the tree learns, we must know the names of its parts.

• Root node: The very first question at the top. In our example, "Is it raining?".
• Internal node (decision node): Any question in the middle of the tree. In our example, "Is it a holiday?" and "Is homework done?".
• Branch: The arrow that comes out of a question, one for each answer (Yes or No).
• Leaf node: The end of a path. It holds the final answer, like "Play Cricket" or "Stay Home". A leaf asks no more questions.
• Depth: How many questions we ask from the root to the deepest leaf. Our tree has a depth of 3.

How does a Decision Tree make a prediction?

Now, let's see how the tree gives an answer for a new day.

Suppose today it is not raining, it is not a holiday, and our friend has finished the homework.

• Step 1: At the root, we ask "Is it raining?". The answer is No, so we go right.
• Step 2: We ask "Is it a holiday?". The answer is No, so we go right.
• Step 3: We ask "Is homework done?". The answer is Yes, so we go left.
• Step 4: We reach the leaf "Play Cricket". That is our prediction.

This is how a Decision Tree makes a prediction. It walks from the top to a leaf.

Note: Decision Trees can also predict numbers, like the price of a house. In that case, each leaf holds a number, usually the average price of all the training houses that ended up in that leaf. Such a tree is called a regression tree. A tree that predicts a category, like "Play" or "Stay Home", is called a classification tree. The idea is the same for both.

Planning your move into AI?

No matter which tech domain you work in today, with your effort, you can transition to these roles:

• AI Engineer
• Forward Deployed Engineer
• AI Agent Engineer
• Harness Engineer
• Voice AI Engineer
• AI Architect
• LLM Engineer
• LLM Inference Engineer
• Gen AI Engineer
• MLOps Engineer
• ML Engineer
• Deep Learning Engineer

Our AI and Machine Learning Program at Outcome School takes you there.

How does a Decision Tree learn which question to ask?

Now, the next big question is: how does the model build this tree from data?

The model learns from training data, which means past examples where we already know the correct answer. Let's say we have records of 10 past days:

Here, we have 6 days of "Yes" and 4 days of "No".

Learning from past examples where the correct answer is already known is called supervised learning. We have a detailed blog on Supervised vs Unsupervised Learning that explains this in depth.

The tree must decide which question to put at the root. It has two choices: "Is it raining?" or "Is it a holiday?".

A good question is one that splits the data into groups that are as pure as possible. A pure group means all the examples inside it have the same answer. If a group has only "Yes" days, we are fully sure about the answer for that group.

So, the tree tries every possible question, checks how pure the resulting groups are, and picks the question that gives the purest groups. To compare questions, we need a number that measures how mixed a group is. Here comes Gini Impurity into the picture.

Measuring mixedness with Gini Impurity

Gini Impurity is a number that tells us how mixed a group is. It is 0 when the group is fully pure, and it grows as the group gets more mixed.

For a group with two possible answers, the formula is as below:

Gini = 1 - (share of Yes)^2 - (share of No)^2

Let's understand with simple cases:

• A group of 4 "Yes" and 0 "No": Gini = 1 - (1)^2 - (0)^2 = 0. Fully pure.
• A group of 2 "Yes" and 2 "No": Gini = 1 - (0.5)^2 - (0.5)^2 = 0.5. As mixed as it gets for two answers.

Now, let's calculate for our 10 days.

Before any split: 6 Yes and 4 No.

Gini = 1 - (6/10)^2 - (4/10)^2 = 1 - 0.36 - 0.16 = 0.48

Option 1: Split on "Is it raining?"

• Raining = Yes: Days 3, 5, 7, 10. That is 1 Yes and 3 No.
• Raining = No: Days 1, 2, 4, 6, 8, 9. That is 5 Yes and 1 No.

Gini (Raining = Yes) = 1 - (1/4)^2 - (3/4)^2 = 1 - 0.0625 - 0.5625 = 0.375
Gini (Raining = No)  = 1 - (5/6)^2 - (1/6)^2 = 1 - 0.694 - 0.028 = 0.278

To get one number for the whole split, we take the weighted average, which means a bigger group counts more:

Weighted Gini = (4/10) x 0.375 + (6/10) x 0.278 = 0.15 + 0.167 = 0.317

Option 2: Split on "Is it a holiday?"

• Holiday = Yes: Days 1, 4, 5, 9, 10. That is 4 Yes and 1 No.
• Holiday = No: Days 2, 3, 6, 7, 8. That is 2 Yes and 3 No.

Gini (Holiday = Yes) = 1 - (4/5)^2 - (1/5)^2 = 1 - 0.64 - 0.04 = 0.32
Gini (Holiday = No)  = 1 - (2/5)^2 - (3/5)^2 = 1 - 0.16 - 0.36 = 0.48
Weighted Gini        = (5/10) x 0.32 + (5/10) x 0.48 = 0.16 + 0.24 = 0.40

Let me tabulate the results for your better understanding.

Here, we can see that "Is it raining?" makes the groups purer. So, the tree picks "Is it raining?" as the root question.

Then, the tree repeats the exact same process inside each group. For example, inside the "Raining = No" group (5 Yes, 1 No), the only question left is "Is it a holiday?":

                    [Is it raining?]
                     /            \
                   Yes             No
                   /                 \
      Days 3,5,7,10           [Is it a holiday?]
      (1 Yes, 3 No)             /           \
                              Yes            No
                              /                \
                      Days 1,4,9          Days 2,6,8
                      (3 Yes, 0 No)       (2 Yes, 1 No)
                      Pure. Play.         Mostly Play.

Here, we can see that the "Holiday = Yes" group became fully pure, so it becomes a leaf with the answer "Play". This keeps going, group by group, and the tree grows downward. This step-by-step splitting is called recursive splitting.

Note: For number inputs, like temperature, the tree tries questions like "Is temperature less than 30?" for many different cut points and picks the best cut. Also, some trees use a different mixedness measure called Entropy instead of Gini. The idea stays the same: pick the question that makes the groups purest.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

When does the tree stop growing?

If we let the tree keep splitting, it stops only when every leaf is fully pure, or when there is nothing left to split. We can also stop it earlier by setting some rules:

• Maximum depth: Do not ask more than, say, 5 questions in a row.
• Minimum samples in a leaf: Every leaf must have at least, say, 10 examples.
• Minimum improvement: Do not split if the purity improves by only a tiny amount.

These rules are very important. Let's see why.

The problem with a single Decision Tree

Suppose we do not stop the tree. It keeps asking more and more specific questions until every single training example has its own perfect leaf.

[Is it raining?]
   -> No -> [Is it a holiday?]
      -> No -> [Is the temperature above 31.5?]
         -> Yes -> [Is it the 3rd Tuesday of the month?]
            -> Yes -> [Did the neighbor's dog bark?]
               -> ... (Day 8 only)

Here, we can notice that the tree is no longer learning the real pattern. It is memorizing the accidental details of each day. This is called overfitting. Such a tree scores perfectly on the training data, but makes bad predictions on new days.

There is one more problem. A deep Decision Tree is very sensitive to small changes in the data. If we change or remove just a few training examples, a different question can win at the root, and then the whole tree below it changes. This shakiness is called high variance.

What is the solution to this problem? Answer: Ask many people and take a vote. So, here comes the Random Forest to the rescue.

What is a Random Forest?

A Random Forest is a Machine Learning model that builds many Decision Trees, each one trained in a slightly different random way, and combines their answers by voting or averaging.

Here, Random means that each tree is built with some randomness, so that the trees are different from each other.

Let's say we want to guess the number of sweets in a big jar. If we ask one person, the guess can be far off. But if we ask 100 people and take the average, the average is usually very close to the real number. This is called the wisdom of the crowd. The personal mistakes of different people go in different directions and cancel each other out.

But, this works only if the people think differently. If all 100 people copy the same person, we get the same wrong guess 100 times. In the same way, if all our trees are exactly the same, voting does not help at all.

So, the key idea of a Random Forest is to make the trees different from each other. Let's see how.

How does a Random Forest add randomness?

A Random Forest uses two kinds of randomness.

Randomness 1: Each tree gets a random sample of the data (Bootstrap Sampling).

Suppose we have 10 training days. For each tree, we pick 10 days at random, but with replacement. With replacement means that after picking a day, we put it back, so the same day can be picked again.

Original data:  [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Tree 1 gets:    [2, 2, 5, 7, 1, 9, 9, 3, 10, 6]
Tree 2 gets:    [4, 8, 1, 1, 3, 6, 10, 10, 2, 7]
Tree 3 gets:    [5, 9, 9, 2, 8, 4, 4, 6, 1, 3]

Here, we can see that each tree sees a slightly different version of the data. Some days appear twice, and some days are missing. When the data is large, each tree sees only about 63% of the different examples on average, and the rest are left out for that tree.

Training many models on such random samples and then combining them is called Bagging, which is short for Bootstrap Aggregating. Bootstrap means the random sampling with replacement, and Aggregating means combining the results.

Randomness 2: Each split looks at only a random few of the questions (Feature Randomness).

Here, a feature means one input column, like "Raining", "Holiday", or "Temperature".

When a normal tree picks a question, it checks all the features. In a Random Forest, at every split, the tree can look at only a random few features and must pick the best question among those.

Why do we need this? Suppose one feature is very strong, like "Raining". Without this rule, every tree will put "Raining" at the root, and all the trees will look alike. By hiding some features at each split, we force the trees to explore other questions too. This makes the trees more different from each other, and the vote becomes more useful.

Note: A common default for classification is to look at about the square root of the total number of features at each split. If we have 25 features, each split looks at about 5 of them.

How does a Random Forest make a prediction?

Now, our forest of trees is ready. When a new day comes, we send it to every tree, and each tree gives its own answer.

New day: Not raining, not a holiday, homework done

  Tree 1  ->  Play
  Tree 2  ->  Play
  Tree 3  ->  Stay Home
  Tree 4  ->  Play
  Tree 5  ->  Play
           ---------------
  Vote:   Play = 4, Stay Home = 1

  Final answer: Play

Here, we can see that:

• For a category (classification), the forest takes a majority vote. The answer with the most votes wins.
• For a number (regression), the forest takes the average of all the trees' numbers.

Even if a few trees make mistakes, the majority is usually right. This is how a Random Forest reduces the shakiness (variance) of a single tree, without making each tree simpler.

Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.

Seeing it in code

Now, let's see both models in Python using the scikit-learn library. We will use a built-in dataset of 569 patients, where each patient has 30 measurements of a tumor, and the answer is whether the tumor is harmful or not.

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42)

tree = DecisionTreeClassifier(random_state=42)
tree.fit(X_train, y_train)

forest = RandomForestClassifier(n_estimators=100, random_state=42)
forest.fit(X_train, y_train)

print("Tree   train:", round(tree.score(X_train, y_train), 3),
      "test:", round(tree.score(X_test, y_test), 3))
print("Forest train:", round(forest.score(X_train, y_train), 3),
      "test:", round(forest.score(X_test, y_test), 3))

Here, we have done the following:

• We have loaded the data. X holds the 30 measurements for each patient, and y holds the correct answers.
• We have kept 70% of the patients for training and 30% as a test set, which means patients that the models never see while learning.
• We have trained one DecisionTreeClassifier with no depth limit.
• We have trained one RandomForestClassifier with n_estimators=100, which means 100 trees.
• random_state=42 fixes the random choices, so that we get the same result every time we run the code.
• score gives the accuracy, which means the fraction of patients for which the model gave the correct answer.

This will print the following:

Tree   train: 1.0 test: 0.942
Forest train: 1.0 test: 0.971

Here, we can see that both models score a perfect 1.0 on the training data. But on new patients, the single tree gets 94.2% right, while the forest gets 97.1% right. The forest makes about half as many mistakes on new data.

Note: The exact numbers will change with a different data split, but the forest usually does better than a single deep tree on new data.

To learn Machine Learning Fundamentals like Supervised and Unsupervised Learning, Hyperparameter Tuning, and Evaluation and Testing from the ground up, check out our AI and Machine Learning Program at Outcome School.

Decision Tree vs Random Forest

Let me tabulate the differences between a Decision Tree and a Random Forest for your better understanding so that you can decide which one to use based on your use case.

Here is when to use each one:

• Use a Decision Tree when we must explain every decision clearly, for example, to a bank customer whose loan was rejected, or when we need a quick and simple first model.
• Use a Random Forest when we care most about accuracy on new data and do not need to explain each decision step by step. It is a strong default choice for table-like data, such as spreadsheets of customers, sales, or patients.

Now we must have understood how Decision Trees and Random Forests work.

Frequently Asked Questions

Can a Decision Tree predict numbers instead of categories?

Yes. A Decision Tree can predict numbers, like the price of a house. In that case, each leaf holds a number, usually the average of all the training examples that ended up in that leaf. Such a tree is called a regression tree. A Random Forest of such trees takes the average of all the trees' numbers.

What is the difference between Gini Impurity and Entropy?

Both are ways to measure how mixed a group is. Gini Impurity is 0 when a group is fully pure, and it grows as the group gets more mixed. Some trees use Entropy instead of Gini. The idea stays the same with both: the tree picks the question that makes the groups purest.

How do we stop a Decision Tree from overfitting?

We stop the tree from growing too deep. We can set a maximum depth, a minimum number of examples in each leaf, or a minimum improvement in purity before a split is allowed. Another way is to use a Random Forest, which overfits much less because many different trees vote together.

How many features does a Random Forest check at each split?

A Random Forest checks only a random few features at each split, not all of them. A common default for classification is about the square root of the total number of features. So, if we have 25 features, each split looks at about 5 of them. This keeps the trees different from each other.

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.