A decision tree is a game of twenty questions that learns which question to ask first.
3 minute read
In short
A decision tree predicts with a chain of yes or no questions.
Each split is chosen to reduce impurity the most, often measured with Gini impurity.
Deep trees overfit, so depth is limited or many trees are combined.
In plain words
A decision tree predicts by asking yes or no questions about the input: 'Is income above ₹50,000 a month?', then another, and another, until it reaches an answer. Each question splits the data into two groups.
The tree is built by choosing, at each step, the question that best separates the classes. To measure 'best', trees use Gini impurity: the chance that two randomly picked examples from a group have different labels. A pure group has impurity 0; a 50/50 mix has 0.5.
Worked example: who repays a loan?
There are 10 past applicants: 5 repaid and 5 didn't. The starting impurity is 1 − 0.5² − 0.5² = 0.5. Two questions are candidates.
A drops impurity from 0.5 to 0.32. B doesn't help at all, so A becomes the first question.
A: 0.50 → 0.32 B: 0.50 → 0.50
The tree repeats this on each side, asking new questions until groups are pure or a limit is reached.
Your turn
A group has 5 applicants who repaid and none who didn't. What is its Gini impurity?
Where you'll see this
Trees are easy to explain and handle mixed data like ages, cities and yes/no fields without preparation. Their weakness is overfitting: a deep enough tree can memorise every example, including the noise. Limiting depth helps, and combining many trees, the next lesson, helps much more.
Common mistakes
Letting a tree grow until every leaf is pure. It memorises the training data and generalises badly.
Assuming trees need scaled features. They don't; splits only care about order.
Reading feature importance from a single deep tree as reliable. It changes a lot with small changes to the data.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
How does a decision tree choose its splits?
A decision tree is built greedily. At each node, it tries every feature and every possible threshold, and picks the split that makes the child groups as pure as possible. Purity is measured with Gini impurity or entropy for classification, and variance for regression; the best split is the one with the biggest drop in impurity, weighted by how many examples go each way. It then repeats on each child until a stopping rule kicks in, like maximum depth or a minimum number of examples per leaf. Trees are easy to explain and handle mixed feature types, but a single tree overfits easily, which is why we usually use ensembles like random forests or boosting.
More to think about
What is Gini impurity?
Why do decision trees overfit, and how do you prevent it?