One tree is easy to fool. Many trees, combined well, are among the strongest models for tables of data.
4 minute read
In short
Random forests average many independent trees, which reduces variance.
Boosting adds trees in sequence, each fitted to the remaining errors, which reduces bias.
Boosted trees are the go-to model for tabular data.
In plain words
There are two famous ways to combine trees. A random forest grows hundreds of trees, each on a random sample of the data and a random subset of features, then averages their votes. Each tree is noisy, but their mistakes differ, so the average is steady.
Boosting grows trees one after another. Each new tree is trained on the mistakes of all the trees so far, and its prediction is added on, usually scaled down by a learning rate. Forests reduce wobble; boosting steadily reduces error.
Worked example: boosting by hand
Predict four house prices (in lakhs) from size: sizes 1, 2, 3, 4 and prices 10, 14, 20, 24.
1
Start with the average
Every house is predicted at 17. The mistakes, called residuals, are price minus prediction.
Smaller steps need more trees, but usually generalise better to new data.
prediction: 14.5, 14.5, 19.5, 19.5
MSE = 10.25
Your turn
In boosting, what does each new tree learn to predict?
Where you'll see this
Gradient-boosted trees (XGBoost, LightGBM, CatBoost) win most competitions on tabular data such as transactions, user records and logs, and often beat neural networks there. Random forests are a robust first model that needs little tuning.
A common point of confusion is bagging versus boosting. Bagging (random forests) trains trees independently and averages them to cut variance. Boosting trains them in sequence to cut bias.
Common mistakes
Training too many boosting rounds without early stopping, which slowly overfits.
Using deep trees in boosting. Shallow trees work better because each one only fixes part of the error.
Answer in your own words, out loud or on paper, before you open the model answer.
What's the difference between bagging and boosting?
Both combine many trees, but differently. Bagging, as in random forests, trains many deep trees independently on random samples of the data, often with random feature subsets too, and averages them. Each tree overfits in its own way, so averaging cancels out much of that variance. Boosting trains small trees one after another, where each new tree focuses on the errors the ensemble is still making; in gradient boosting it literally fits the remaining errors. That steadily reduces bias. Boosting is usually more accurate on tabular data but needs careful tuning of learning rate, depth and number of trees, with early stopping. Random forests are more forgiving and harder to break.
More to think about
Why does a random forest use random subsets of features?
When would you choose gradient-boosted trees over a neural network?