Random Forests: Ensemble Averaging
Random forests build many decision trees and average their predictions. The key insight: if individual trees have high variance but low bias, averaging many uncorrelated trees dramatically reduces variance while preserving the low bias.
Two mechanisms ensure the trees are sufficiently uncorrelated:
Bootstrap sampling (bagging). Each tree is trained on a different random sample of the training data, drawn with replacement. About 37% of the original data is excluded from each tree's training set. This means each tree sees a somewhat different dataset.
Feature subsampling. At each split, the tree is only allowed to consider a random subset of features (typically the square root of the total number of features for classification, or one third for regression). This forces trees to use different features, further decorrelating them.
The result: 100 or 500 trees, each somewhat wrong in a different way. Their errors cancel when averaged. The ensemble prediction is far more stable than any individual tree.
Random forests also provide a bonus: out-of-bag (OOB) error estimation. Because each tree was not trained on approximately 37% of the data, you can use those excluded examples to estimate generalization error without a separate validation set.
Feature importance falls out naturally: features that appear in more splits, higher in the trees, and reduce impurity more are more important. This is an interpretability advantage over neural networks.
XGBoost and Gradient Boosting
Random forests build trees in parallel (each tree is independent). Gradient boosting builds trees sequentially. Each tree is trained specifically to correct the errors of the ensemble so far.
The first tree predicts the target directly. The second tree predicts the residuals (errors) of the first tree. The third tree predicts the residuals of the first two trees combined. And so on. The final prediction is the sum of all trees' outputs.
"Gradient" in gradient boosting refers to using the gradient of the loss function to determine what each tree should predict. This is conceptually similar to gradient descent, but instead of updating continuous weights, you are adding new trees.
XGBoost (eXtreme Gradient Boosting) made gradient boosting practical at scale by adding: regularization terms (L1 and L2) directly in the objective function to control tree complexity, an efficient split-finding algorithm that handles missing values natively, column and row subsampling similar to random forests, and hardware-aware parallelism for fast training.
LightGBM from Microsoft and CatBoost from Yandex are alternative implementations with additional innovations (histogram-based splitting, native categorical feature handling). In practice, XGBoost, LightGBM, and CatBoost are all competitive and the differences matter less than hyperparameter tuning.
When Tree Methods Beat Neural Networks
Neural networks dominate images, audio, and text. For tabular data, the picture is different.
Small to medium datasets. Neural networks require large amounts of data to realize their capacity advantage. On datasets with thousands to hundreds of thousands of rows, tree methods with proper hyperparameter tuning typically match or beat neural networks with far less effort.
Heterogeneous features. Tabular data often mixes numeric features on very different scales (age in years, income in dollars, number of clicks) with categorical features (country, product category, user tier). Tree methods handle this naturally. Neural networks require careful preprocessing.
Missing values. XGBoost handles missing values natively by learning which branch to take when a feature is missing. Neural networks require explicit imputation strategies.
Interpretability requirements. A single shallow decision tree is fully explainable to a non-technical stakeholder. Even random forests provide meaningful feature importance scores. When a business or regulator needs to understand why a decision was made, tree methods are far easier to explain.
Training speed and compute constraints. Training an XGBoost model on a million-row dataset takes minutes on a laptop. Training a competitive neural network on the same data could take hours on a GPU.
When Neural Networks Win
Neural networks win when: you have very large datasets (tens of millions of rows or more), the relationships between features are highly non-linear and require deep composition, you can invest in feature engineering specifically designed for the architecture, or the data is not truly tabular (e.g., text encoded as counts, image features flattened, time series).
For a concrete benchmark: on most Kaggle tabular competitions with fewer than 10 million rows, XGBoost or LightGBM with careful tuning will be in the top solutions. Neural networks occasionally win when ensemble with tree methods, or when the data has strong sequential or spatial structure that tree methods cannot capture.
Practical Starting Point
For any new tabular ML problem:
- Start with a random forest with default hyperparameters. This gives you a strong baseline in 10 minutes.
- Switch to XGBoost or LightGBM and tune learning rate, max depth, number of trees, and regularization.
- Only consider neural networks if the tree models plateau well below your performance target or if the data has structure (sequences, spatial patterns) that trees cannot exploit.
Tree methods are not inferior to neural networks. They are the right tool for a different domain, and knowing when to use each is a fundamental ML engineering skill.
Keep Reading
- Feature Engineering Practical Guide -- tree methods benefit enormously from good features, and feature importance from trees guides where to invest
- ML Model Evaluation Metrics Guide -- how to correctly measure whether your tree model is actually better
- Machine Learning Complete Guide for Software Developers -- where tree methods fit in the full ML landscape
Pristren builds AI-powered software for teams. Zlyqor is our all-in-one workspace -- chat, projects, time tracking, AI meeting summaries, and invoicing -- in one tool. Try it free.
