Random Forests and Gradient Boosting

A single decision tree is easy to read and easy to break: change a few training rows and the branches rearrange. The cure is an old statistical trick. Ask many independent, slightly different estimators and combine their answers. The errors of each one partly cancel, and what’s left is more reliable than any single member. Models built this way are ensembles, and for tabular data like churn they’re what most practitioners reach for after the baselines and a linear model.

Random forests: many trees, averaged

A random forest grows a few hundred deep trees, each on a different bootstrap sample (the training rows sampled with replacement, so each tree sees a slightly different dataset) and, at every split, only a random subset of the features. That second trick stops every tree from opening with the same dominant feature, which makes the trees more different from each other and their mistakes less correlated. To predict, the forest averages the trees’ probabilities.

Gradient boosting: trees that fix each other

Boosting takes the opposite approach. It builds shallow trees one at a time. The first makes rough predictions; the second is fitted to the errors the first one left; the third to what the first two still get wrong, and so on. Each tree’s contribution is scaled down by the learning_rate, so the ensemble creeps towards a good answer instead of lurching. More trees (max_iter) and a higher learning rate fit the training data more closely, and eventually overfit it.

Both on churn, with the missing column back

So far you’ve left out avg_satisfaction because half its values are blank. HistGradientBoostingClassifier, scikit-learn’s fast boosting implementation, handles missing values natively: at each split it learns which side the blanks should go to. Recent versions of DecisionTreeClassifier and RandomForestClassifier accept NaN too. LogisticRegression and k-NN still don’t; they raise ValueError: Input X contains NaN.

import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share",
            "avg_satisfaction"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])

no_gaps = features[:-1]  # logistic regression can't take the NaNs in avg_satisfaction
models = {
    "logistic (no satisfaction)": (make_pipeline(StandardScaler(), LogisticRegression()), no_gaps),
    "random forest": (RandomForestClassifier(n_estimators=300, min_samples_leaf=5,
                                             random_state=42), features),
    "boosting, defaults": (HistGradientBoostingClassifier(random_state=42), features),
    "boosting, gentle": (HistGradientBoostingClassifier(max_depth=3, learning_rate=0.05,
                                                        max_iter=200, random_state=42), features),
}
for name, (model, cols) in models.items():
    model.fit(X_train[cols], y_train)
    print(f"{name:27} train {model.score(X_train[cols], y_train):.2f}  "
          f"test {model.score(X_test[cols], y_test):.2f}")

The forest reaches 0.76 and gently tuned boosting 0.78, against 0.72 for logistic regression and the rule. Boosting with default settings fits the training set almost perfectly (0.99) and does worse than the gentle version: on 298 rows, the defaults are too eager. Shallower trees and a smaller learning rate keep it in check.

Hold these numbers loosely. On a 100-customer test set, two points is two customers. The next lesson adds a metric for ranking quality, and section 4’s cross-validation will tell you whether the ensembles’ edge survives a different shuffle.

What do the trees rely on?

Forests offer feature_importances_, but those numbers are computed on the training data from how much each feature reduced impurity, and they’re biased towards continuous features with many possible thresholds. Permutation importance asks a sharper question on held-out data: if I shuffle this one column, breaking its link with the target, how much does the score drop?

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share",
            "avg_satisfaction"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])
forest = RandomForestClassifier(n_estimators=300, min_samples_leaf=5, random_state=42).fit(X_train, y_train)

result = permutation_importance(forest, X_test, y_test, n_repeats=10, random_state=42)
table = pd.DataFrame({"impurity (train)": forest.feature_importances_,
                      "permutation (test)": result.importances_mean}, index=features)
print(table.sort_values("permutation (test)", ascending=False).round(3))

Impurity importance ranks tenure_days second. Permutation importance on the test set says it barely matters: the forest used it to fit noise in the training data. Shuffling recency costs about 13 points of accuracy; shuffling satisfaction costs nothing (the small negative values are noise). The honest summary for Cartwheel: recency and order count drive the predictions, and the satisfaction survey, in its current half-empty state, adds nothing.

Accuracy has carried you this far, but it treats a missed churner and a false alarm as equally bad, and it ignores the ranking you built for marketing. Next lesson replaces it with metrics that look at each kind of mistake separately.