A single decision tree is easy to read and easy to break: change a few training rows and the branches rearrange. The cure is an old statistical trick. Ask many independent, slightly different estimators and combine their answers. The errors of each one partly cancel, and what’s left is more reliable than any single member. Models built this way are ensembles, and for tabular data like churn they’re what most practitioners reach for after the baselines and a linear model.
\nA random forest grows a few hundred deep trees, each on a different bootstrap sample (the training rows sampled with replacement, so each tree sees a slightly different dataset) and, at every split, only a random subset of the features. That second trick stops every tree from opening with the same dominant feature, which makes the trees more different from each other and their mistakes less correlated. To predict, the forest averages the trees’ probabilities.
\nBoosting takes the opposite approach. It builds shallow trees one at a time. The first makes rough predictions; the second is fitted to the errors the first one left; the third to what the first two still get wrong, and so on. Each tree’s contribution is scaled down by the learning_rate, so the ensemble creeps towards a good answer instead of lurching. More trees (max_iter) and a higher learning rate fit the training data more closely, and eventually overfit it.
So far you’ve left out avg_satisfaction because half its values are blank. HistGradientBoostingClassifier, scikit-learn’s fast boosting implementation, handles missing values natively: at each split it learns which side the blanks should go to. Recent versions of DecisionTreeClassifier and RandomForestClassifier accept NaN too. LogisticRegression and k-NN still don’t; they raise ValueError: Input X contains NaN.
import pandas as pd\nfrom sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\",\n \"avg_satisfaction\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\n\nno_gaps = features[:-1] # logistic regression can't take the NaNs in avg_satisfaction\nmodels = {\n \"logistic (no satisfaction)\": (make_pipeline(StandardScaler(), LogisticRegression()), no_gaps),\n \"random forest\": (RandomForestClassifier(n_estimators=300, min_samples_leaf=5,\n random_state=42), features),\n \"boosting, defaults\": (HistGradientBoostingClassifier(random_state=42), features),\n \"boosting, gentle\": (HistGradientBoostingClassifier(max_depth=3, learning_rate=0.05,\n max_iter=200, random_state=42), features),\n}\nfor name, (model, cols) in models.items():\n model.fit(X_train[cols], y_train)\n print(f\"{name:27} train {model.score(X_train[cols], y_train):.2f} \"\n f\"test {model.score(X_test[cols], y_test):.2f}\")The forest reaches 0.76 and gently tuned boosting 0.78, against 0.72 for logistic regression and the rule. Boosting with default settings fits the training set almost perfectly (0.99) and does worse than the gentle version: on 298 rows, the defaults are too eager. Shallower trees and a smaller learning rate keep it in check.
\nHold these numbers loosely. On a 100-customer test set, two points is two customers. The next lesson adds a metric for ranking quality, and section 4’s cross-validation will tell you whether the ensembles’ edge survives a different shuffle.
\nForests offer feature_importances_, but those numbers are computed on the training data from how much each feature reduced impurity, and they’re biased towards continuous features with many possible thresholds. Permutation importance asks a sharper question on held-out data: if I shuffle this one column, breaking its link with the target, how much does the score drop?
import pandas as pd\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.inspection import permutation_importance\nfrom sklearn.model_selection import train_test_split\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\",\n \"avg_satisfaction\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\nforest = RandomForestClassifier(n_estimators=300, min_samples_leaf=5, random_state=42).fit(X_train, y_train)\n\nresult = permutation_importance(forest, X_test, y_test, n_repeats=10, random_state=42)\ntable = pd.DataFrame({\"impurity (train)\": forest.feature_importances_,\n \"permutation (test)\": result.importances_mean}, index=features)\nprint(table.sort_values(\"permutation (test)\", ascending=False).round(3))Impurity importance ranks tenure_days second. Permutation importance on the test set says it barely matters: the forest used it to fit noise in the training data. Shuffling recency costs about 13 points of accuracy; shuffling satisfaction costs nothing (the small negative values are noise). The honest summary for Cartwheel: recency and order count drive the predictions, and the satisfaction survey, in its current half-empty state, adds nothing.
Accuracy has carried you this far, but it treats a missed churner and a false alarm as equally bad, and it ignores the ranking you built for marketing. Next lesson replaces it with metrics that look at each kind of mistake separately.
\n","toc":[{"id":"sec-1","text":"Random forests: many trees, averaged","level":2},{"id":"sec-2","text":"Gradient boosting: trees that fix each other","level":2},{"id":"sec-3","text":"Both on churn, with the missing column back","level":2},{"id":"sec-4","text":"What do the trees rely on?","level":2}],"takeaways":["A random forest trains many deep trees on bootstrap samples with random feature subsets and averages their votes, which cancels much of a single tree’s variance.","Gradient boosting builds shallow trees one after another, each fitted to the errors of the ensemble so far;learning_rate and max_iter trade off speed and overfitting.","HistGradientBoostingClassifier is scikit-learn’s fast boosting model and accepts missing values (NaN) directly.","Impurity-based feature_importances_ favour features with many distinct values; permutation importance on held-out data measures what the model really relies on.","On small tabular datasets, ensembles often add only a few points over a good linear model, so compare them against it rather than assuming they win."],"further":[{"title":"Ensembles: Gradient boosting, random forests, bagging","url":"https://scikit-learn.org/stable/modules/ensemble.html"},{"title":"HistGradientBoostingClassifier","url":"https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingClassifier.html"},{"title":"Permutation feature importance","url":"https://scikit-learn.org/stable/modules/permutation_importance.html"}],"quiz":[{"q":"Why does a random forest usually beat a single unconstrained tree on new data?
","options":[{"text":"Each of its trees is shallower and therefore cannot overfit.","why":"Forest trees are typically grown deep. The benefit comes from averaging, not from limiting each tree."},{"text":"It uses gradient descent to tune the trees together.","why":"Forest trees are trained independently; nothing ties them together during training."},{"text":"It trains on more data than the single tree.","why":"Each tree sees a bootstrap sample of the same training set, so no extra data is involved."},{"text":"Averaging many trees trained on different samples and feature subsets cancels out their individual mistakes.","why":"Correct. Each tree overfits in its own way; the errors are partly independent, so the average is more stable."}],"answer":3},{"q":"In gradient boosting, what does each new tree learn?
","options":[{"text":"To correct the errors that the ensemble built so far still makes.","why":"Correct. Boosting is sequential; every tree targets what’s left over."},{"text":"The same target as the first tree, on a fresh bootstrap sample.","why":"That describes bagging, the idea behind random forests."},{"text":"A random subset of the features, ignoring previous trees.","why":"Random feature subsets are a forest trick. Boosting trees depend on the trees before them."}],"answer":0},{"q":"avg_satisfaction is blank for half the customers. Which model can you fit on it with no extra preprocessing?
LogisticRegression, because it ignores missing values.","why":"It raises ValueError: Input X contains NaN. You would need an imputer first."},{"text":"HistGradientBoostingClassifier, which learns which branch missing values should follow at each split.","why":"Correct. Native NaN support is one of its main practical advantages."},{"text":"KNeighborsClassifier, because it skips missing columns when measuring distance.","why":"k-NN raises an error on NaN too; distances can’t be computed with gaps."}],"answer":1},{"q":"A forest’s feature_importances_ ranks tenure_days second, but permutation importance on the test set gives it almost zero. What’s the most likely explanation?
tenure_days.","why":"Every row has the feature. Small test sets add noise, but this pattern has a known cause."},{"text":"The forest used tenure_days in many splits that fit training noise, which inflates impurity importance but doesn’t help on new data.","why":"Correct. Impurity importance is computed on training data and favours continuous features with many possible thresholds."},{"text":"Shuffling tenure_days made the model more accurate, which proves the feature is harmful.","why":"A near-zero result means shuffling changed little, not that the model improved."}],"answer":2}],"exercise":{"kind":"python","files":["churn.csv"],"starter":"import pandas as pd\nfrom sklearn.ensemble import HistGradientBoostingClassifier\nfrom sklearn.inspection import permutation_importance\nfrom sklearn.model_selection import train_test_split\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\",\n \"avg_satisfaction\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\n\n# 1. Gradient boosting on all nine features (missing values included):\n# max_depth=3, learning_rate=0.05, max_iter=200, random_state=42\nbooster = HistGradientBoostingClassifier(random_state=42)\nbooster.fit(X_train.drop(columns=\"avg_satisfaction\"), y_train)\n\n# 2. Test accuracy\ntest_acc = None\n\n# 3. Permutation importance on the TEST set (n_repeats=5, random_state=0),\n# then the feature names sorted from most to least important\nranking = []\n","solution":"import pandas as pd\nfrom sklearn.ensemble import HistGradientBoostingClassifier\nfrom sklearn.inspection import permutation_importance\nfrom sklearn.model_selection import train_test_split\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\",\n \"avg_satisfaction\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\n\nbooster = HistGradientBoostingClassifier(max_depth=3, learning_rate=0.05, max_iter=200,\n random_state=42)\nbooster.fit(X_train, y_train)\n\ntest_acc = booster.score(X_test, y_test)\n\nresult = permutation_importance(booster, X_test, y_test, n_repeats=5, random_state=0)\nranking = pd.Series(result.importances_mean, index=features).sort_values(ascending=False).index.tolist()\nprint(test_acc, ranking[:3])\n","checks":[{"id":"nine-features","test":"booster.n_features_in_ == 9"},{"id":"settings","test":"booster.max_depth == 3 and booster.learning_rate == 0.05 and booster.max_iter == 200"},{"id":"accuracy","test":"test_acc is not None and 0.74 <= float(test_acc) <= 0.82 and abs(float(test_acc) - booster.score(X_test, y_test)) < 1e-9"},{"id":"ranking","test":"len(ranking) == 9 and set(ranking) == set(features) and ranking[0] == 'days_since_last_order'"}],"title":"Boost with the gaps left in","prompt":"Fit a gradient boosting model on all nine features, including the half-empty avg_satisfaction, and find out what it relies on.
booster: HistGradientBoostingClassifier(max_depth=3, learning_rate=0.05, max_iter=200, random_state=42) fitted on all of X_train (no columns dropped).test_acc: its test accuracy.ranking: a list of the nine feature names sorted by permutation importance on the test set (n_repeats=5, random_state=0), most important first.HistGradientBoostingClassifier routes missing values itself.","permutation_importance(booster, X_test, y_test, n_repeats=5, random_state=0) returns an object with importances_mean.","Wrap the means in a pd.Series indexed by features, sort descending, and take .index.tolist()."],"explanation":"Gentle boosting lands around 0.78 on the test set, a few points above logistic regression. Permutation importance puts days_since_last_order far ahead of everything else, and avg_satisfaction near zero or below: the model can use the column, but the survey as collected doesn’t help. Computing importance on the test set measures what generalises, not what the trees memorised.