Decision Trees

Ask a support lead how they spot customers who are about to leave and you’ll get something like: “If they haven’t ordered in months, they’re gone. If they’re recent but only bought once or twice, it’s a toss-up. Regulars who ordered recently are fine.” That’s a decision tree. This lesson grows one from data and checks whether it agrees.

Twenty questions, chosen by data

A decision tree predicts by asking yes/no questions about one feature at a time. To build it, the algorithm looks at every feature and every possible threshold, and picks the question that splits the training customers into the two purest groups: ideally all churners on one side and all stayers on the other. Purity is measured with the Gini impurity by default (0 for a group of one class only, 0.5 for a 50/50 mix). Then it repeats the search inside each group, and so on down.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, export_text

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])

tree = DecisionTreeClassifier(max_depth=2, random_state=42).fit(X_train, y_train)
print(export_text(tree, feature_names=features, show_weights=True))

export_text prints the rules, and show_weights=True adds how many training customers of each class (stayed, churned) reached each leaf.

Three leaves match the support lead almost word for word. Long-silent customers churn (84 of 95). Recent regulars with four or more orders stay (60 of 73). Recent customers with few orders are a toss-up (67 of 127 churn). The tree also discovered an interaction nobody had to code: order count matters, but only among recent customers.

The fourth leaf is a warning. The tree spent a whole question isolating three very long-tenured customers. That’s the greedy search at work: it picks the split that helps most right now, even if the help comes from a group too small to mean anything.

How a tree predicts probabilities

A customer who falls into a leaf gets that leaf’s class shares as probabilities. Land in the “regulars” leaf and predict_proba returns about [0.82, 0.18]. So a depth-2 tree can only ever output four distinct probabilities, one per leaf. That’s coarse, but honest. The trouble starts when the tree has no limits.

Growing too far, and how to stop it

An unconstrained tree keeps splitting until every leaf is pure. On the churn data that’s 64 leaves, 11 levels deep, and 100% training accuracy. Every leaf is pure, so every probability is exactly 0 or 1: the tree claims total certainty about every test customer while being right only 72% of the time. Ranked by such probabilities, almost all customers tie, and the call list from last lesson becomes a lottery.

Two hyperparameters do most of the work of keeping a tree honest:

• max_depth caps how many questions in a row the tree can ask.
• min_samples_leaf refuses any split that would leave fewer than that many training customers in a leaf. It targets exactly the problem in the fourth leaf above, and it lets the tree grow deep where there’s plenty of data and stay shallow where there isn’t.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])

for settings in [{}, {"max_depth": 3}, {"min_samples_leaf": 20}]:
    t = DecisionTreeClassifier(random_state=42, **settings).fit(X_train, y_train)
    distinct = len(set(t.predict_proba(X_test)[:, 1]))
    print(f"{str(settings):24} leaves {t.get_n_leaves():3}  train {t.score(X_train, y_train):.2f}  "
          f"test {t.score(X_test, y_test):.2f}  distinct probabilities {distinct}")

t = DecisionTreeClassifier(min_samples_leaf=20, random_state=42).fit(X_train, y_train)
print(pd.Series(t.feature_importances_, index=features).sort_values(ascending=False).round(2))

feature_importances_ says how much each feature reduced impurity across all its splits, normalised to sum to 1. Recency takes over half, order count comes second; coupons and support tickets get zero because the tree never used them. Treat these numbers as a description of this tree, not of the world: features the tree didn’t need can still be informative, and the next lesson shows a more trustworthy way to measure importance.

That instability is also an opportunity. If one tree is noisy, average many trees grown on slightly different data and the noise cancels out. That’s the idea behind random forests and gradient boosting, next.