Cross-Validation

Through section 3 every comparison came with a hedge: “on 100 test customers, two points is two customers”. The forest scored 0.76, boosting 0.78, logistic regression 0.72. Was boosting really better, or did it get a lucky test set? One split can’t tell you. You need several, and that’s what cross-validation gives you.

Five splits instead of one

k-fold cross-validation cuts the training data into k equal folds (5 is the usual default). It trains k models: each one learns from k−1 folds and is scored on the fold it didn’t see. You end up with k scores, and every training row has been in a validation fold exactly once.

The mean of the k scores is a steadier estimate than any single split. The spread between them is just as valuable: it shows how much a score can move for reasons that have nothing to do with the model.

Cross-validating the churn models

cross_val_score runs the whole loop for one metric; cross_validate handles several metrics at once. For classification, pass a StratifiedKFold with shuffle=True so each fold keeps the 55% churn rate and the rows are mixed first. Create the splitter once and reuse it for every model so they all face exactly the same folds.

import pandas as pd
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
models = {
    "dummy": DummyClassifier(),
    "logistic": make_pipeline(StandardScaler(), LogisticRegression()),
    "tree depth 3": DecisionTreeClassifier(max_depth=3, random_state=42),
    "forest": RandomForestClassifier(n_estimators=200, min_samples_leaf=5, random_state=42),
    "boosting": HistGradientBoostingClassifier(max_depth=3, learning_rate=0.05,
                                               max_iter=100, random_state=42),
}
for name, model in models.items():
    r = cross_validate(model, X_train, y_train, cv=cv, scoring=["accuracy", "roc_auc"])
    acc, auc = r["test_accuracy"], r["test_roc_auc"]
    print(f"{name:13} accuracy {acc.mean():.3f} ± {acc.std():.3f}   ROC AUC {auc.mean():.3f} ± {auc.std():.3f}")

Read the spreads first. Logistic regression’s accuracy ranges from 0.68 to 0.81 across folds, a 13-point swing from nothing but which customers landed where. Against that noise, the accuracy differences between the four real models (0.73 to 0.76) mean very little: every gap is smaller than one standard deviation.

ROC AUC, which grades the full ranking, is more decisive: logistic regression and the forest sit around 0.81, boosting around 0.79, and the depth-3 tree trails at 0.74 with the largest spread. That’s a pattern you can act on.

For the 80-day rule, cross-validation means re-choosing the cut-off on each set of training folds and scoring it on the held-out fold. Done that way it averages 0.70 accuracy. Every real model beats it, by a few points.

What you can conclude

Put together, cross-validation tells a calmer story than section 3’s single split did. Boosting’s 0.78 test accuracy was partly luck; across five folds it’s 0.75, within noise of everything else, and it ranks worse than logistic regression. The forest and logistic regression are tied on ranking. Given a tie, prefer the simpler model: logistic regression trains in milliseconds, its coefficients can be explained, and it has nothing to tune but C.

That’s the skeptic’s payoff. Without cross-validation you’d have shipped boosting because of a lucky number.

Choosing k, and what CV doesn’t fix

• k = 5 is the standard. k = 10 gives each model more training data and a slightly less pessimistic estimate, at twice the cost. With very small datasets (a few dozen rows), more folds help; with large ones, even 3 folds are plenty.
• The fold scores aren’t independent (the training folds overlap), so treat the standard deviation as a rough guide to noise, not a formal error bar.
• Cross-validation doesn’t protect you from leakage. If information from the validation fold sneaks into training, every fold is optimistic. The leakage lesson is about exactly that.

So far every model has used the eight numeric columns. The data also has countries, segments and a half-empty satisfaction score that need encoding and imputing first. Next lesson builds a pipeline that does all of that inside each fold, so cross-validation stays honest.