{"id":"l-ml-4-1","courseId":"ml-crash-course","lang":"en","kind":"lesson","summary":"Replace a single train/test split with k-fold cross-validation, read the mean and spread of the fold scores, and decide whether a difference between models is real or noise.","html":"

Through section 3 every comparison came with a hedge: “on 100 test customers, two points is two customers”. The forest scored 0.76, boosting 0.78, logistic regression 0.72. Was boosting really better, or did it get a lucky test set? One split can’t tell you. You need several, and that’s what cross-validation gives you.

\n

Five splits instead of one

\n

k-fold cross-validation cuts the training data into k equal folds (5 is the usual default). It trains k models: each one learns from k−1 folds and is scored on the fold it didn’t see. You end up with k scores, and every training row has been in a validation fold exactly once.

\n
\n Five rows, one per round. In each row the training data is shown as five blocks; a different block is highlighted as the validation fold each time, and the other four are used for training. Each round produces one score.\n round\n score\n 1\n \n 0.72\n 2\n \n 0.75\n 3\n \n 0.68\n 4\n \n 0.81\n 5\n \n 0.68\n highlighted = validation fold, others = training folds\n
5-fold cross-validation
\n

The mean of the k scores is a steadier estimate than any single split. The spread between them is just as valuable: it shows how much a score can move for reasons that have nothing to do with the model.

\n

Cross-validating the churn models

\n

cross_val_score runs the whole loop for one metric; cross_validate handles several metrics at once. For classification, pass a StratifiedKFold with shuffle=True so each fold keeps the 55% churn rate and the rows are mixed first. Create the splitter once and reuse it for every model so they all face exactly the same folds.

\n
Python
import pandas as pd\nfrom sklearn.dummy import DummyClassifier\nfrom sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.tree import DecisionTreeClassifier\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n            \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n    churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n    stratify=churn[\"churned\"])\n\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\nmodels = {\n    \"dummy\": DummyClassifier(),\n    \"logistic\": make_pipeline(StandardScaler(), LogisticRegression()),\n    \"tree depth 3\": DecisionTreeClassifier(max_depth=3, random_state=42),\n    \"forest\": RandomForestClassifier(n_estimators=200, min_samples_leaf=5, random_state=42),\n    \"boosting\": HistGradientBoostingClassifier(max_depth=3, learning_rate=0.05,\n                                               max_iter=100, random_state=42),\n}\nfor name, model in models.items():\n    r = cross_validate(model, X_train, y_train, cv=cv, scoring=[\"accuracy\", \"roc_auc\"])\n    acc, auc = r[\"test_accuracy\"], r[\"test_roc_auc\"]\n    print(f\"{name:13} accuracy {acc.mean():.3f} ± {acc.std():.3f}   ROC AUC {auc.mean():.3f} ± {auc.std():.3f}\")
\n

Read the spreads first. Logistic regression’s accuracy ranges from 0.68 to 0.81 across folds, a 13-point swing from nothing but which customers landed where. Against that noise, the accuracy differences between the four real models (0.73 to 0.76) mean very little: every gap is smaller than one standard deviation.

\n

ROC AUC, which grades the full ranking, is more decisive: logistic regression and the forest sit around 0.81, boosting around 0.79, and the depth-3 tree trails at 0.74 with the largest spread. That’s a pattern you can act on.

\n

For the 80-day rule, cross-validation means re-choosing the cut-off on each set of training folds and scoring it on the held-out fold. Done that way it averages 0.70 accuracy. Every real model beats it, by a few points.

\n

What you can conclude

\n

Put together, cross-validation tells a calmer story than section 3’s single split did. Boosting’s 0.78 test accuracy was partly luck; across five folds it’s 0.75, within noise of everything else, and it ranks worse than logistic regression. The forest and logistic regression are tied on ranking. Given a tie, prefer the simpler model: logistic regression trains in milliseconds, its coefficients can be explained, and it has nothing to tune but C.

\n

That’s the skeptic’s payoff. Without cross-validation you’d have shipped boosting because of a lucky number.

\n\n

Choosing k, and what CV doesn’t fix

\n\n\n

So far every model has used the eight numeric columns. The data also has countries, segments and a half-empty satisfaction score that need encoding and imputing first. Next lesson builds a pipeline that does all of that inside each fold, so cross-validation stays honest.

\n","toc":[{"id":"sec-1","text":"Five splits instead of one","level":2},{"id":"sec-2","text":"Cross-validating the churn models","level":2},{"id":"sec-3","text":"What you can conclude","level":2},{"id":"sec-4","text":"Choosing k, and what CV doesn’t fix","level":2}],"takeaways":["k-fold cross-validation trains k models, each validated on a different fold, so every training row is used for validation exactly once.","The mean of the fold scores estimates performance; their standard deviation tells you how much a single split could mislead you.","Use StratifiedKFold with shuffle=True and a fixed random_state for classification, and pass the same splitter to every model you compare.","cross_validate returns several metrics at once, plus fit and score times.","Cross-validate on the training set to choose; keep the test set for one final check."],"further":[{"title":"Cross-validation: evaluating estimator performance","url":"https://scikit-learn.org/stable/modules/cross_validation.html"},{"title":"cross_validate","url":"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html"},{"title":"StratifiedKFold","url":"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html"}],"quiz":[{"q":"

In 5-fold cross-validation on 298 training rows, how many models are fitted and how many rows does each one train on?

","options":[{"text":"One model, trained on all 298 rows and scored five times.","why":"Scoring one model five times on data it trained on would measure memorisation, not generalisation."},{"text":"Five models, each trained on about 60 rows and validated on the rest.","why":"That’s backwards. Each model validates on one fold (about 60 rows) and trains on the other four."},{"text":"Five models, each trained on about 238 rows and validated on the remaining 60.","why":"Correct. Every row sits in the validation fold once and in the training folds four times."},{"text":"Twenty-five models, one for every pair of folds.","why":"Plain k-fold fits one model per fold. Repeated or nested schemes fit more, but that’s not what this is."}],"answer":2},{"q":"

Model A scores 0.75 ± 0.04 accuracy across five folds; model B scores 0.73 ± 0.05. What’s the fair conclusion?

","options":[{"text":"The difference is small compared with the fold-to-fold spread, so treat them as roughly tied.","why":"Correct. A two-point gap inside a four-to-five-point spread could easily reverse with another shuffle."},{"text":"A is clearly better, because its mean is higher.","why":"Means alone ignore the noise. With spreads this size, the ranking isn’t reliable."},{"text":"B is better, because a larger standard deviation means it adapts more.","why":"A larger spread means less stable performance, not a strength."}],"answer":0},{"q":"

Why use the same StratifiedKFold(..., random_state=42) object for every model you compare?

","options":[{"text":"Different splitters would make some models train faster.","why":"Speed is unaffected; the concern is fairness of the comparison."},{"text":"scikit-learn requires a single splitter per script.","why":"There’s no such rule; you can create as many as you like."},{"text":"It shuffles the data differently for each model, which is fairer.","why":"The opposite. Different shuffles add noise to the comparison."},{"text":"Every model is then scored on exactly the same folds, so differences come from the models, not the splits.","why":"Correct. Paired comparisons on identical folds are much less noisy."}],"answer":3},{"q":"

After cross-validation picks logistic regression, what should you do with the test set?

","options":[{"text":"Add it to the cross-validation to get more folds.","why":"Then nothing is left for an unbiased final estimate."},{"text":"Fit the chosen model on all training data and score it on the test set once.","why":"Correct. The test score is the number you report, and it hasn’t influenced any decision."},{"text":"Skip it, since the cross-validation score is already unbiased.","why":"The CV score was used to choose between models, so it’s slightly optimistic for the winner."}],"answer":1}],"exercise":{"kind":"python","files":["churn.csv"],"starter":"import pandas as pd\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\n\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n\n# 1. Cross-validate both models on the TRAINING data with the same `cv`,\n# scoring both \"accuracy\" and \"roc_auc\"\nlogistic = make_pipeline(StandardScaler(), LogisticRegression())\nknn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=15))\n\n# 2. Mean ROC AUC for each model, and the standard deviation of k-NN's fold AUCs\nlogistic_auc = None\nknn_auc = None\nknn_auc_std = None\n\n# 3. Fit the model with the higher mean AUC on all training data; score accuracy on the test set once\nfinal_test_acc = None\n","solution":"import pandas as pd\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import StratifiedKFold, cross_validate, train_test_split\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\n\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n\nlogistic = make_pipeline(StandardScaler(), LogisticRegression())\nknn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=15))\n\nr_log = cross_validate(logistic, X_train, y_train, cv=cv, scoring=[\"accuracy\", \"roc_auc\"])\nr_knn = cross_validate(knn, X_train, y_train, cv=cv, scoring=[\"accuracy\", \"roc_auc\"])\n\nlogistic_auc = r_log[\"test_roc_auc\"].mean()\nknn_auc = r_knn[\"test_roc_auc\"].mean()\nknn_auc_std = r_knn[\"test_roc_auc\"].std()\n\nwinner = logistic if logistic_auc >= knn_auc else knn\nfinal_test_acc = winner.fit(X_train, y_train).score(X_test, y_test)\nprint(round(logistic_auc, 3), round(knn_auc, 3), round(knn_auc_std, 3), final_test_acc)\n","checks":[{"id":"logistic-auc","test":"logistic_auc is not None and abs(float(logistic_auc) - 0.807) < 0.002"},{"id":"knn-auc","test":"knn_auc is not None and abs(float(knn_auc) - float(__import__('sklearn.model_selection', fromlist=['x']).cross_val_score(__import__('sklearn.pipeline', fromlist=['x']).make_pipeline(__import__('sklearn.preprocessing', fromlist=['x']).StandardScaler(), __import__('sklearn.neighbors', fromlist=['x']).KNeighborsClassifier(n_neighbors=15)), X_train, y_train, cv=__import__('sklearn.model_selection', fromlist=['x']).StratifiedKFold(n_splits=5, shuffle=True, random_state=42), scoring='roc_auc').mean())) < 1e-6"},{"id":"knn-std","test":"knn_auc_std is not None and abs(float(knn_auc_std) - float(__import__('sklearn.model_selection', fromlist=['x']).cross_val_score(__import__('sklearn.pipeline', fromlist=['x']).make_pipeline(__import__('sklearn.preprocessing', fromlist=['x']).StandardScaler(), __import__('sklearn.neighbors', fromlist=['x']).KNeighborsClassifier(n_neighbors=15)), X_train, y_train, cv=__import__('sklearn.model_selection', fromlist=['x']).StratifiedKFold(n_splits=5, shuffle=True, random_state=42), scoring='roc_auc').std())) < 1e-6"},{"id":"final","test":"final_test_acc is not None and abs(float(final_test_acc) - 0.72) < 0.005"}],"title":"Logistic regression or k-NN?","prompt":"

Settle a model choice with cross-validation, then use the test set once.

\n
    \n
  1. Cross-validate logistic and knn on the training data with the provided cv, scoring "accuracy" and "roc_auc".
    2. \n
  2. logistic_auc and knn_auc: each model’s mean ROC AUC across the folds. knn_auc_std: the standard deviation of k-NN’s fold AUCs.
    3. \n
  3. final_test_acc: fit whichever model has the higher mean AUC on all of X_train, then compute its test accuracy.
    4. \n
\n","hints":["cross_validate(model, X_train, y_train, cv=cv, scoring=["accuracy", "roc_auc"]) returns a dict with a "test_roc_auc" array.",".mean() and .std() work directly on those arrays.","Pick the winner with a plain if, then winner.fit(X_train, y_train).score(X_test, y_test)."],"explanation":"

Scaled k-NN with 15 neighbours ranks customers a little worse than logistic regression on average, and its fold scores vary noticeably. Because both models faced identical folds, the comparison is fair. The winner is refitted on all training rows and scored once on the test set (0.72): that’s the number you report, and it played no part in the choice.

\n","checkLabels":{"logistic-auc":"logistic_auc is the mean cross-validated ROC AUC","knn-auc":"knn_auc is k-NN’s mean ROC AUC on the same folds","knn-std":"knn_auc_std is the spread of k-NN’s fold AUCs","final":"final_test_acc is the winner’s test accuracy"}},"words":680,"readMinutes":3}