A model that scores 99% is either very good or has seen the answers. You can’t tell which from the number alone. The single most useful habit in machine learning is setting aside data the model never sees during training, so that its score on that data means something.
\nEvery supervised project you’ll work on follows roughly the same loop. The steps are boring on purpose; skipping one is how projects fail.
\nFrame the question precisely: what is predicted, for whom, at what moment, and what action follows. For Cartwheel: “On 30 September, for every customer who has ordered before, predict whether they will order in the next 90 days, so marketing can target win-back offers.” Collect examples where the answer is known: that’s churn.csv. Split off a test set. Build a baseline (next lesson but one). Train models, evaluate them, iterate, and only then ship and keep watching.
scikit-learn expects two things: a feature matrix X, one row per example and one column per feature, and a target y with one answer per row. A pandas DataFrame works as X and a Series works as y.
For now you’ll use the eight numeric columns. Text columns such as country need encoding first (section 4 handles that), and avg_satisfaction has gaps that some models can’t accept yet.
import pandas as pd\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX = churn[features]\ny = churn[\"churned\"]\nprint(X.shape, y.shape)\nprint(y.value_counts())customer_id stays out. An id is an arbitrary label, and a flexible model could still find splits on it that fit the training rows by coincidence. Anything that wouldn’t make sense to a person explaining churn shouldn’t be a feature.
train_test_split shuffles the rows and cuts them into two parts. You train on one and measure on the other.
import pandas as pd\nfrom sklearn.model_selection import train_test_split\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX, y = churn[features], churn[\"churned\"]\n\nX_train, X_test, y_train, y_test = train_test_split(\n X, y, test_size=0.25, random_state=42, stratify=y\n)\nprint(\"train:\", X_train.shape, \"churn rate\", round(y_train.mean(), 3))\nprint(\"test: \", X_test.shape, \"churn rate\", round(y_test.mean(), 3))Three arguments do the work:
\ntest_size=0.25 keeps a quarter of the rows back: 100 customers for testing, 298 for training. With a few hundred rows, 20–30% is a sensible test share; with millions, 1–5% is plenty.random_state=42 fixes the shuffle so you get the same split every run. Without it, every rerun reshuffles and your score wobbles for reasons that have nothing to do with your model.stratify=y keeps the churn rate the same in both parts (55%). With 100 test rows, an unlucky shuffle could otherwise give you a test set that’s 45% or 62% churned, and every number you compute on it would shift.The order of the four returned values is fixed: X_train, X_test, y_train, y_test. Mixing them up is a common source of nonsense scores.
Think of the split as a contract with your future self. Everything you do from here on (choosing features, choosing models, tuning settings) happens on the training part. The test part stays sealed until you’re ready to report a result. With X_train and y_train you can fit any model you like. With X_test and y_test you get one honest answer to “how well does this work on customers it hasn’t seen?”. That’s the number that predicts what happens when marketing uses your model in December. Next lesson you’ll fit the first model and see how big the gap between those two numbers can be.
stratify=y for classification so the train and test sets keep the same class balance, and set random_state so the split is repeatable.","The test set is for the final check; looking at it while you make decisions turns it into training data."],"further":[{"title":"train_test_split","url":"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html"},{"title":"Common pitfalls and recommended practices","url":"https://scikit-learn.org/stable/common_pitfalls.html"}],"quiz":[{"q":"You fit a model on all 398 churn rows and it scores 0.99 on those same rows. What can you conclude?
","options":[{"text":"The model will be about 99% accurate on next month’s customers.","why":"The model has seen every one of those answers. A high score on seen data is consistent with pure memorisation."},{"text":"Very little about future performance; you need rows the model never saw.","why":"Correct. Only held-out data tells you how the model generalises to new customers."},{"text":"The model is overfitting, so you should delete features until the score drops.","why":"You can’t diagnose overfitting without a held-out score to compare against, and deleting features blindly is not a fix."}],"answer":1},{"q":"What does stratify=y change in train_test_split(X, y, test_size=0.25, stratify=y)?
Why drop customer_id from the features?
id.","why":"scikit-learn doesn’t inspect column names for meaning. It would happily use the id, which is the problem."},{"text":"It is the target, so it must not be in X.","why":"The target is churned. The id is a row label, not an answer."},{"text":"Ids make training slower because they are large numbers.","why":"Large numbers don’t slow training meaningfully. The reason is about what the model can learn, not speed."}],"answer":0},{"q":"You try twenty model settings, keep the one with the best test score, and report that score. What is wrong?
","options":[{"text":"Nothing; that is what the test set is for.","why":"The test set is for one final, unbiased estimate. Choosing among twenty options by it makes it part of training."},{"text":"Twenty settings is too few to find a good model.","why":"The number of settings isn’t the issue. Any selection based on the test set biases the reported score upward."},{"text":"The reported score is optimistic, because you picked the setting that happened to suit this test set.","why":"Correct. Use a validation split or cross-validation to choose, then touch the test set once."}],"answer":2}],"exercise":{"kind":"python","files":["churn.csv"],"starter":"import pandas as pd\nfrom sklearn.model_selection import train_test_split\n\nchurn = pd.read_csv(\"churn.csv\")\n\n# 1. Feature matrix: the eight numeric feature columns (no id, no target)\nX = churn.drop(columns=[\"country\", \"segment\", \"avg_satisfaction\"])\n# 2. Target\ny = churn[\"churned\"]\n\n# 3. Split: 25% test, random_state=42, keep the churn rate equal in both parts\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n","solution":"import pandas as pd\nfrom sklearn.model_selection import train_test_split\n\nchurn = pd.read_csv(\"churn.csv\")\n\nX = churn.drop(columns=[\"customer_id\", \"country\", \"segment\", \"avg_satisfaction\", \"churned\"])\ny = churn[\"churned\"]\n\nX_train, X_test, y_train, y_test = train_test_split(\n X, y, test_size=0.25, random_state=42, stratify=y\n)\n","checks":[{"id":"features","test":"set(X.columns) == {'tenure_days', 'orders', 'total_spent', 'avg_order_value', 'days_since_last_order', 'used_coupon', 'support_tickets', 'mobile_share'}"},{"id":"target","test":"list(y.unique()) != [] and set(y.unique()) == {0, 1} and len(y) == 398"},{"id":"sizes","test":"X_train.shape == (298, 8) and X_test.shape == (100, 8) and len(y_test) == 100"},{"id":"stratified","test":"abs(float(y_test.mean()) - float(y.mean())) < 0.01 and abs(float(y_train.mean()) - float(y.mean())) < 0.01"}],"title":"Build X and y, then split","prompt":"Prepare the churn data for modelling.
\nX: the eight numeric feature columns: everything except customer_id, country, segment, avg_satisfaction and the target.y: the churned column.X_train, X_test, y_train, y_test with a 25% test set, random_state=42, and the churn rate kept equal in both parts.customer_id and churned are still inside X in the starter. Drop them too.","test_size=0.25 gives 100 test rows from 398.","Pass stratify=y so both parts keep the 55% churn rate."],"explanation":"Dropping the id and the target leaves eight numeric features. test_size=0.25 holds back 100 customers, random_state=42 makes the split repeatable, and stratify=y keeps the churn rate at 55% in both parts so the test score isn’t skewed by an unlucky shuffle. Leaving churned inside X would let any model read the answer straight off the features, which is the most extreme form of leakage.