The Workflow and the Train/Test Split

A model that scores 99% is either very good or has seen the answers. You can’t tell which from the number alone. The single most useful habit in machine learning is setting aside data the model never sees during training, so that its score on that data means something.

The workflow, start to finish

Every supervised project you’ll work on follows roughly the same loop. The steps are boring on purpose; skipping one is how projects fail.

Frame the question precisely: what is predicted, for whom, at what moment, and what action follows. For Cartwheel: “On 30 September, for every customer who has ordered before, predict whether they will order in the next 90 days, so marketing can target win-back offers.” Collect examples where the answer is known: that’s churn.csv. Split off a test set. Build a baseline (next lesson but one). Train models, evaluate them, iterate, and only then ship and keep watching.

Features and target

scikit-learn expects two things: a feature matrix X, one row per example and one column per feature, and a target y with one answer per row. A pandas DataFrame works as X and a Series works as y.

For now you’ll use the eight numeric columns. Text columns such as country need encoding first (section 4 handles that), and avg_satisfaction has gaps that some models can’t accept yet.

import pandas as pd

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X = churn[features]
y = churn["churned"]
print(X.shape, y.shape)
print(y.value_counts())

customer_id stays out. An id is an arbitrary label, and a flexible model could still find splits on it that fit the training rows by coincidence. Anything that wouldn’t make sense to a person explaining churn shouldn’t be a feature.

Holding out a test set

train_test_split shuffles the rows and cuts them into two parts. You train on one and measure on the other.

import pandas as pd
from sklearn.model_selection import train_test_split

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X, y = churn[features], churn["churned"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y
)
print("train:", X_train.shape, "churn rate", round(y_train.mean(), 3))
print("test: ", X_test.shape, "churn rate", round(y_test.mean(), 3))

Three arguments do the work:

• test_size=0.25 keeps a quarter of the rows back: 100 customers for testing, 298 for training. With a few hundred rows, 20–30% is a sensible test share; with millions, 1–5% is plenty.
• random_state=42 fixes the shuffle so you get the same split every run. Without it, every rerun reshuffles and your score wobbles for reasons that have nothing to do with your model.
• stratify=y keeps the churn rate the same in both parts (55%). With 100 test rows, an unlucky shuffle could otherwise give you a test set that’s 45% or 62% churned, and every number you compute on it would shift.

The order of the four returned values is fixed: X_train, X_test, y_train, y_test. Mixing them up is a common source of nonsense scores.

What the split buys you

Think of the split as a contract with your future self. Everything you do from here on (choosing features, choosing models, tuning settings) happens on the training part. The test part stays sealed until you’re ready to report a result. With X_train and y_train you can fit any model you like. With X_test and y_test you get one honest answer to “how well does this work on customers it hasn’t seen?”. That’s the number that predicts what happens when marketing uses your model in December. Next lesson you’ll fit the first model and see how big the gap between those two numbers can be.