Data Leakage

Every experienced data scientist has a story about the model that was too good. Accuracy of 99% in the notebook, coin-flip performance in production. The cause is almost always the same: during training or evaluation the model had access to information it wouldn’t have in real life. That’s data leakage, and it’s dangerous because it doesn’t look like a bug. It looks like success.

Target leakage: features from the future

Cartwheel’s churn table is a snapshot: every feature describes a customer as of 30 September 2025, and churned records what happened in the 90 days after. That timeline is the contract the model lives by. At prediction time, you only know what happened before the snapshot.

Now imagine an analyst rebuilds the table in January and computes days_since_last_order as of the export date instead of the snapshot date. For churners, who didn’t order in the window, the value is 92 days larger. For customers who stayed, it’s small, because they ordered recently. The column now encodes the answer:

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
rng = np.random.default_rng(1)
# Recency measured at export time (year end), not at the 30 September snapshot
churn["recency_at_export"] = np.where(churn["churned"] == 1,
                                      churn["days_since_last_order"] + 92,
                                      rng.integers(0, 92, len(churn)))

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
model = make_pipeline(StandardScaler(), LogisticRegression())
for cols in [features, features + ["recency_at_export"]]:
    auc = cross_val_score(model, churn[cols], churn["churned"], cv=cv, scoring="roc_auc").mean()
    print(f"{len(cols)} features: ROC AUC {auc:.3f}")

ROC AUC jumps from 0.81 to 0.999. Nothing in the code is wrong; cross-validation is done properly. The leak lives in how the data was built, which is why you have to look for it in the data’s history, not the model.

Target leaks hide in innocent names: account_status, last_contact_reason, refund_amount, total_orders (counted at export, including future orders). The test for every column is one question: would I know this value, exactly as stored, at the moment I make the prediction?

Preprocessing before the split

The second kind of leak happens in code. Any step that learns from data (scaling, imputing, encoding, selecting features) must learn from training rows only. Do it on the whole dataset first and the validation rows have shaped the model.

With scaling, the damage is usually small. With feature selection it can be enormous. Here’s a dataset of pure noise: 200 rows, 2,000 random columns, random labels. No model should beat 50%.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(0)
X = rng.normal(size=(200, 2000))
y = rng.integers(0, 2, size=200)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

X_selected = SelectKBest(f_classif, k=20).fit_transform(X, y)   # selection saw every label
leaky = cross_val_score(LogisticRegression(), X_selected, y, cv=cv).mean()

pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression())
honest = cross_val_score(pipe, X, y, cv=cv).mean()
print(f"select then CV: {leaky:.3f}   select inside CV: {honest:.3f}")

Selecting on all 200 rows finds 20 columns that happen to line up with the labels of every row, including the ones later used for validation. Cross-validation then confirms a pattern that only exists because it was chosen using those very rows: 0.75 accuracy on noise. Inside a pipeline, selection sees only training folds and the score falls back to chance.

Rows that know each other

The third kind comes from how rows relate. Random splits assume rows are independent. When they aren’t, the test set contains near-copies of training rows.

• Repeated entities. If a table has several rows per customer, patient or device, a random split puts the same customer on both sides, and the model gets graded partly on people it has memorised. Use GroupKFold (or GroupShuffleSplit) with the customer id as groups.
• Time. If you’re predicting the future, a random split lets the model learn from rows after the ones it’s tested on. Use TimeSeriesSplit, or a simple cut-off date: train on the past, validate on later periods.
• Duplicates. Exact duplicate rows split across train and test are free answers. Drop them before splitting.

Cartwheel’s churn table has one row per customer and a single snapshot date, so a stratified random split is fine. Change either of those and it isn’t.

With leaks closed off, you can trust cross-validation to judge new ideas, and the most productive new ideas are usually better features. Next lesson builds some from domain knowledge and tests whether they earn their place.