{"id":"l-ml-4-4","courseId":"ml-crash-course","lang":"en","kind":"lesson","summary":"Recognise the three common kinds of data leakage (target leakage, preprocessing before the split, and overlapping rows or time) and prevent them with snapshots, pipelines and the right splitter.","html":"

Every experienced data scientist has a story about the model that was too good. Accuracy of 99% in the notebook, coin-flip performance in production. The cause is almost always the same: during training or evaluation the model had access to information it wouldn’t have in real life. That’s data leakage, and it’s dangerous because it doesn’t look like a bug. It looks like success.

\n

Target leakage: features from the future

\n

Cartwheel’s churn table is a snapshot: every feature describes a customer as of 30 September 2025, and churned records what happened in the 90 days after. That timeline is the contract the model lives by. At prediction time, you only know what happened before the snapshot.

\n
\n A timeline. Customer history up to the 30 September snapshot is where features may come from. The 90 days after the snapshot are the outcome window that defines churned. A feature computed from events inside the outcome window, such as recency measured at year end, is a leak.\n \n \n history: features come from here\n \n outcome window (90 days)\n \n 30 Sep: predict\n 29 Dec: churned known\n \n \"recency at year end\" leaks\n
Features must come from before the prediction date
\n

Now imagine an analyst rebuilds the table in January and computes days_since_last_order as of the export date instead of the snapshot date. For churners, who didn’t order in the window, the value is 92 days larger. For customers who stayed, it’s small, because they ordered recently. The column now encodes the answer:

\n
Python
import numpy as np\nimport pandas as pd\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import StratifiedKFold, cross_val_score\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n            \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nrng = np.random.default_rng(1)\n# Recency measured at export time (year end), not at the 30 September snapshot\nchurn[\"recency_at_export\"] = np.where(churn[\"churned\"] == 1,\n                                      churn[\"days_since_last_order\"] + 92,\n                                      rng.integers(0, 92, len(churn)))\n\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\nmodel = make_pipeline(StandardScaler(), LogisticRegression())\nfor cols in [features, features + [\"recency_at_export\"]]:\n    auc = cross_val_score(model, churn[cols], churn[\"churned\"], cv=cv, scoring=\"roc_auc\").mean()\n    print(f\"{len(cols)} features: ROC AUC {auc:.3f}\")
\n

ROC AUC jumps from 0.81 to 0.999. Nothing in the code is wrong; cross-validation is done properly. The leak lives in how the data was built, which is why you have to look for it in the data’s history, not the model.

\n

Target leaks hide in innocent names: account_status, last_contact_reason, refund_amount, total_orders (counted at export, including future orders). The test for every column is one question: would I know this value, exactly as stored, at the moment I make the prediction?

\n

Preprocessing before the split

\n

The second kind of leak happens in code. Any step that learns from data (scaling, imputing, encoding, selecting features) must learn from training rows only. Do it on the whole dataset first and the validation rows have shaped the model.

\n

With scaling, the damage is usually small. With feature selection it can be enormous. Here’s a dataset of pure noise: 200 rows, 2,000 random columns, random labels. No model should beat 50%.

\n
Python
import numpy as np\nfrom sklearn.feature_selection import SelectKBest, f_classif\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import StratifiedKFold, cross_val_score\nfrom sklearn.pipeline import make_pipeline\n\nrng = np.random.default_rng(0)\nX = rng.normal(size=(200, 2000))\ny = rng.integers(0, 2, size=200)\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n\nX_selected = SelectKBest(f_classif, k=20).fit_transform(X, y)   # selection saw every label\nleaky = cross_val_score(LogisticRegression(), X_selected, y, cv=cv).mean()\n\npipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression())\nhonest = cross_val_score(pipe, X, y, cv=cv).mean()\nprint(f\"select then CV: {leaky:.3f}   select inside CV: {honest:.3f}\")
\n

Selecting on all 200 rows finds 20 columns that happen to line up with the labels of every row, including the ones later used for validation. Cross-validation then confirms a pattern that only exists because it was chosen using those very rows: 0.75 accuracy on noise. Inside a pipeline, selection sees only training folds and the score falls back to chance.

\n\n

Rows that know each other

\n

The third kind comes from how rows relate. Random splits assume rows are independent. When they aren’t, the test set contains near-copies of training rows.

\n\n

Cartwheel’s churn table has one row per customer and a single snapshot date, so a stratified random split is fine. Change either of those and it isn’t.

\n\n

With leaks closed off, you can trust cross-validation to judge new ideas, and the most productive new ideas are usually better features. Next lesson builds some from domain knowledge and tests whether they earn their place.

\n","toc":[{"id":"sec-1","text":"Target leakage: features from the future","level":2},{"id":"sec-2","text":"Preprocessing before the split","level":2},{"id":"sec-3","text":"Rows that know each other","level":2}],"takeaways":["Leakage is any information available during training or evaluation that won’t be available when the model makes real predictions.","Target leakage comes from features recorded after the moment of prediction; ask of every column, would I know this value at prediction time?","Fitting a scaler, imputer or feature selector before splitting lets validation rows shape the model; put every learned step inside a pipeline.","When rows belong to the same customer or follow each other in time, use GroupKFold or TimeSeriesSplit instead of a random split.","A score that looks too good, or a single feature that dominates permutation importance, is a reason to hunt for leaks before celebrating."],"further":[{"title":"Common pitfalls: data leakage","url":"https://scikit-learn.org/stable/common_pitfalls.html#data-leakage"},{"title":"Cross-validation iterators for grouped data","url":"https://scikit-learn.org/stable/modules/cross_validation.html#group-k-fold"},{"title":"TimeSeriesSplit","url":"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html"}],"quiz":[{"q":"

A teammate adds refund_issued_in_q4 to the churn features and ROC AUC jumps from 0.81 to 0.97. The prediction is made on 30 September. What’s the problem?

","options":[{"text":"The feature describes events after the prediction date, so it won’t exist when the model runs; it’s target leakage.","why":"Correct. Q4 refunds happen during the outcome window, so the model is partly reading the answer."},{"text":"The feature is binary, and binary features always inflate AUC.","why":"Many legitimate features are binary (used_coupon). The problem is timing, not type."},{"text":"Nothing; refunds are a known driver of churn.","why":"They may well be, but a model can only use what’s known at prediction time."},{"text":"The model is overfitting and needs a smaller C.","why":"A stronger penalty wouldn’t stop the model using a feature that encodes the outcome."}],"answer":0},{"q":"

On 2,000 columns of pure noise, selecting the 20 ‘best’ columns on all the data and then cross-validating gives 0.75 accuracy. Inside a pipeline it gives 0.48. Why?

","options":[{"text":"The pipeline version uses fewer training rows, so it’s weaker.","why":"Both use the same folds; the difference is where selection happens."},{"text":"Selecting on all rows picked columns that happen to correlate with the labels of the validation rows too, so those rows were effectively seen during training.","why":"Correct. In the pipeline, selection only sees each fold’s training rows, so the noise can’t line up with validation labels."},{"text":"SelectKBest is random, so the two runs chose different columns.","why":"SelectKBest is deterministic; the difference is which rows it looked at."}],"answer":1},{"q":"

The order history has several rows per customer, and you want to predict at order level. Which splitter avoids leakage between folds?

","options":[{"text":"StratifiedKFold(shuffle=True), because it balances the classes.","why":"Class balance doesn’t stop the same customer’s orders landing in both training and validation folds."},{"text":"KFold(shuffle=False), so rows keep their order.","why":"Unshuffled folds still split a customer’s rows across folds when they aren’t contiguous."},{"text":"GroupKFold with the customer id as the group, so each customer’s rows stay in one fold.","why":"Correct. The model is then always validated on customers it never saw."}],"answer":2},{"q":"

Which signal most strongly suggests a leak?

","options":[{"text":"Cross-validated accuracy is a few points above the rule baseline.","why":"That’s what a modest, honest model looks like."},{"text":"The fold scores vary by five points.","why":"Variation reflects small folds and noise, not leakage."},{"text":"Training accuracy is higher than test accuracy.","why":"That’s normal for almost every model."},{"text":"One unfamiliar feature carries almost all the permutation importance and the score is near perfect.","why":"Correct. Real-world problems rarely have a single near-perfect predictor; when one appears, check when and how it was recorded."}],"answer":3}],"exercise":{"kind":"spot-bug","language":"python","code":"import pandas as pd\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import cross_val_score, train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nchurn = pd.read_csv(\"churn_export_january.csv\")\nchurn[\"lifetime_orders\"] = churn[\"orders\"] + churn[\"orders_oct_to_dec\"]\nX = churn.drop(columns=[\"customer_id\", \"churned\", \"country\", \"segment\", \"orders_oct_to_dec\"])\nX = SimpleImputer(strategy=\"median\").fit_transform(X)\ny = churn[\"churned\"]\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)\nmodel = make_pipeline(StandardScaler(), LogisticRegression())\nprint(cross_val_score(model, X_train, y_train, cv=5, scoring=\"roc_auc\").mean())","bugLines":[9,11],"title":"Find the leaks","prompt":"

A colleague’s churn notebook reports a cross-validated ROC AUC of 0.98. The model predicts on 30 September; the January export also includes orders_oct_to_dec, the orders placed between October and December. Select every line that leaks information the model won’t have at prediction time, or lets validation rows influence training.

\n","hints":["Check every feature against the timeline: what would you know on 30 September?","Look for anything that calls fit or fit_transform on data before the split.","There are two leaky lines: one in the data, one in the code."],"explanation":"

Line 9 builds a feature from orders_oct_to_dec, which covers the 90-day outcome window: customers who churned have zero orders there by definition, so lifetime_orders partly encodes the label. Line 10 drops the raw column, but too late: its values already live on inside lifetime_orders. Drop the derived feature too, or compute any order counts strictly up to the snapshot. Line 11 fits the imputer on all rows before splitting, so medians are computed with test and validation rows included. Move it into the pipeline instead:

\n
Python
model = make_pipeline(SimpleImputer(strategy=\"median\"), StandardScaler(), LogisticRegression())
\n

The rest is fine: the scaler is already inside the pipeline, the split is stratified, and cross-validation runs on the training part only.

\n","checkLabels":{}},"words":710,"readMinutes":3}