{"id":"l-ml-4-5","courseId":"ml-crash-course","lang":"en","kind":"lesson","summary":"Turn domain knowledge into rates, ratios and log transforms, package them in a FunctionTransformer inside the pipeline, and keep only the features that beat cross-validation noise.","html":"

Models only see the columns you give them. A support lead looking at a customer doesn’t think “4 orders and 600 days of tenure”; they think “orders about once every five months”. That rate, not either raw number, is the useful idea. Feature engineering turns that kind of domain knowledge into columns, and on tabular data it’s often worth more than any amount of tuning.

\n

Ideas that come from the business

\n

Good churn features usually fall into a few families:

\n\n

Each of these must be computable from information available at the snapshot date. A rate built from tenure and orders is fine; anything that peeks into the outcome window is the leak from last lesson.

\n

Features inside the pipeline

\n

Write the feature code once, as a function from DataFrame to DataFrame, and wrap it in a FunctionTransformer. Inside a pipeline it runs in cross-validation, in the final fit and in production, always the same way.

\n
Python
import numpy as np\nimport pandas as pd\nfrom sklearn.preprocessing import FunctionTransformer\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n            \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\n\ndef add_features(df):\n    df = df.copy()\n    months = df[\"tenure_days\"] / 30.4\n    df[\"orders_per_month\"] = df[\"orders\"] / months\n    df[\"typical_gap\"] = df[\"tenure_days\"] / df[\"orders\"]\n    df[\"overdue\"] = df[\"days_since_last_order\"] / df[\"typical_gap\"]\n    df[\"log_spent\"] = np.log1p(df[\"total_spent\"])\n    return df\n\nengineered = FunctionTransformer(add_features).fit_transform(churn[features])\nprint(engineered[[\"orders\", \"tenure_days\", \"orders_per_month\", \"overdue\", \"log_spent\"]].head().round(2))\nprint(\"skew of total_spent:\", round(churn[\"total_spent\"].skew(), 2),\n      \"| after log1p:\", round(np.log1p(churn[\"total_spent\"]).skew(), 2))
\n

df.copy() matters: the function shouldn’t modify the caller’s DataFrame. Since every customer has at least one order and positive tenure, the divisions are safe here; in your own data, guard against zeros.

\n

The function is stateless: each row’s new values depend only on that row. Stateless features can’t leak across folds. A feature like “spend compared with the average customer” is different, because the average is learned from data; it needs a proper transformer with fit, or it will quietly use validation rows.

\n

Do they help? Ask cross-validation

\n
Python
import numpy as np\nimport pandas as pd\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import FunctionTransformer, StandardScaler\nfrom sklearn.tree import DecisionTreeClassifier\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n            \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n    churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n    stratify=churn[\"churned\"])\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n\ndef add_features(df):\n    df = df.copy()\n    df[\"orders_per_month\"] = df[\"orders\"] / (df[\"tenure_days\"] / 30.4)\n    df[\"typical_gap\"] = df[\"tenure_days\"] / df[\"orders\"]\n    df[\"overdue\"] = df[\"days_since_last_order\"] / df[\"typical_gap\"]\n    df[\"log_spent\"] = np.log1p(df[\"total_spent\"])\n    return df\n\nmodels = {\"tree\": DecisionTreeClassifier(min_samples_leaf=20, random_state=42),\n          \"logistic\": make_pipeline(StandardScaler(), LogisticRegression())}\nfor name, model in models.items():\n    plain = cross_val_score(model, X_train, y_train, cv=cv, scoring=\"roc_auc\")\n    richer = cross_val_score(make_pipeline(FunctionTransformer(add_features), model),\n                             X_train, y_train, cv=cv, scoring=\"roc_auc\")\n    print(f\"{name:8} plain {plain.mean():.3f} ± {plain.std():.3f}   \"\n          f\"engineered {richer.mean():.3f} ± {richer.std():.3f}\")
\n

The single tree improves from 0.716 to 0.759, and its folds become steadier. Look inside and you’ll see why: orders_per_month becomes the root question. A tree can only compare one column with a threshold, so “few orders for this much tenure” would take a staircase of splits to imitate. Handed the ratio, it needs one.

\n

Logistic regression moves from 0.807 to 0.811, well inside a spread of ±0.04. That’s no change. A linear model can already weigh orders against tenure_days, and recency, the dominant signal, was there from the start. Different model families need different features: ratios and interactions for trees, straightening transforms for linear models.

\n\n

Closing the section: the champion

\n

Section 4 tested ensembles, extra columns, tuning and engineered features with honest cross-validation. Most of them didn’t beat the simple model. So the champion is still the scaled logistic regression on eight numeric columns, and now it’s time to spend the test set:

\n
Python
import pandas as pd\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n            \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n    churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n    stratify=churn[\"churned\"])\n\nchampion = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)\nprint(\"test ROC AUC:\", round(roc_auc_score(y_test, champion.predict_proba(X_test)[:, 1]), 3))\nprint(\"test accuracy:\", champion.score(X_test, y_test))
\n

Test ROC AUC 0.81 against a cross-validated 0.81: no surprise, which is exactly what a leak-free process looks like. That agreement is what section 4 bought you. Not a higher number, but a number you can stand behind in a meeting.

\n\n

Section 5 steps beyond supervised tables: finding structure without labels, a first look at neural networks, where LLMs fit in, and what it takes to put the champion in front of real customers responsibly.

\n","toc":[{"id":"sec-1","text":"Ideas that come from the business","level":2},{"id":"sec-2","text":"Features inside the pipeline","level":2},{"id":"sec-3","text":"Do they help? Ask cross-validation","level":2},{"id":"sec-4","text":"Closing the section: the champion","level":2}],"takeaways":["Good features express what an expert would look at: rates per unit of time, ratios between related columns, and flags for meaningful states.","Trees split one column at a time, so they benefit most from ratio features that combine columns; linear models benefit from transforms like logs that straighten relationships.","Put feature code in a FunctionTransformer inside the pipeline so training, cross-validation and production compute features identically.","A new feature earns its place only if the cross-validated gain is larger than the fold-to-fold spread.","After choosing, refit the champion on all training data and score it once on the test set."],"further":[{"title":"FunctionTransformer","url":"https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.FunctionTransformer.html"},{"title":"Preprocessing data: custom transformers","url":"https://scikit-learn.org/stable/modules/preprocessing.html#custom-transformers"}],"quiz":[{"q":"

Adding orders_per_month lifts a single tree’s CV AUC from 0.72 to 0.76 but leaves logistic regression unchanged at 0.81. What’s the most likely reason?

","options":[{"text":"The tree couldn’t form “orders relative to tenure” from its one-column-at-a-time splits, while the linear model could already approximate it by weighting both columns.","why":"Correct. A tree needs many splits to imitate a ratio; handing it the ratio directly gives it one clean question."},{"text":"Logistic regression ignores new features after the first eight.","why":"It uses every column it receives; the new one simply adds little beyond what it already had."},{"text":"The tree is overfitting the new feature.","why":"Overfitting would hurt validation scores, not improve them."},{"text":"The ratio leaks the target.","why":"Orders and tenure are both known at the snapshot date, so their ratio is safe."}],"answer":0},{"q":"

Why compute features inside a FunctionTransformer in the pipeline instead of adding columns to the DataFrame up front?

","options":[{"text":"FunctionTransformer makes features faster to compute.","why":"Speed is the same; it’s the same pandas code either way."},{"text":"The production service then computes exactly the same features from raw input, with no separate code to keep in sync.","why":"Correct. One object holds the feature logic and the model, so they can’t drift apart."},{"text":"Features added to a DataFrame can’t be used by scikit-learn models.","why":"Any numeric DataFrame column can be used; the benefit is consistency."}],"answer":1},{"q":"

total_spent is strongly right-skewed (a few customers spent thousands). Which transform usually helps a linear model?

","options":[{"text":"Squaring it, to spread the large values further.","why":"Squaring makes the skew worse and gives the extreme customers even more influence."},{"text":"One-hot encoding it.","why":"That treats every distinct amount as a separate category, which throws away order and explodes the column count."},{"text":"np.log1p(total_spent), which compresses large values so a $100 difference matters more at $50 than at $3,000.","why":"Correct. Log transforms tame skew and often make relationships closer to linear. Trees don’t need them."}],"answer":2},{"q":"

A new feature raises mean CV AUC from 0.807 to 0.811, with fold standard deviations around 0.035. What should you do?

","options":[{"text":"Keep it; any improvement is worth having.","why":"Every feature has a running cost (computing, monitoring, explaining), and a gain this size is indistinguishable from noise."},{"text":"Report it as a 0.4-point improvement.","why":"That overstates evidence that isn’t there."},{"text":"Tune the model again until the gap is bigger.","why":"Tuning to widen a noisy gap is overfitting your validation folds."},{"text":"Treat it as no change; keep the simpler feature set unless the feature has another reason to exist.","why":"Correct. A gain well inside the spread is noise; the default is to leave the feature out."}],"answer":3}],"exercise":{"kind":"python","files":["churn.csv"],"starter":"import numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import FunctionTransformer\nfrom sklearn.tree import DecisionTreeClassifier\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\ntree = DecisionTreeClassifier(min_samples_leaf=20, random_state=42)\n\n# 1. Return a COPY of df with two new columns:\n# orders_per_month = orders / (tenure_days / 30.4)\n# tickets_per_order = support_tickets / orders\ndef add_features(df):\n return df\n\n# 2. Pipeline: FunctionTransformer(add_features), then `tree`\npipe = make_pipeline(tree)\n\n# 3. Mean CV ROC AUC on the training data, for the plain tree and for your pipeline\nplain_auc = None\nengineered_auc = None\n","solution":"import numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import FunctionTransformer\nfrom sklearn.tree import DecisionTreeClassifier\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\ntree = DecisionTreeClassifier(min_samples_leaf=20, random_state=42)\n\ndef add_features(df):\n df = df.copy()\n df[\"orders_per_month\"] = df[\"orders\"] / (df[\"tenure_days\"] / 30.4)\n df[\"tickets_per_order\"] = df[\"support_tickets\"] / df[\"orders\"]\n return df\n\npipe = make_pipeline(FunctionTransformer(add_features), tree)\n\nplain_auc = cross_val_score(tree, X_train, y_train, cv=cv, scoring=\"roc_auc\").mean()\nengineered_auc = cross_val_score(pipe, X_train, y_train, cv=cv, scoring=\"roc_auc\").mean()\nprint(round(plain_auc, 3), round(engineered_auc, 3))\n","checks":[{"id":"new-columns","test":"list(add_features(X_train.head(3)).columns) == features + ['orders_per_month', 'tickets_per_order'] or set(add_features(X_train.head(3)).columns) == set(features) | {'orders_per_month', 'tickets_per_order'}"},{"id":"values","test":"abs(float(add_features(X_train)['orders_per_month'].iloc[0]) - float(X_train['orders'].iloc[0] / (X_train['tenure_days'].iloc[0] / 30.4))) < 1e-9 and abs(float(add_features(X_train)['tickets_per_order'].iloc[0]) - float(X_train['support_tickets'].iloc[0] / X_train['orders'].iloc[0])) < 1e-9"},{"id":"no-mutation","test":"list(X_train.columns) == features"},{"id":"pipeline","test":"type(pipe[0]).__name__ == 'FunctionTransformer' and type(pipe[-1]).__name__ == 'DecisionTreeClassifier'"},{"id":"scores","test":"plain_auc is not None and engineered_auc is not None and abs(float(plain_auc) - 0.716) < 0.002 and float(engineered_auc) > float(plain_auc) + 0.02"}],"title":"Give the tree a rate to split on","prompt":"

Help a single decision tree with two engineered features.

\n
    \n
  1. Complete add_features(df) so it returns a copy of df with orders_per_month (orders / (tenure_days / 30.4)) and tickets_per_order (support_tickets / orders) added. Don’t modify the DataFrame passed in.
    2. \n
  2. pipe: FunctionTransformer(add_features) followed by tree.
    3. \n
  3. plain_auc and engineered_auc: mean cross-validated ROC AUC on the training data (with the provided cv) for tree alone and for pipe.
    4. \n
\n","hints":["Start the function with df = df.copy() so the caller’s data is untouched.","make_pipeline(FunctionTransformer(add_features), tree)","cross_val_score(model, X_train, y_train, cv=cv, scoring="roc_auc").mean() for each."],"explanation":"

The plain tree averages about 0.72; with the two rates it climbs by several points, mostly thanks to orders_per_month, which becomes the tree’s first question. Because the features live in the pipeline, cross-validation and any later prediction compute them the same way, and since each value depends only on its own row, nothing leaks between folds. Run the same comparison with logistic regression and you’d see almost no change: different models need different help.

\n","checkLabels":{"new-columns":"add_features adds both new columns","values":"The new columns hold the right values","no-mutation":"X_train itself is not modified","pipeline":"pipe is FunctionTransformer → DecisionTreeClassifier","scores":"Both CV scores are computed, and the engineered tree beats the plain one"}},"words":710,"readMinutes":3}