Hyperparameter Search

Every model so far has had settings you picked by hand or left at defaults: C=1.0, max_depth=3, min_samples_leaf=5. Some of those choices matter a lot, some barely at all, and the only way to know is to try. Trying by hand means loops, bookkeeping and the temptation to peek at the test set. scikit-learn’s search tools do it properly, with cross-validation built in.

Grid search

GridSearchCV takes an estimator, a dictionary of parameter values and a cross-validation scheme. It cross-validates every combination, picks the one with the best mean score, and refits it on all the training data. Afterwards it behaves like the winning model.

Parameters inside a pipeline are addressed by step name, two underscores, and parameter name. That’s why naming your steps pays off.

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

pipe = Pipeline([("scale", StandardScaler()), ("model", LogisticRegression())])
search = GridSearchCV(pipe, {"model__C": [0.01, 0.03, 0.1, 0.3, 1, 3, 10]},
                      cv=cv, scoring="roc_auc")
search.fit(X_train, y_train)

results = pd.DataFrame(search.cv_results_)
print(results[["param_model__C", "mean_test_score", "std_test_score"]].round(3))
print("best:", search.best_params_, "CV AUC", round(search.best_score_, 3))

cv_results_ holds every combination’s fold scores; as a DataFrame it’s easy to scan. Here the scan is the finding: from C=0.03 to C=10 the mean AUC moves by less than half a point, far inside the fold-to-fold spread. Eight well-behaved features don’t give logistic regression room to overfit, so its penalty barely matters. Tuning can’t rescue a model that’s already at its ceiling; better features or a different model family might.

Notice the values are spaced by factors of about three. For parameters that act multiplicatively (C, alpha, learning rates), search on a log scale; 0.01, 0.1, 1, 10 covers far more ground than 1, 2, 3, 4.

A grid where it matters

Trees are another story. Depth and minimum leaf size control how much a tree can memorise, and the right values depend on the data.

import pandas as pd
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.tree import DecisionTreeClassifier

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

grid = {"max_depth": [2, 3, 4, 5, None], "min_samples_leaf": [1, 5, 10, 20, 40]}
search = GridSearchCV(DecisionTreeClassifier(random_state=42), grid, cv=cv, scoring="roc_auc")
search.fit(X_train, y_train)

top = pd.DataFrame(search.cv_results_).sort_values("rank_test_score")
print(top[["param_max_depth", "param_min_samples_leaf", "mean_test_score"]].head(5).round(3))
print("worst:", round(top["mean_test_score"].min(), 3))
print("best:", search.best_params_, "| test AUC", round(search.score(X_test, y_test), 3))

25 combinations × 5 folds = 125 fits, plus one final refit. The spread is wide: the worst trees (unlimited depth, single-customer leaves) score far below the best. search.score uses the same scoring as the search, so it reports test ROC AUC here.

Random search

Grids grow multiplicatively. Four parameters with six values each is 1,296 combinations, times five folds. RandomizedSearchCV samples a fixed number of combinations (n_iter) instead, and it accepts distributions as well as lists.

When only one or two parameters really matter, which is the usual case, random search explores many more distinct values of them for the same budget.

import pandas as pd
from scipy.stats import randint
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold, train_test_split

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

space = {"max_features": [1, 2, 3, 4, "sqrt"], "min_samples_leaf": randint(1, 30)}
search = RandomizedSearchCV(RandomForestClassifier(n_estimators=60, random_state=42), space,
                            n_iter=8, cv=cv, scoring="roc_auc", random_state=42)
search.fit(X_train, y_train)
print("best:", search.best_params_)
print("CV AUC", round(search.best_score_, 3), "| test AUC", round(search.score(X_test, y_test), 3))

randint(1, 30) from SciPy draws integers from 1 to 29; loguniform(0.01, 10) is the usual choice for C or learning rates. Set random_state on the search so it samples the same candidates each run.

Searching more than model settings

Because the search tunes the whole pipeline, preprocessing choices are fair game too. A grid entry like "prep__sat__simpleimputer__strategy": ["mean", "median"] compares imputation strategies with the same honest folds. You can even pass a list of grids, one per model family, and let the search swap the final step: [{"model": [LogisticRegression()], "model__C": [0.1, 1]}, {"model": [RandomForestClassifier()], "model__min_samples_leaf": [5, 20]}]. That’s tidy, but it multiplies fits quickly, and it hides the comparison table you’d want to show a colleague. For a handful of candidate models, a plain loop of cross_validate calls with the same splitter is often clearer.

Searches multiply the number of models you fit, and with them the number of chances for information to leak from validation data into training. Next lesson is a field guide to leakage: how it happens, how to spot it, and why it produces the best scores you’ll ever see.