{"id":"l-ml-4-3","courseId":"ml-crash-course","lang":"en","kind":"lesson","summary":"Tune hyperparameters with GridSearchCV and RandomizedSearchCV using pipeline step names, read cv_results_, and keep the final test score honest after the search.","html":"

Every model so far has had settings you picked by hand or left at defaults: C=1.0, max_depth=3, min_samples_leaf=5. Some of those choices matter a lot, some barely at all, and the only way to know is to try. Trying by hand means loops, bookkeeping and the temptation to peek at the test set. scikit-learn’s search tools do it properly, with cross-validation built in.

\n

Grid search

\n

GridSearchCV takes an estimator, a dictionary of parameter values and a cross-validation scheme. It cross-validates every combination, picks the one with the best mean score, and refits it on all the training data. Afterwards it behaves like the winning model.

\n

Parameters inside a pipeline are addressed by step name, two underscores, and parameter name. That’s why naming your steps pays off.

\n
Python
import pandas as pd\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n            \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n    churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n    stratify=churn[\"churned\"])\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n\npipe = Pipeline([(\"scale\", StandardScaler()), (\"model\", LogisticRegression())])\nsearch = GridSearchCV(pipe, {\"model__C\": [0.01, 0.03, 0.1, 0.3, 1, 3, 10]},\n                      cv=cv, scoring=\"roc_auc\")\nsearch.fit(X_train, y_train)\n\nresults = pd.DataFrame(search.cv_results_)\nprint(results[[\"param_model__C\", \"mean_test_score\", \"std_test_score\"]].round(3))\nprint(\"best:\", search.best_params_, \"CV AUC\", round(search.best_score_, 3))
\n

cv_results_ holds every combination’s fold scores; as a DataFrame it’s easy to scan. Here the scan is the finding: from C=0.03 to C=10 the mean AUC moves by less than half a point, far inside the fold-to-fold spread. Eight well-behaved features don’t give logistic regression room to overfit, so its penalty barely matters. Tuning can’t rescue a model that’s already at its ceiling; better features or a different model family might.

\n

Notice the values are spaced by factors of about three. For parameters that act multiplicatively (C, alpha, learning rates), search on a log scale; 0.01, 0.1, 1, 10 covers far more ground than 1, 2, 3, 4.

\n

A grid where it matters

\n

Trees are another story. Depth and minimum leaf size control how much a tree can memorise, and the right values depend on the data.

\n
Python
import pandas as pd\nfrom sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split\nfrom sklearn.tree import DecisionTreeClassifier\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n            \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n    churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n    stratify=churn[\"churned\"])\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n\ngrid = {\"max_depth\": [2, 3, 4, 5, None], \"min_samples_leaf\": [1, 5, 10, 20, 40]}\nsearch = GridSearchCV(DecisionTreeClassifier(random_state=42), grid, cv=cv, scoring=\"roc_auc\")\nsearch.fit(X_train, y_train)\n\ntop = pd.DataFrame(search.cv_results_).sort_values(\"rank_test_score\")\nprint(top[[\"param_max_depth\", \"param_min_samples_leaf\", \"mean_test_score\"]].head(5).round(3))\nprint(\"worst:\", round(top[\"mean_test_score\"].min(), 3))\nprint(\"best:\", search.best_params_, \"| test AUC\", round(search.score(X_test, y_test), 3))
\n

25 combinations × 5 folds = 125 fits, plus one final refit. The spread is wide: the worst trees (unlimited depth, single-customer leaves) score far below the best. search.score uses the same scoring as the search, so it reports test ROC AUC here.

\n

Random search

\n

Grids grow multiplicatively. Four parameters with six values each is 1,296 combinations, times five folds. RandomizedSearchCV samples a fixed number of combinations (n_iter) instead, and it accepts distributions as well as lists.

\n
\n Two squares with an important parameter on the horizontal axis and an unimportant one on the vertical. The grid places nine points in a 3 by 3 lattice, testing only three distinct values of the important parameter. Random search scatters nine points, testing nine distinct values.\n \n \n \n \n \n \n \n \n Grid: 3 values tried\n Random: 9 values tried\n important →\n
Grid search versus random search with the same nine trials
\n

When only one or two parameters really matter, which is the usual case, random search explores many more distinct values of them for the same budget.

\n
Python
import pandas as pd\nfrom scipy.stats import randint\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import RandomizedSearchCV, StratifiedKFold, train_test_split\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n            \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n    churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n    stratify=churn[\"churned\"])\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n\nspace = {\"max_features\": [1, 2, 3, 4, \"sqrt\"], \"min_samples_leaf\": randint(1, 30)}\nsearch = RandomizedSearchCV(RandomForestClassifier(n_estimators=60, random_state=42), space,\n                            n_iter=8, cv=cv, scoring=\"roc_auc\", random_state=42)\nsearch.fit(X_train, y_train)\nprint(\"best:\", search.best_params_)\nprint(\"CV AUC\", round(search.best_score_, 3), \"| test AUC\", round(search.score(X_test, y_test), 3))
\n

randint(1, 30) from SciPy draws integers from 1 to 29; loguniform(0.01, 10) is the usual choice for C or learning rates. Set random_state on the search so it samples the same candidates each run.

\n

Searching more than model settings

\n

Because the search tunes the whole pipeline, preprocessing choices are fair game too. A grid entry like "prep__sat__simpleimputer__strategy": ["mean", "median"] compares imputation strategies with the same honest folds. You can even pass a list of grids, one per model family, and let the search swap the final step: [{"model": [LogisticRegression()], "model__C": [0.1, 1]}, {"model": [RandomForestClassifier()], "model__min_samples_leaf": [5, 20]}]. That’s tidy, but it multiplies fits quickly, and it hides the comparison table you’d want to show a colleague. For a handful of candidate models, a plain loop of cross_validate calls with the same splitter is often clearer.

\n\n\n

Searches multiply the number of models you fit, and with them the number of chances for information to leak from validation data into training. Next lesson is a field guide to leakage: how it happens, how to spot it, and why it produces the best scores you’ll ever see.

\n","toc":[{"id":"sec-1","text":"Grid search","level":2},{"id":"sec-2","text":"A grid where it matters","level":2},{"id":"sec-3","text":"Random search","level":2},{"id":"sec-4","text":"Searching more than model settings","level":2}],"takeaways":["GridSearchCV tries every combination in a grid, cross-validating each; RandomizedSearchCV samples a fixed number of combinations from lists or distributions.","Inside a pipeline, parameters are addressed as step__parameter, for example model__C.","After the search, best_params_ holds the winner and best_estimator_ is that model refit on all the training data.","best_score_ is slightly optimistic because it’s the best of many tries; the test set gives the unbiased number.","Search on a log scale for parameters like C, alpha and learning_rate, and expect tuning to add points, not miracles."],"further":[{"title":"Tuning the hyper-parameters of an estimator","url":"https://scikit-learn.org/stable/modules/grid_search.html"},{"title":"GridSearchCV","url":"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html"},{"title":"RandomizedSearchCV","url":"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html"}],"quiz":[{"q":"

Your pipeline is Pipeline([("prep", preprocess), ("clf", LogisticRegression())]). Which grid tunes the regularisation strength?

","options":[{"text":"{"C": [0.1, 1, 10]}","why":"The pipeline has no parameter called C; the search would raise an invalid-parameter error."},{"text":"{"LogisticRegression__C": [0.1, 1, 10]}","why":"The prefix is the step’s name, not the class name."},{"text":"{"clf.C": [0.1, 1, 10]}","why":"scikit-learn uses a double underscore, not a dot, to reach into steps."},{"text":"{"clf__C": [0.1, 1, 10]}","why":"Correct. Step name, two underscores, parameter name."}],"answer":3},{"q":"

A grid has 5 values for max_depth and 5 for min_samples_leaf, with cv=5. How many models are fitted, including the final refit?

","options":[{"text":"126: 25 combinations × 5 folds, plus one refit on all the training data.","why":"Correct. With the default refit=True the winner is retrained once more at the end."},{"text":"25: one per combination.","why":"Each combination is cross-validated, so it’s fitted once per fold."},{"text":"10: five values plus five values.","why":"A grid tries every combination, not each value on its own."}],"answer":0},{"q":"

Why might RandomizedSearchCV with 20 iterations beat a 400-combination grid on the same budget of time?

","options":[{"text":"Random search uses a smarter optimiser that learns from previous trials.","why":"Plain random search doesn’t learn between trials; each sample is independent."},{"text":"Random search always finds the true optimum.","why":"It samples, so it can miss the optimum; the point is efficiency, not guarantees."},{"text":"When only a few parameters matter, random sampling tries many more distinct values of those parameters than a grid does.","why":"Correct. A grid wastes trials repeating the same values of the important parameter while varying unimportant ones."}],"answer":2},{"q":"

The search reports best_score_ = 0.813, and the refit model scores 0.805 on the test set. What’s going on?

","options":[{"text":"The test set must be leaking, since scores should match exactly.","why":"Scores differ on different data; a leak would push the test score up, not down."},{"text":"The best of many cross-validated scores is a little optimistic; the test score is the honest estimate.","why":"Correct. Picking the maximum of noisy scores favours lucky configurations, a winner’s curse."},{"text":"The refit on all training data made the model worse.","why":"More training data rarely hurts; the gap comes from how the winner was selected."}],"answer":1}],"exercise":{"kind":"python","files":["churn.csv"],"starter":"import pandas as pd\nfrom sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n\npipe = Pipeline([(\"scale\", StandardScaler()), (\"knn\", KNeighborsClassifier())])\n\n# 1. Grid: n_neighbors in [5, 15, 25, 35, 51] and weights in [\"uniform\", \"distance\"]\ngrid = {\"n_neighbors\": [5, 15, 25, 35, 51], \"weights\": [\"uniform\", \"distance\"]}\n\n# 2. Search with the provided cv, scoring ROC AUC, fitted on the training data\nsearch = None\n\n# 3. Results\nbest_params = None\nbest_cv_auc = None\ntest_auc = None # ROC AUC of the refit best model on the test set\n","solution":"import pandas as pd\nfrom sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\ncv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n\npipe = Pipeline([(\"scale\", StandardScaler()), (\"knn\", KNeighborsClassifier())])\n\ngrid = {\"knn__n_neighbors\": [5, 15, 25, 35, 51], \"knn__weights\": [\"uniform\", \"distance\"]}\n\nsearch = GridSearchCV(pipe, grid, cv=cv, scoring=\"roc_auc\")\nsearch.fit(X_train, y_train)\n\nbest_params = search.best_params_\nbest_cv_auc = search.best_score_\ntest_auc = search.score(X_test, y_test)\nprint(best_params, round(best_cv_auc, 3), round(test_auc, 3))\n","checks":[{"id":"grid-names","test":"set(grid) == {'knn__n_neighbors', 'knn__weights'}"},{"id":"searched","test":"search is not None and len(search.cv_results_['params']) == 10 and search.scoring == 'roc_auc' and search.cv is cv and hasattr(search, 'best_estimator_')"},{"id":"best","test":"best_params is not None and best_params == search.best_params_ and best_params['knn__n_neighbors'] in [15, 25, 35, 51] and abs(float(best_cv_auc) - float(search.best_score_)) < 1e-9 and 0.77 < float(best_cv_auc) < 0.82"},{"id":"test-auc","test":"test_auc is not None and abs(float(test_auc) - float(search.score(X_test, y_test))) < 1e-9 and 0.75 < float(test_auc) < 0.82"}],"title":"Tune k-nearest neighbours","prompt":"

Find the best number of neighbours for a scaled k-NN pipeline.

\n
    \n
  1. Fix grid so its keys address the pipeline’s knn step (the starter keys would raise an invalid-parameter error).
    2. \n
  2. search: a GridSearchCV over pipe with the provided cv and scoring="roc_auc", fitted on the training data.
    3. \n
  3. best_params, best_cv_auc: the winning settings and their mean cross-validated AUC. test_auc: the refit winner’s ROC AUC on the test set.
    4. \n
\n","hints":["Pipeline parameters are <step name>__<parameter>: here knn__n_neighbors and knn__weights.","GridSearchCV(pipe, grid, cv=cv, scoring="roc_auc").fit(X_train, y_train)","search.best_params_, search.best_score_, and search.score(X_test, y_test) (which uses the search’s scoring)."],"explanation":"

Five neighbours is too few: each prediction rests on a handful of customers and the AUC is about 0.75. Around 25–35 neighbours the ranking improves to about 0.79, and the two weighting schemes barely differ. The winner (35, uniform) scores 0.795 in cross-validation and 0.785 on the test set: slightly lower, as expected for the best of ten tries. Still a touch behind logistic regression, so the simpler model keeps its place.

\n","checkLabels":{"grid-names":"grid keys use the knn__ prefix","searched":"search tried all 10 combinations with ROC AUC","best":"best_params and best_cv_auc come from the search","test-auc":"test_auc is the refit winner’s test ROC AUC"}},"words":697,"readMinutes":3}