{"id":"l-ml-2-4","courseId":"ml-crash-course","lang":"en","kind":"lesson","summary":"Control overfitting by penalising large coefficients with Ridge and Lasso, scale features so the penalty is fair, and use Lasso’s zeros as automatic feature selection.","html":"

Last lesson, degree-2 polynomial features gave the diabetes model 65 columns, and the test score dropped from 0.49 to 0.42. You could throw the extra columns away. Regularisation offers something better: keep them all, but make the model pay for every unit of coefficient it uses. Columns that earn their keep get weight; the rest get shrunk.

\n

A penalty on the loss

\n

Ordinary least squares minimises the squared error and nothing else. Regularised regression minimises the squared error plus a penalty that grows with the size of the coefficients:

\n\n

alpha sets the exchange rate. At 0 you’re back to least squares. As it grows, a big coefficient must buy a big reduction in error to be worth it. Coefficients that only fit noise can’t, so they shrink. Squared values make Ridge shrink everything smoothly; absolute values make Lasso push weak coefficients all the way to zero.

\n

Scale first, then penalise

\n

The penalty adds up coefficients as if they were comparable. They only are if the features share a scale. Measure spend in cents instead of dollars and its coefficient becomes 100 times smaller, so the penalty barely notices it. Always put a StandardScaler in front of a penalised model.

\n
Python
from sklearn.datasets import load_diabetes\nfrom sklearn.linear_model import LinearRegression, Ridge, Lasso\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import PolynomialFeatures, StandardScaler\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n\ndef degree2(model):\n    return make_pipeline(PolynomialFeatures(degree=2, include_bias=False), StandardScaler(), model)\n\nprint(\"no penalty:\", round(degree2(LinearRegression()).fit(X_train, y_train).score(X_test, y_test), 3))\nfor alpha in [1, 10, 100, 1000]:\n    m = degree2(Ridge(alpha=alpha)).fit(X_train, y_train)\n    print(f\"Ridge alpha={alpha:<5} train {m.score(X_train, y_train):.3f}  test {m.score(X_test, y_test):.3f}\")
\n

As alpha rises the training score falls steadily (the model is allowed less freedom) while the test score climbs from 0.42 to 0.50 at alpha=100, better than the plain ten-feature model. Push to 1,000 and both scores drop: now the model underfits. Regularisation turns the overfitting/underfitting trade-off into a dial.

\n

Lasso picks features for you

\n

Run the same experiment with Lasso and count the coefficients that end up at exactly zero.

\n
Python
from sklearn.datasets import load_diabetes\nfrom sklearn.linear_model import Lasso\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import PolynomialFeatures, StandardScaler\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n\nfor alpha in [0.1, 1, 2, 5]:\n    m = make_pipeline(PolynomialFeatures(degree=2, include_bias=False), StandardScaler(),\n                      Lasso(alpha=alpha)).fit(X_train, y_train)\n    zeros = (m[-1].coef_ == 0).sum()\n    print(f\"Lasso alpha={alpha:<4} test R2 {m.score(X_test, y_test):.3f}  zero coefficients: {zeros} of 65\")\n\nnames = m[0].get_feature_names_out()\nkept = [n for n, c in zip(names, m[-1].coef_) if c != 0]\nprint(\"kept at alpha=5:\", kept)
\n

At alpha=2 Lasso reaches a test R² of 0.53, the best regression score in this section, while ignoring 39 of the 65 columns. At alpha=5 it keeps 11 columns: BMI, blood pressure and s5 from the earlier models, plus a few interactions such as bmi bp. m[-1] reaches the last step of the pipeline, and m[0].get_feature_names_out() names the generated columns.

\n

Taming correlated coefficients

\n

Remember s1 at -918 and s2 at +508 in the plain regression, two correlated blood measurements pulling against each other? With standardised features the same story shows up on a smaller scale, and the penalty settles it.

\n
\n Coefficients for s1 and s2 on standardised features. Least squares: s1 minus 44, s2 plus 25. Ridge with alpha 10: s1 minus 13, s2 near zero. Lasso with alpha 1: s1 minus 8, s2 exactly zero.\n \n 0\n Least squares\n s1 −44\n s2 +25\n Ridge, alpha=10\n s1 −13\n s2 +0.5\n Lasso, alpha=1\n s1 −8\n s2 0 (dropped)\n
What happens to two correlated coefficients
\n

Ridge spreads weight sensibly between correlated features instead of letting them cancel each other out; Lasso tends to keep one and drop the other. Either way the coefficients become stable enough that a small change in the training data won’t flip them.

\n\n

Ridge or Lasso?

\n

Both cure the same disease, so the choice comes down to what you want from the final model. Ridge usually wins when many features each carry a little signal, which is common with measurements that overlap, like the six blood readings here. Lasso wins when a few features matter and the rest are noise, and it hands you a shorter shopping list of inputs, which matters when every feature has to be collected, cleaned and monitored in production. When features come in correlated groups, Lasso’s habit of keeping one member, somewhat arbitrarily, can be unstable from one training run to the next. ElasticNet mixes the two penalties for exactly that case. On a dataset this size, the honest answer is to try both on validation data and keep whichever wins, then prefer the simpler one when they tie.

\n\n

That closes regression. The same penalty idea comes straight back in the next section: LogisticRegression is regularised by default, with a parameter C that works as the inverse of alpha. Section 3 switches the target back to churn, yes or no.

\n","toc":[{"id":"sec-1","text":"A penalty on the loss","level":2},{"id":"sec-2","text":"Scale first, then penalise","level":2},{"id":"sec-3","text":"Lasso picks features for you","level":2},{"id":"sec-4","text":"Taming correlated coefficients","level":2},{"id":"sec-5","text":"Ridge or Lasso?","level":2}],"takeaways":["Regularisation adds a penalty on coefficient size to the loss, so the model trades a little training fit for simpler, more stable coefficients.","Ridge penalises squared coefficients and shrinks them all; Lasso penalises absolute values and sets some exactly to zero.","alpha is the strength of the penalty: 0 is plain least squares, very large values flatten the model into underfitting.","Scale features before a penalised model, or the penalty punishes features for their units rather than their usefulness.","Choose alpha on validation data or with cross-validation, the same way you choose any hyperparameter."],"further":[{"title":"Ridge regression and classification","url":"https://scikit-learn.org/stable/modules/linear_model.html#ridge-regression-and-classification"},{"title":"Lasso","url":"https://scikit-learn.org/stable/modules/linear_model.html#lasso"},{"title":"Ridge","url":"https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html"}],"quiz":[{"q":"

You raise Ridge’s alpha from 1 to 100,000. What happens to the model?

","options":[{"text":"Coefficients shrink toward zero and predictions approach the mean of the target; the model underfits.","why":"Correct. A huge penalty makes any non-zero coefficient too expensive, so the model falls back to roughly the intercept."},{"text":"Coefficients grow so the model fits the training data more closely.","why":"A larger alpha makes large coefficients more expensive, not cheaper."},{"text":"Most coefficients become exactly zero, leaving a few features.","why":"That’s Lasso’s behaviour. Ridge shrinks coefficients smoothly but almost never to exactly zero."}],"answer":0},{"q":"

Why put StandardScaler before Lasso in the pipeline?

","options":[{"text":"Lasso can’t handle negative numbers without scaling.","why":"Lasso works on any real numbers. The issue is fairness of the penalty."},{"text":"The penalty treats all coefficients alike, so features must share a scale for it to judge them on usefulness rather than units.","why":"Correct. A feature measured in large units needs a tiny coefficient and escapes the penalty; one in small units gets punished."},{"text":"Scaling makes Lasso converge to the same answer as ordinary least squares.","why":"Scaling doesn’t remove the penalty; with alpha > 0 Lasso still differs from least squares."}],"answer":1},{"q":"

You need a churn model that uses as few of 60 candidate features as possible. Which is the natural first choice?

","options":[{"text":"Ridge, because it shrinks all 60 coefficients.","why":"Ridge keeps every feature with a small weight, so you’d still need all 60 inputs at prediction time."},{"text":"Plain linear regression, then drop the features with the smallest coefficients.","why":"Unpenalised coefficients of correlated features are unstable, so “smallest” is unreliable."},{"text":"An L1 (Lasso-style) penalty, because it drives unhelpful coefficients to exactly zero.","why":"Correct. The zeros are a built-in feature selection; you can then inspect which features survived."}],"answer":2}],"exercise":{"kind":"python","starter":"from sklearn.datasets import load_diabetes\nfrom sklearn.linear_model import Lasso, Ridge\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import PolynomialFeatures, StandardScaler\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n\n# 1. Pipeline: degree-2 PolynomialFeatures (include_bias=False) -> StandardScaler -> Lasso(alpha=1)\nlasso_model = make_pipeline(PolynomialFeatures(degree=2, include_bias=False), Lasso(alpha=1))\nlasso_model.fit(X_train, y_train)\n\n# 2. Test R2 and the number of coefficients that are exactly zero\nlasso_r2 = None\nn_zero = None\n\n# 3. Names of the generated columns Lasso kept (non-zero coefficient), as a list of strings\nkept = None\n","solution":"from sklearn.datasets import load_diabetes\nfrom sklearn.linear_model import Lasso, Ridge\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import PolynomialFeatures, StandardScaler\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n\nlasso_model = make_pipeline(PolynomialFeatures(degree=2, include_bias=False), StandardScaler(), Lasso(alpha=1))\nlasso_model.fit(X_train, y_train)\n\nlasso_r2 = lasso_model.score(X_test, y_test)\ncoefs = lasso_model[-1].coef_\nn_zero = int((coefs == 0).sum())\n\nnames = lasso_model[0].get_feature_names_out()\nkept = [name for name, c in zip(names, coefs) if c != 0]\nprint(round(lasso_r2, 3), n_zero, len(kept))\n","checks":[{"id":"scaled","test":"[type(s).__name__ for s in lasso_model] == ['PolynomialFeatures', 'StandardScaler', 'Lasso'] and lasso_model[-1].alpha == 1"},{"id":"r2","test":"lasso_r2 is not None and abs(float(lasso_r2) - 0.515) < 0.005"},{"id":"zeros","test":"n_zero is not None and int(n_zero) == int((lasso_model[-1].coef_ == 0).sum()) == 30"},{"id":"kept","test":"kept is not None and len(kept) == 35 and 'bmi' in list(kept) and all(isinstance(k, str) for k in kept)"}],"title":"Let Lasso choose the features","prompt":"

The starter pipeline forgets something important. Fix it, then inspect what Lasso did.

\n
    \n
  1. lasso_model: degree-2 PolynomialFeatures(include_bias=False), then StandardScaler, then Lasso(alpha=1), fitted on the training data.
    2. \n
  2. lasso_r2: its test R². n_zero: how many of its 65 coefficients are exactly zero.
    3. \n
  3. kept: a list with the names of the generated columns whose coefficient is not zero.
    4. \n
\n","hints":["Without a scaler the penalty judges features by their units. Add StandardScaler() between the two existing steps.","lasso_model[-1].coef_ gives the Lasso coefficients; (coef == 0).sum() counts zeros.","lasso_model[0].get_feature_names_out() lists the 65 generated column names in the same order as the coefficients."],"explanation":"

With scaling in place, Lasso at alpha=1 reaches a test R² of about 0.515 and sets 30 of 65 coefficients to zero, keeping 35 columns. Without the scaler, the diabetes features’ tiny scale means the penalty overwhelms most coefficients for the wrong reason: their units, not their usefulness. Pairing names with coefficients is how you report which inputs the final model actually needs.

\n","checkLabels":{"scaled":"The pipeline is PolynomialFeatures → StandardScaler → Lasso(alpha=1)","r2":"lasso_r2 is the test R²","zeros":"n_zero counts the zero coefficients","kept":"kept lists the names of the non-zero columns"}},"words":749,"readMinutes":4}