Regularisation with Ridge and Lasso

Last lesson, degree-2 polynomial features gave the diabetes model 65 columns, and the test score dropped from 0.49 to 0.42. You could throw the extra columns away. Regularisation offers something better: keep them all, but make the model pay for every unit of coefficient it uses. Columns that earn their keep get weight; the rest get shrunk.

A penalty on the loss

Ordinary least squares minimises the squared error and nothing else. Regularised regression minimises the squared error plus a penalty that grows with the size of the coefficients:

• Ridge adds alpha * sum(coef ** 2), the L2 penalty.
• Lasso adds alpha * sum(abs(coef)), the L1 penalty (scikit-learn also scales the error term slightly differently, which is why good alpha values for the two models differ).

alpha sets the exchange rate. At 0 you’re back to least squares. As it grows, a big coefficient must buy a big reduction in error to be worth it. Coefficients that only fit noise can’t, so they shrink. Squared values make Ridge shrink everything smoothly; absolute values make Lasso push weak coefficients all the way to zero.

Scale first, then penalise

The penalty adds up coefficients as if they were comparable. They only are if the features share a scale. Measure spend in cents instead of dollars and its coefficient becomes 100 times smaller, so the penalty barely notices it. Always put a StandardScaler in front of a penalised model.

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

def degree2(model):
    return make_pipeline(PolynomialFeatures(degree=2, include_bias=False), StandardScaler(), model)

print("no penalty:", round(degree2(LinearRegression()).fit(X_train, y_train).score(X_test, y_test), 3))
for alpha in [1, 10, 100, 1000]:
    m = degree2(Ridge(alpha=alpha)).fit(X_train, y_train)
    print(f"Ridge alpha={alpha:<5} train {m.score(X_train, y_train):.3f}  test {m.score(X_test, y_test):.3f}")

As alpha rises the training score falls steadily (the model is allowed less freedom) while the test score climbs from 0.42 to 0.50 at alpha=100, better than the plain ten-feature model. Push to 1,000 and both scores drop: now the model underfits. Regularisation turns the overfitting/underfitting trade-off into a dial.

Lasso picks features for you

Run the same experiment with Lasso and count the coefficients that end up at exactly zero.

from sklearn.datasets import load_diabetes
from sklearn.linear_model import Lasso
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

for alpha in [0.1, 1, 2, 5]:
    m = make_pipeline(PolynomialFeatures(degree=2, include_bias=False), StandardScaler(),
                      Lasso(alpha=alpha)).fit(X_train, y_train)
    zeros = (m[-1].coef_ == 0).sum()
    print(f"Lasso alpha={alpha:<4} test R2 {m.score(X_test, y_test):.3f}  zero coefficients: {zeros} of 65")

names = m[0].get_feature_names_out()
kept = [n for n, c in zip(names, m[-1].coef_) if c != 0]
print("kept at alpha=5:", kept)

At alpha=2 Lasso reaches a test R² of 0.53, the best regression score in this section, while ignoring 39 of the 65 columns. At alpha=5 it keeps 11 columns: BMI, blood pressure and s5 from the earlier models, plus a few interactions such as bmi bp. m[-1] reaches the last step of the pipeline, and m[0].get_feature_names_out() names the generated columns.

Taming correlated coefficients

Remember s1 at -918 and s2 at +508 in the plain regression, two correlated blood measurements pulling against each other? With standardised features the same story shows up on a smaller scale, and the penalty settles it.

Ridge spreads weight sensibly between correlated features instead of letting them cancel each other out; Lasso tends to keep one and drop the other. Either way the coefficients become stable enough that a small change in the training data won’t flip them.

Ridge or Lasso?

Both cure the same disease, so the choice comes down to what you want from the final model. Ridge usually wins when many features each carry a little signal, which is common with measurements that overlap, like the six blood readings here. Lasso wins when a few features matter and the rest are noise, and it hands you a shorter shopping list of inputs, which matters when every feature has to be collected, cleaned and monitored in production. When features come in correlated groups, Lasso’s habit of keeping one member, somewhat arbitrarily, can be unstable from one training run to the next. ElasticNet mixes the two penalties for exactly that case. On a dataset this size, the honest answer is to try both on validation data and keep whichever wins, then prefer the simpler one when they tie.

That closes regression. The same penalty idea comes straight back in the next section: LogisticRegression is regularised by default, with a parameter C that works as the inverse of alpha. Section 3 switches the target back to churn, yes or no.