Polynomial Features and Overfitting

A straight line is a strong assumption. If disease progression rises faster at high BMI than at low BMI, no straight line can capture that, however long gradient descent runs. The fix sounds harmless: let the model bend. This lesson shows how, and how quickly bending turns into memorising.

Bending a linear model

The trick is to keep linear regression and change the inputs. If you add a column bmi², a “linear” model in bmi and bmi² traces a parabola in bmi. PolynomialFeatures generates these columns: every power up to the chosen degree, plus every product of features (the interactions).

from sklearn.datasets import load_diabetes
from sklearn.preprocessing import PolynomialFeatures

X, y = load_diabetes(return_X_y=True, as_frame=True)
poly = PolynomialFeatures(degree=2, include_bias=False).fit(X[["bmi", "bp"]])
print(poly.get_feature_names_out())

for degree in [1, 2, 3]:
    n = PolynomialFeatures(degree=degree, include_bias=False).fit(X).n_output_features_
    print(f"degree {degree}: {n} columns from 10 features")

Two features become five at degree 2. Ten features become 65 at degree 2 and 285 at degree 3. include_bias=False drops the constant column, because LinearRegression already fits an intercept.

To chain the expansion and the regression, use make_pipeline. It creates one estimator that runs each step in order; fit fits each step on the training data and predict sends new data through the same steps. You’ll build richer pipelines in section 4.

The training score keeps climbing

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

for degree in [1, 2, 3]:
    model = make_pipeline(PolynomialFeatures(degree=degree, include_bias=False),
                          LinearRegression()).fit(X_train, y_train)
    print(f"degree {degree}: train R2 {model.score(X_train, y_train):6.3f}  "
          f"test R2 {model.score(X_test, y_test):7.3f}  "
          f"test MAE {mean_absolute_error(y_test, model.predict(X_test)):6.1f}")

Read the three rows slowly. Degree 2 improves the training score and worsens the test score. Degree 3 is a disaster dressed as a success: training R² of 0.90, test R² of -8.5, and a typical test miss of 151 points, worse than predicting the average for everyone. With 285 columns and only 331 training patients, the model has nearly enough knobs to pass through every training point. It has learned the noise.

Underfitting and overfitting

Those three rows map to a picture you’ll see in every ML course, because it’s the central trade-off of the field.

• Underfitting (high bias): the model is too simple for the pattern. Training and test scores are both poor and close together. More flexibility helps.
• Overfitting (high variance): the model is flexible enough to chase noise. Training score is high, test score much lower. Less flexibility, more data or regularisation helps.

You diagnose which side you’re on by comparing the two scores, never by looking at the training score alone. A small gap with poor scores says “add capacity”; a big gap says “take it away”.

The amount of data shifts the balance. With 331 patients, 285 columns is reckless; with 300,000 patients, the same degree-3 model would have far less room to memorise, because noise in individual rows averages out across so many examples. That’s why “collect more data” is a legitimate cure for overfitting and a useless one for underfitting: a model that can’t express the pattern doesn’t get better at it by seeing more of it. When you’re unsure which side you’re on, a learning curve (training and validation scores plotted against the number of training rows) settles it: two curves that meet low need a richer model, two that stay apart need more data or a simpler model.

Choosing the degree honestly

Here’s that recipe with three features. The validation split comes from X_train; the test set isn’t touched until the very last line.

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures

X, y = load_diabetes(return_X_y=True, as_frame=True)
X = X[["bmi", "bp", "s5"]]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
X_fit, X_val, y_fit, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=0)

def poly_model(degree):
    return make_pipeline(PolynomialFeatures(degree=degree, include_bias=False), LinearRegression())

val_scores = {d: poly_model(d).fit(X_fit, y_fit).score(X_val, y_val) for d in range(1, 7)}
for d, s in val_scores.items():
    print(f"degree {d}: validation R2 {s:6.3f}")
best = max(val_scores, key=val_scores.get)
final = poly_model(best).fit(X_train, y_train)
print("chosen degree", best, "| test R2", round(final.score(X_test, y_test), 3))

Validation picks degree 1, the straight line. On this data the curve isn’t worth its extra variance. That’s a common and useful outcome: the honest procedure often prefers the simple model that a peek at the test set would have talked you out of.

One validation split of about 80 patients is itself noisy; a different random_state could favour degree 2. Section 4 replaces it with cross-validation, which averages over several splits. The principle stays the same.

Lowering the degree is a blunt way to control complexity: you either have the bmi × bp column or you don’t. Next lesson keeps all the columns and instead penalises the model for using them heavily, which is a finer dial.