{"id":"l-ml-2-3","courseId":"ml-crash-course","lang":"en","kind":"lesson","summary":"Add polynomial features to make a linear model bend, watch the training score rise while the test score collapses, and use a validation split to pick model complexity honestly.","html":"

A straight line is a strong assumption. If disease progression rises faster at high BMI than at low BMI, no straight line can capture that, however long gradient descent runs. The fix sounds harmless: let the model bend. This lesson shows how, and how quickly bending turns into memorising.

\n

Bending a linear model

\n

The trick is to keep linear regression and change the inputs. If you add a column bmi², a “linear” model in bmi and bmi² traces a parabola in bmi. PolynomialFeatures generates these columns: every power up to the chosen degree, plus every product of features (the interactions).

\n
Python
from sklearn.datasets import load_diabetes\nfrom sklearn.preprocessing import PolynomialFeatures\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\npoly = PolynomialFeatures(degree=2, include_bias=False).fit(X[[\"bmi\", \"bp\"]])\nprint(poly.get_feature_names_out())\n\nfor degree in [1, 2, 3]:\n    n = PolynomialFeatures(degree=degree, include_bias=False).fit(X).n_output_features_\n    print(f\"degree {degree}: {n} columns from 10 features\")
\n

Two features become five at degree 2. Ten features become 65 at degree 2 and 285 at degree 3. include_bias=False drops the constant column, because LinearRegression already fits an intercept.

\n

To chain the expansion and the regression, use make_pipeline. It creates one estimator that runs each step in order; fit fits each step on the training data and predict sends new data through the same steps. You’ll build richer pipelines in section 4.

\n

The training score keeps climbing

\n
Python
from sklearn.datasets import load_diabetes\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.metrics import mean_absolute_error\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import PolynomialFeatures\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n\nfor degree in [1, 2, 3]:\n    model = make_pipeline(PolynomialFeatures(degree=degree, include_bias=False),\n                          LinearRegression()).fit(X_train, y_train)\n    print(f\"degree {degree}: train R2 {model.score(X_train, y_train):6.3f}  \"\n          f\"test R2 {model.score(X_test, y_test):7.3f}  \"\n          f\"test MAE {mean_absolute_error(y_test, model.predict(X_test)):6.1f}\")
\n

Read the three rows slowly. Degree 2 improves the training score and worsens the test score. Degree 3 is a disaster dressed as a success: training R² of 0.90, test R² of -8.5, and a typical test miss of 151 points, worse than predicting the average for everyone. With 285 columns and only 331 training patients, the model has nearly enough knobs to pass through every training point. It has learned the noise.

\n

Underfitting and overfitting

\n

Those three rows map to a picture you’ll see in every ML course, because it’s the central trade-off of the field.

\n
\n Three panels of the same curved scatter. Left: a straight line misses the curve (underfit). Middle: a gentle curve follows the trend (good fit). Right: a wiggly line passes through every point (overfit).\n \n \n \n \n \n \n \n \n \n Underfit\n Good fit\n Overfit\n
Underfitting, a good fit, and overfitting
\n\n

You diagnose which side you’re on by comparing the two scores, never by looking at the training score alone. A small gap with poor scores says “add capacity”; a big gap says “take it away”.

\n

The amount of data shifts the balance. With 331 patients, 285 columns is reckless; with 300,000 patients, the same degree-3 model would have far less room to memorise, because noise in individual rows averages out across so many examples. That’s why “collect more data” is a legitimate cure for overfitting and a useless one for underfitting: a model that can’t express the pattern doesn’t get better at it by seeing more of it. When you’re unsure which side you’re on, a learning curve (training and validation scores plotted against the number of training rows) settles it: two curves that meet low need a richer model, two that stay apart need more data or a simpler model.

\n\n

Choosing the degree honestly

\n

Here’s that recipe with three features. The validation split comes from X_train; the test set isn’t touched until the very last line.

\n
Python
from sklearn.datasets import load_diabetes\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import PolynomialFeatures\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX = X[[\"bmi\", \"bp\", \"s5\"]]\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\nX_fit, X_val, y_fit, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=0)\n\ndef poly_model(degree):\n    return make_pipeline(PolynomialFeatures(degree=degree, include_bias=False), LinearRegression())\n\nval_scores = {d: poly_model(d).fit(X_fit, y_fit).score(X_val, y_val) for d in range(1, 7)}\nfor d, s in val_scores.items():\n    print(f\"degree {d}: validation R2 {s:6.3f}\")\nbest = max(val_scores, key=val_scores.get)\nfinal = poly_model(best).fit(X_train, y_train)\nprint(\"chosen degree\", best, \"| test R2\", round(final.score(X_test, y_test), 3))
\n

Validation picks degree 1, the straight line. On this data the curve isn’t worth its extra variance. That’s a common and useful outcome: the honest procedure often prefers the simple model that a peek at the test set would have talked you out of.

\n

One validation split of about 80 patients is itself noisy; a different random_state could favour degree 2. Section 4 replaces it with cross-validation, which averages over several splits. The principle stays the same.

\n

Lowering the degree is a blunt way to control complexity: you either have the bmi × bp column or you don’t. Next lesson keeps all the columns and instead penalises the model for using them heavily, which is a finer dial.

\n","toc":[{"id":"sec-1","text":"Bending a linear model","level":2},{"id":"sec-2","text":"The training score keeps climbing","level":2},{"id":"sec-3","text":"Underfitting and overfitting","level":2},{"id":"sec-4","text":"Choosing the degree honestly","level":2}],"takeaways":["PolynomialFeatures adds squared and interaction columns, so a linear model can fit curves; the number of columns grows very fast with degree.","Underfitting means the model is too simple to capture the pattern; both training and test scores are poor.","Overfitting means the model has learned noise in the training rows; the training score is high and the test score much lower.","Choose complexity (degree, depth, number of features) on a validation split carved from the training data, never on the test set.","The usual cures for overfitting are a simpler model, more data, fewer features, or regularisation."],"further":[{"title":"PolynomialFeatures","url":"https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html"},{"title":"Underfitting vs. Overfitting","url":"https://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html"},{"title":"Validation curves","url":"https://scikit-learn.org/stable/modules/learning_curve.html"}],"quiz":[{"q":"

A model scores R² 0.52 on training data and 0.49 on test data. A more complex version scores 0.90 and -8.5. Which diagnosis fits?

","options":[{"text":"The simple model underfits and the complex one is better because it learned more.","why":"The complex model’s test score is far below zero, so whatever it learned doesn’t transfer. A small train/test gap is a healthy sign, not underfitting."},{"text":"The complex model overfits; it memorised the training rows and fails on new ones.","why":"Correct. A huge jump in training score paired with a collapse in test score is the signature of overfitting."},{"text":"Both models underfit because neither reaches 0.9 on the test set.","why":"Many real problems top out well below 0.9. Underfitting is judged by the training score being poor too, which isn’t the case for the complex model."}],"answer":1},{"q":"

PolynomialFeatures(degree=2, include_bias=False) on two columns a and b produces which columns?

","options":[{"text":"a, b, a^2, a b, b^2","why":"Correct. Degree 2 adds every square and every pairwise product, the interaction term."},{"text":"a^2, b^2","why":"The original columns are kept, and the interaction a b is included too."},{"text":"1, a, b, a^2, b^2","why":"include_bias=False drops the constant column, and the a b interaction is missing from this list."}],"answer":0},{"q":"

How should you choose the polynomial degree?

","options":[{"text":"Pick the degree with the highest training score.","why":"The training score keeps rising with degree, so this always picks the most overfit model."},{"text":"Pick the degree with the highest test score, then report that score.","why":"That uses the test set to make a decision, so the reported score is optimistic."},{"text":"Split a validation set off the training data, pick the degree that scores best there, then measure once on the test set.","why":"Correct. The validation set absorbs the selection; the test set stays an unbiased final check."}],"answer":2},{"q":"

Which change is least likely to reduce overfitting?

","options":[{"text":"Collecting more training rows.","why":"More data makes it harder to memorise noise, so it usually helps."},{"text":"Adding more polynomial degrees so the model can fit the data more precisely.","why":"Correct. Extra flexibility is what causes overfitting in the first place."},{"text":"Lowering the degree or dropping weak features.","why":"A simpler model has less capacity to memorise, which is a direct cure."},{"text":"Adding a penalty on large coefficients.","why":"That’s regularisation, the subject of the next lesson, and a standard cure."}],"answer":1}],"exercise":{"kind":"python","starter":"from sklearn.datasets import load_diabetes\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import PolynomialFeatures\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX = X[[\"bmi\", \"bp\", \"s5\", \"s3\"]]\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\nX_fit, X_val, y_fit, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=0)\n\n# 1. val_scores: {degree: validation R2} for degrees 1 to 6.\n# Fit each model on X_fit / y_fit, score it on X_val / y_val.\nval_scores = {}\n\n# 2. The degree with the best validation score\nbest_degree = None\n\n# 3. Refit that degree on ALL the training data, then score once on the test set\ntest_r2 = None\n","solution":"from sklearn.datasets import load_diabetes\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import PolynomialFeatures\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX = X[[\"bmi\", \"bp\", \"s5\", \"s3\"]]\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\nX_fit, X_val, y_fit, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=0)\n\ndef poly_model(degree):\n return make_pipeline(PolynomialFeatures(degree=degree, include_bias=False), LinearRegression())\n\nval_scores = {d: poly_model(d).fit(X_fit, y_fit).score(X_val, y_val) for d in range(1, 7)}\nbest_degree = max(val_scores, key=val_scores.get)\ntest_r2 = poly_model(best_degree).fit(X_train, y_train).score(X_test, y_test)\nprint(val_scores, best_degree, test_r2)\n","checks":[{"id":"degrees","test":"sorted(val_scores) == [1, 2, 3, 4, 5, 6]"},{"id":"validation","test":"abs(float(val_scores[2]) - __import__('sklearn.pipeline', fromlist=['make_pipeline']).make_pipeline(__import__('sklearn.preprocessing', fromlist=['PolynomialFeatures']).PolynomialFeatures(degree=2, include_bias=False), __import__('sklearn.linear_model', fromlist=['LinearRegression']).LinearRegression()).fit(X_fit, y_fit).score(X_val, y_val)) < 1e-6"},{"id":"best","test":"best_degree == max(val_scores, key=val_scores.get)"},{"id":"test-score","test":"test_r2 is not None and abs(float(test_r2) - __import__('sklearn.pipeline', fromlist=['make_pipeline']).make_pipeline(__import__('sklearn.preprocessing', fromlist=['PolynomialFeatures']).PolynomialFeatures(degree=best_degree, include_bias=False), __import__('sklearn.linear_model', fromlist=['LinearRegression']).LinearRegression()).fit(X_train, y_train).score(X_test, y_test)) < 1e-6"}],"title":"Pick a polynomial degree without touching the test set","prompt":"

Using four diabetes features, choose the polynomial degree honestly.

\n
    \n
  1. val_scores: a dict mapping each degree from 1 to 6 to its validation R². Each model is PolynomialFeatures(degree=d, include_bias=False) followed by LinearRegression, fitted on X_fit, y_fit and scored on X_val, y_val.
    2. \n
  2. best_degree: the degree with the highest validation R².
    3. \n
  3. test_r2: refit that degree on all of X_train, y_train and score it once on X_test, y_test.
    4. \n
\n","hints":["A helper poly_model(degree) that returns make_pipeline(PolynomialFeatures(...), LinearRegression()) keeps the loop short.","A dict comprehension works: {d: poly_model(d).fit(X_fit, y_fit).score(X_val, y_val) for d in range(1, 7)}.","max(val_scores, key=val_scores.get) returns the key with the largest value."],"explanation":"

Validation scores fall away as the degree grows, and some higher degrees go negative, so the procedure settles on a low degree. Refitting on all of X_train uses every training row for the final model, and the single test score is an unbiased estimate because the test set played no part in the choice.

\n","checkLabels":{"degrees":"val_scores has degrees 1 to 6","validation":"Validation scores come from models fitted on X_fit and scored on X_val","best":"best_degree has the highest validation score","test-score":"test_r2 comes from the chosen degree refitted on all training data"}},"words":706,"readMinutes":3}