{"id":"l-ml-2-1","courseId":"ml-crash-course","lang":"en","kind":"lesson","summary":"Fit a linear regression with scikit-learn, read its intercept and coefficients, and judge it with MAE, RMSE and R² against a mean-predicting baseline.","html":"

Churn is a yes/no question, and you’ll come back to it in section 3. Regression comes first because its simplest model, a straight line, lets you see exactly what “learning” means: picking numbers so that predictions miss by as little as possible.

\n

The data is scikit-learn’s diabetes dataset: 442 patients, ten baseline measurements each (age, sex, body mass index, blood pressure and six blood serum readings s1–s6), and a target that measures how far the disease progressed one year later. The features are already centred and scaled, which is why they look like small decimals.

\n

One feature, one line

\n

Start with body mass index alone. Linear regression predicts progression = intercept + coef * bmi, and fit chooses the two numbers.

\n
Python
from sklearn.datasets import load_diabetes\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n\nline = LinearRegression().fit(X_train[[\"bmi\"]], y_train)\nprint(\"intercept:\", round(line.intercept_, 1))\nprint(\"coef for bmi:\", round(line.coef_[0], 1))\nprint(\"first predictions:\", line.predict(X_test[[\"bmi\"]])[:3].round(1))
\n

The intercept (152) is the prediction for an average BMI, since the feature is centred at 0. The coefficient (975) is the slope: one unit of this scaled BMI adds 975 to the prediction. Note the double brackets in X_train[["bmi"]]: scikit-learn wants a 2D table even for one feature.

\n

Residuals and the loss

\n

How did fit choose those numbers? For each patient, the residual is the actual value minus the predicted value: the vertical gap between the point and the line. Ordinary least squares picks the intercept and slope that make the sum of squared residuals as small as possible. That sum, the thing being minimised, is the loss.

\n
\n Scatter of points around a rising line. Dashed vertical segments join each point to the line; these residuals are squared and summed, and least squares picks the line that makes that sum smallest.\n \n \n bmi\n progression\n \n \n \n \n \n \n \n \n \n \n \n \n \n residual\n
Residuals are the vertical gaps between points and the line
\n

Squaring does two jobs: it makes every miss count as positive, and it makes big misses count much more than small ones. A miss of 20 costs 400; a miss of 40 costs 1,600. Keep that in mind, because it explains the difference between two of the metrics below.

\n

All ten features

\n

Multiple regression is the same idea with more terms: one coefficient per feature, plus the intercept.

\n
Python
from sklearn.datasets import load_diabetes\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n\nmodel = LinearRegression().fit(X_train, y_train)\nfor name, coef in zip(X.columns, model.coef_):\n    print(f\"{name:>4} {coef:8.1f}\")
\n\n

Measuring a regression

\n

score gives R², but you should know three metrics and when each one fits.

\n
Python
from sklearn.datasets import load_diabetes\nfrom sklearn.dummy import DummyRegressor\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.metrics import mean_absolute_error, root_mean_squared_error, r2_score\nfrom sklearn.model_selection import train_test_split\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n\nmodels = {\n    \"mean (dummy)\": (DummyRegressor(), X.columns),\n    \"bmi only\": (LinearRegression(), [\"bmi\"]),\n    \"all features\": (LinearRegression(), X.columns),\n}\nfor name, (model, cols) in models.items():\n    pred = model.fit(X_train[cols], y_train).predict(X_test[cols])\n    print(f\"{name:13} MAE {mean_absolute_error(y_test, pred):5.1f}  \"\n          f\"RMSE {root_mean_squared_error(y_test, pred):5.1f}  R2 {r2_score(y_test, pred):6.3f}\")
\n\n

The full model cuts MAE from 65.5 (the mean baseline) to 41.5 and explains about half the variance. That’s honest progress, and also a reminder that half of what drives this disease isn’t in these ten columns.

\n\n

You’ve used fit as a black box that returns the best line. Next lesson opens the box: you’ll write the search for the best line yourself, with gradient descent, the same idea that trains neural networks.

\n","toc":[{"id":"sec-1","text":"One feature, one line","level":2},{"id":"sec-2","text":"Residuals and the loss","level":2},{"id":"sec-3","text":"All ten features","level":2},{"id":"sec-4","text":"Measuring a regression","level":2}],"takeaways":["Linear regression predicts intercept + coef1 * x1 + coef2 * x2 + … and picks the numbers that minimise the sum of squared residuals.","A residual is actual minus predicted; every regression metric is a different summary of the residuals.","MAE is the typical miss in the target’s own units; RMSE is in the same units but punishes big misses more.","R² is the share of variance explained compared with always predicting the mean: 0 means no better than the mean, and it can go negative.","Coefficients of correlated features can be large and opposite in sign, so don’t read them as importance or cause."],"further":[{"title":"Linear Models: Ordinary Least Squares","url":"https://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares"},{"title":"Regression metrics","url":"https://scikit-learn.org/stable/modules/model_evaluation.html#regression-metrics"},{"title":"Diabetes dataset","url":"https://scikit-learn.org/stable/datasets/toy_dataset.html#diabetes-dataset"}],"quiz":[{"q":"

A model predicting next-year spend has MAE = $40 and RMSE = $150. What does the gap suggest?

","options":[{"text":"The model is biased and predicts too high on average.","why":"Neither metric shows direction; both use absolute or squared misses. You’d look at the mean residual for bias."},{"text":"Most misses are small, but a few are very large.","why":"Correct. Squaring makes rare big misses dominate RMSE, while MAE weighs every miss equally."},{"text":"RMSE was computed on the training set and MAE on the test set.","why":"Nothing in the numbers says that. RMSE is always at least MAE on the same data; a large gap is about the shape of the errors."}],"answer":1},{"q":"

A model scores R² = -0.2 on the test set. What does that mean?

","options":[{"text":"It explains 20% of the variance, in the negative direction.","why":"R² has no direction. A negative value is a comparison with the mean predictor, not a signed share."},{"text":"The calculation is broken, because R² is a square and can’t be negative.","why":"The name is historical. On test data R² is 1 minus a ratio of errors, and the ratio can exceed 1."},{"text":"Its predictions are worse than predicting the mean for everyone.","why":"Correct. R² compares the model’s squared error with that of always predicting the mean of the true values; below 0 means the model loses."}],"answer":2},{"q":"

In the diabetes model, s1 has coefficient -918 and s2 has +508. Both are cholesterol-related blood measurements. What’s the safest reading?

","options":[{"text":"s1 strongly lowers disease progression, so raising it would help patients.","why":"Coefficients describe the fitted model, not cause and effect, and correlated inputs make them unstable."},{"text":"s1 is the most important feature because its coefficient is the largest.","why":"Size alone misleads when features are correlated; the two terms largely cancel each other."},{"text":"The two features are correlated, so the model balanced large opposite weights; their individual values aren’t reliable.","why":"Correct. With correlated features many coefficient combinations fit almost equally well. Regularisation in this section’s last lesson tames this."}],"answer":2}],"exercise":{"kind":"python","starter":"from sklearn.datasets import load_diabetes\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.metrics import mean_absolute_error, root_mean_squared_error, r2_score\nfrom sklearn.model_selection import train_test_split\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\ncols = [\"bmi\", \"bp\", \"s5\"]\n\n# 1. Fit a LinearRegression on the three columns in `cols`\nmodel = LinearRegression()\n\n# 2. Predict on the test rows and compute the three metrics\npred = None\nmae = None\nrmse = None\nr2 = None\n","solution":"from sklearn.datasets import load_diabetes\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.metrics import mean_absolute_error, root_mean_squared_error, r2_score\nfrom sklearn.model_selection import train_test_split\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\ncols = [\"bmi\", \"bp\", \"s5\"]\n\nmodel = LinearRegression().fit(X_train[cols], y_train)\n\npred = model.predict(X_test[cols])\nmae = mean_absolute_error(y_test, pred)\nrmse = root_mean_squared_error(y_test, pred)\nr2 = r2_score(y_test, pred)\nprint(f\"MAE {mae:.1f} RMSE {rmse:.1f} R2 {r2:.3f}\")\n","checks":[{"id":"fitted","test":"hasattr(model, 'coef_') and len(model.coef_) == 3"},{"id":"pred","test":"pred is not None and len(pred) == len(y_test) and abs(float(pred[0]) - float(LinearRegression().fit(X_train[['bmi', 'bp', 's5']], y_train).predict(X_test[['bmi', 'bp', 's5']])[0])) < 1e-6"},{"id":"mae","test":"mae is not None and abs(float(mae) - float((y_test - LinearRegression().fit(X_train[['bmi', 'bp', 's5']], y_train).predict(X_test[['bmi', 'bp', 's5']])).abs().mean())) < 1e-6"},{"id":"rmse","test":"rmse is not None and abs(float(rmse) - float(((y_test - LinearRegression().fit(X_train[['bmi', 'bp', 's5']], y_train).predict(X_test[['bmi', 'bp', 's5']])) ** 2).mean() ** 0.5)) < 1e-6"},{"id":"r2","test":"r2 is not None and 0.3 < float(r2) < 0.6 and float(rmse) >= float(mae)"}],"title":"A three-feature regression, measured three ways","prompt":"

Fit a LinearRegression on body mass index, blood pressure and s5 (the columns in cols) and measure it on the test set.

\n\n","hints":["Fit with model = LinearRegression().fit(X_train[cols], y_train).","The metric functions take (y_true, y_pred) in that order.","root_mean_squared_error is in sklearn.metrics; it’s already imported."],"explanation":"

Three well-chosen features get MAE about 42 and R² about 0.50, a slightly higher R² than all ten features reach on this split (0.49). That’s common: a few features carry most of the signal, and extra correlated features add noise to the coefficients. RMSE is larger than MAE because a handful of patients are predicted badly, and squaring makes those misses count more.

\n","checkLabels":{"fitted":"model is a fitted regression on three features","pred":"pred holds the test predictions","mae":"mae is the mean absolute error","rmse":"rmse is the root mean squared error","r2":"r2 is the test R² (and RMSE ≥ MAE)"}},"words":661,"readMinutes":3}