Linear Regression and Its Metrics

Churn is a yes/no question, and you’ll come back to it in section 3. Regression comes first because its simplest model, a straight line, lets you see exactly what “learning” means: picking numbers so that predictions miss by as little as possible.

The data is scikit-learn’s diabetes dataset: 442 patients, ten baseline measurements each (age, sex, body mass index, blood pressure and six blood serum readings s1–s6), and a target that measures how far the disease progressed one year later. The features are already centred and scaled, which is why they look like small decimals.

One feature, one line

Start with body mass index alone. Linear regression predicts progression = intercept + coef * bmi, and fit chooses the two numbers.

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

line = LinearRegression().fit(X_train[["bmi"]], y_train)
print("intercept:", round(line.intercept_, 1))
print("coef for bmi:", round(line.coef_[0], 1))
print("first predictions:", line.predict(X_test[["bmi"]])[:3].round(1))

The intercept (152) is the prediction for an average BMI, since the feature is centred at 0. The coefficient (975) is the slope: one unit of this scaled BMI adds 975 to the prediction. Note the double brackets in X_train[["bmi"]]: scikit-learn wants a 2D table even for one feature.

Residuals and the loss

How did fit choose those numbers? For each patient, the residual is the actual value minus the predicted value: the vertical gap between the point and the line. Ordinary least squares picks the intercept and slope that make the sum of squared residuals as small as possible. That sum, the thing being minimised, is the loss.

Squaring does two jobs: it makes every miss count as positive, and it makes big misses count much more than small ones. A miss of 20 costs 400; a miss of 40 costs 1,600. Keep that in mind, because it explains the difference between two of the metrics below.

All ten features

Multiple regression is the same idea with more terms: one coefficient per feature, plus the intercept.

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

model = LinearRegression().fit(X_train, y_train)
for name, coef in zip(X.columns, model.coef_):
    print(f"{name:>4} {coef:8.1f}")

Measuring a regression

score gives R², but you should know three metrics and when each one fits.

from sklearn.datasets import load_diabetes
from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, root_mean_squared_error, r2_score
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

models = {
    "mean (dummy)": (DummyRegressor(), X.columns),
    "bmi only": (LinearRegression(), ["bmi"]),
    "all features": (LinearRegression(), X.columns),
}
for name, (model, cols) in models.items():
    pred = model.fit(X_train[cols], y_train).predict(X_test[cols])
    print(f"{name:13} MAE {mean_absolute_error(y_test, pred):5.1f}  "
          f"RMSE {root_mean_squared_error(y_test, pred):5.1f}  R2 {r2_score(y_test, pred):6.3f}")

• MAE (mean absolute error) is the average size of a miss, in the target’s units. “On average we’re off by 42 points” is a sentence a doctor or a manager understands. Use it to communicate.
• RMSE (root mean squared error) is the square root of the average squared miss, also in target units. Because of the squaring it’s always at least as large as MAE, and much larger when a few predictions are badly wrong. Use it when big misses are disproportionately costly.
• R² compares the model’s squared error with that of always predicting the mean: 1 is perfect, 0 is no better than predicting the mean, and negative is worse than the mean. It’s unitless, so it’s handy for comparing models on the same data, but it hides how big the misses are in real terms.

The full model cuts MAE from 65.5 (the mean baseline) to 41.5 and explains about half the variance. That’s honest progress, and also a reminder that half of what drives this disease isn’t in these ten columns.

You’ve used fit as a black box that returns the best line. Next lesson opens the box: you’ll write the search for the best line yourself, with gradient descent, the same idea that trains neural networks.