How Models Learn: Loss and Gradient Descent

LinearRegression().fit returned the best line instantly, with no visible search. For a straight line there’s an exact formula, so that’s fair. But most models (logistic regression, neural networks, the gradient boosting you’ll meet in section 3) have no formula. They find their parameters by search, and nearly all of them use some form of the same search: gradient descent.

Learning as walking downhill

Take the BMI-only model from last lesson: prediction = w * x + b. For any pair of w and b you can compute the mean squared error on the training data. Picture every possible w laid out left to right and the loss for each one as a height. For a straight line with squared error, that picture is a smooth bowl, and the best w sits at the bottom.

You can’t see the bowl, but at any point you can compute its slope: the gradient. If the slope is positive, increasing w makes things worse, so decrease it. If negative, increase it. Take a small step against the slope, recompute, repeat. That’s gradient descent.

The steps shrink on their own as you approach the bottom, because the slope gets flatter. The size of each step is the gradient multiplied by the learning rate, a hyperparameter you choose.

Gradient descent in eight lines

For squared error, the gradients have short formulas: the slope for w is 2 * mean(error * x) and for b it’s 2 * mean(error), where error = prediction - actual. Here it is in NumPy, with BMI standardised to mean 0 and standard deviation 1 first (you’ll see why shortly).

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
x = ((X_train["bmi"] - X_train["bmi"].mean()) / X_train["bmi"].std(ddof=0)).to_numpy()
target = y_train.to_numpy()

w, b, learning_rate = 0.0, 0.0, 0.1
for step in range(101):
    error = (w * x + b) - target
    if step % 25 == 0:
        print(f"step {step:3}  loss {np.mean(error ** 2):9.1f}  w {w:6.2f}  b {b:7.2f}")
    w -= learning_rate * 2 * np.mean(error * x)
    b -= learning_rate * 2 * np.mean(error)

exact = LinearRegression().fit(x.reshape(-1, 1), target)
print("LinearRegression:", exact.coef_.round(2), round(exact.intercept_, 2))

The loss falls from about 29,900 to 3,931 within 25 steps and then stops moving. The w and b it settles on match LinearRegression to two decimals. You’ve trained a model by hand.

The learning rate decides everything

Change learning_rate and rerun. With 0.01, a hundred steps reach only w ≈ 40 and b ≈ 134: the search crawls. With 0.5, it lands in a single step. With 1.05, each step overshoots the bottom by more than the last, and after a hundred steps the loss is in the trillions. That’s divergence, and when you see a loss that grows instead of shrinking, a too-large learning rate is the first suspect.

Why scale features

Real tables mix scales. In the churn data, total_spent ranges into the thousands while used_coupon is 0 or 1. With two features like that, the loss surface is no longer a round bowl but a long, narrow valley. A learning rate small enough to stay stable along the steep direction is painfully slow along the flat one, and the path zig-zags across the valley.

Scale also hits models that never use gradients. k-nearest neighbours measures distance between customers, and a $300 difference in spend swamps a difference of four support tickets. Standardising every column to mean 0 and standard deviation 1 puts them on equal footing.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])

scaler = StandardScaler().fit(X_train)          # learn means and stds from TRAIN only
X_train_s, X_test_s = scaler.transform(X_train), scaler.transform(X_test)
print("learned means:", scaler.mean_[:3].round(1))

raw = KNeighborsClassifier().fit(X_train, y_train).score(X_test, y_test)
scaled = KNeighborsClassifier().fit(X_train_s, y_train).score(X_test_s, y_test)
print(f"k-NN raw {raw:.2f}  scaled {scaled:.2f}")

Scaling alone takes k-NN from 0.72 to 0.75, past the 80-day rule from section 1. Same model, same data; the only change is that every feature now gets a fair vote.

A scaler is an estimator too. It has fit (learn each column’s mean and standard deviation) and transform (apply them), and the same rule applies as for models: fit on training data only. Juggling X_train_s and X_test_s by hand gets error-prone, so section 4 bundles scaler and model into a single pipeline object.

Gradient descent will find the best line, but “best on the training data” can be a trap when the model is flexible. Next lesson makes the line bend, and watches it bend too far.