LinearRegression().fit returned the best line instantly, with no visible search. For a straight line there’s an exact formula, so that’s fair. But most models (logistic regression, neural networks, the gradient boosting you’ll meet in section 3) have no formula. They find their parameters by search, and nearly all of them use some form of the same search: gradient descent.
Take the BMI-only model from last lesson: prediction = w * x + b. For any pair of w and b you can compute the mean squared error on the training data. Picture every possible w laid out left to right and the loss for each one as a height. For a straight line with squared error, that picture is a smooth bowl, and the best w sits at the bottom.
You can’t see the bowl, but at any point you can compute its slope: the gradient. If the slope is positive, increasing w makes things worse, so decrease it. If negative, increase it. Take a small step against the slope, recompute, repeat. That’s gradient descent.
The steps shrink on their own as you approach the bottom, because the slope gets flatter. The size of each step is the gradient multiplied by the learning rate, a hyperparameter you choose.
\nFor squared error, the gradients have short formulas: the slope for w is 2 * mean(error * x) and for b it’s 2 * mean(error), where error = prediction - actual. Here it is in NumPy, with BMI standardised to mean 0 and standard deviation 1 first (you’ll see why shortly).
import numpy as np\nfrom sklearn.datasets import load_diabetes\nfrom sklearn.linear_model import LinearRegression\nfrom sklearn.model_selection import train_test_split\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\nx = ((X_train[\"bmi\"] - X_train[\"bmi\"].mean()) / X_train[\"bmi\"].std(ddof=0)).to_numpy()\ntarget = y_train.to_numpy()\n\nw, b, learning_rate = 0.0, 0.0, 0.1\nfor step in range(101):\n error = (w * x + b) - target\n if step % 25 == 0:\n print(f\"step {step:3} loss {np.mean(error ** 2):9.1f} w {w:6.2f} b {b:7.2f}\")\n w -= learning_rate * 2 * np.mean(error * x)\n b -= learning_rate * 2 * np.mean(error)\n\nexact = LinearRegression().fit(x.reshape(-1, 1), target)\nprint(\"LinearRegression:\", exact.coef_.round(2), round(exact.intercept_, 2))The loss falls from about 29,900 to 3,931 within 25 steps and then stops moving. The w and b it settles on match LinearRegression to two decimals. You’ve trained a model by hand.
Change learning_rate and rerun. With 0.01, a hundred steps reach only w ≈ 40 and b ≈ 134: the search crawls. With 0.5, it lands in a single step. With 1.05, each step overshoots the bottom by more than the last, and after a hundred steps the loss is in the trillions. That’s divergence, and when you see a loss that grows instead of shrinking, a too-large learning rate is the first suspect.
Real tables mix scales. In the churn data, total_spent ranges into the thousands while used_coupon is 0 or 1. With two features like that, the loss surface is no longer a round bowl but a long, narrow valley. A learning rate small enough to stay stable along the steep direction is painfully slow along the flat one, and the path zig-zags across the valley.
Scale also hits models that never use gradients. k-nearest neighbours measures distance between customers, and a $300 difference in spend swamps a difference of four support tickets. Standardising every column to mean 0 and standard deviation 1 puts them on equal footing.
\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.preprocessing import StandardScaler\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\n\nscaler = StandardScaler().fit(X_train) # learn means and stds from TRAIN only\nX_train_s, X_test_s = scaler.transform(X_train), scaler.transform(X_test)\nprint(\"learned means:\", scaler.mean_[:3].round(1))\n\nraw = KNeighborsClassifier().fit(X_train, y_train).score(X_test, y_test)\nscaled = KNeighborsClassifier().fit(X_train_s, y_train).score(X_test_s, y_test)\nprint(f\"k-NN raw {raw:.2f} scaled {scaled:.2f}\")Scaling alone takes k-NN from 0.72 to 0.75, past the 80-day rule from section 1. Same model, same data; the only change is that every feature now gets a fair vote.
\nA scaler is an estimator too. It has fit (learn each column’s mean and standard deviation) and transform (apply them), and the same rule applies as for models: fit on training data only. Juggling X_train_s and X_test_s by hand gets error-prone, so section 4 bundles scaler and model into a single pipeline object.
Gradient descent will find the best line, but “best on the training data” can be a trap when the model is flexible. Next lesson makes the line bend, and watches it bend too far.
\n","toc":[{"id":"sec-1","text":"Learning as walking downhill","level":2},{"id":"sec-2","text":"Gradient descent in eight lines","level":2},{"id":"sec-3","text":"The learning rate decides everything","level":2},{"id":"sec-4","text":"Why scale features","level":2}],"takeaways":["Training a model means searching for the parameter values that make the loss as small as possible.","Gradient descent repeats one step: compute the slope of the loss for each parameter, then move each parameter a little in the downhill direction.","The learning rate sets the step size: too small crawls, too large overshoots and diverges.","Features on very different scales stretch the loss surface, which slows gradient descent and distorts distance-based models such as k-NN.","StandardScaler learns each column’s mean and standard deviation from the training data and rescales to mean 0, standard deviation 1."],"further":[{"title":"StandardScaler","url":"https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html"},{"title":"Stochastic Gradient Descent","url":"https://scikit-learn.org/stable/modules/sgd.html"},{"title":"Importance of feature scaling","url":"https://scikit-learn.org/stable/auto_examples/preprocessing/plot_scaling_importance.html"}],"quiz":[{"q":"During gradient descent, the gradient of the loss with respect to w is positive. What does the update do to w?
w, because a positive gradient means w should grow.","why":"A positive gradient means the loss rises as w grows. Moving with it climbs uphill."},{"text":"Decreases w, because the update subtracts learning rate times gradient.","why":"Correct. Stepping against the gradient is what makes the loss go down."},{"text":"Leaves w unchanged until the gradient becomes zero.","why":"A non-zero gradient is exactly the signal to move. Only at the minimum is it zero."}],"answer":1},{"q":"You double the learning rate and the printed loss goes 3,900 → 12,000 → 95,000 → 2,000,000. What happened?
","options":[{"text":"The model found a better minimum further away.","why":"A better minimum would show a lower loss, not an exploding one."},{"text":"The data has outliers that the larger rate exposed.","why":"Outliers don’t make the loss grow without bound step after step. Overshooting does."},{"text":"Each step overshoots the minimum by more than the last, so the search diverges.","why":"Correct. Lower the learning rate (or scale the features) until the loss falls steadily."}],"answer":2},{"q":"Where should you fit a StandardScaler?
Which model’s predictions change when you rescale total_spent from dollars to cents?
total_spent.","why":"Correct. Distance-based models treat a 100-fold larger range as 100 times more important."},{"text":"Ordinary linear regression, because its coefficient changes.","why":"The coefficient shrinks 100-fold to compensate, and predictions stay identical. Scale matters for regularised models, not plain least squares."}],"answer":1}],"exercise":{"kind":"python","starter":"import numpy as np\nfrom sklearn.datasets import load_diabetes\nfrom sklearn.model_selection import train_test_split\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\nbmi = X_train[\"bmi\"]\nx = ((bmi - bmi.mean()) / bmi.std(ddof=0)).to_numpy() # standardised feature\ntarget = y_train.to_numpy()\n\nw, b = 0.0, 0.0\nlearning_rate = 0.05\nlosses = []\n\nfor step in range(200):\n error = (w * x + b) - target\n losses.append(np.mean(error ** 2))\n # Update w and b here: step against the gradient of the mean squared error\n pass\n","solution":"import numpy as np\nfrom sklearn.datasets import load_diabetes\nfrom sklearn.model_selection import train_test_split\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\nbmi = X_train[\"bmi\"]\nx = ((bmi - bmi.mean()) / bmi.std(ddof=0)).to_numpy() # standardised feature\ntarget = y_train.to_numpy()\n\nw, b = 0.0, 0.0\nlearning_rate = 0.05\nlosses = []\n\nfor step in range(200):\n error = (w * x + b) - target\n losses.append(np.mean(error ** 2))\n w -= learning_rate * 2 * np.mean(error * x)\n b -= learning_rate * 2 * np.mean(error)\n\nprint(round(w, 2), round(b, 2), round(losses[-1], 1))\n","checks":[{"id":"losses","test":"len(losses) == 200 and all(float(losses[i + 1]) <= float(losses[i]) + 1e-9 for i in range(199))"},{"id":"loss-drops","test":"float(losses[-1]) < 0.2 * float(losses[0])"},{"id":"w","test":"abs(float(w) - 45.97) < 0.5"},{"id":"b","test":"abs(float(b) - 154.34) < 0.5"}],"title":"Train a line with gradient descent","prompt":"Finish the gradient descent loop so it fits prediction = w * x + b to the diabetes target, where x is standardised BMI.
Inside the loop, update w and b by subtracting learning_rate times their gradients. For mean squared error the gradients are 2 * mean(error * x) for w and 2 * mean(error) for b. Keep the losses list: the checks look at it.
error before changing either parameter.","w -= learning_rate * 2 * np.mean(error * x)","Do the same for b with np.mean(error). The final values should land near w ≈ 46 and b ≈ 154."],"explanation":"Each pass computes the error of the current line, turns it into a slope for each parameter, and steps against that slope. With a learning rate of 0.05 the loss falls every step and the parameters settle where LinearRegression puts them (w ≈ 45.97, b ≈ 154.34). If you flip the sign (+=), the loss grows every step: you’re walking uphill.