“Deep learning” gets used as a synonym for AI, and it powers image recognition, speech-to-text and the large language models in the next lesson. Underneath, it’s built from pieces you already know: weighted sums like linear regression, a squashing function like the sigmoid, and gradient descent. This lesson assembles them, trains a small network, and then puts it next to logistic regression to see what it’s really worth.
\nA neuron does what logistic regression does without the final squash: multiply each input by a weight, add them up, add a bias. Then it applies an activation function. The most common is ReLU, which outputs the sum if it’s positive and 0 otherwise. A layer is a set of neurons that all read the same inputs, each with its own weights. A network stacks layers: the outputs of one become the inputs of the next.
\nThe activation is the whole trick. Without it, a layer is a linear function, and a linear function of a linear function is still linear: ten layers would be no more powerful than one logistic regression. With ReLU in between, each layer can bend and combine what the previous one produced, so the network can represent curves and interactions you never wrote down. In effect, the hidden layers learn their own features.
\nFor classification, the output layer has one neuron per class and a softmax, which turns the outputs into probabilities that sum to 1, the multi-class cousin of the sigmoid.
\nTraining repeats four steps:
\nadam, the default, is gradient descent with a per-weight adaptive learning rate.One pass through the whole training set is an epoch. Training runs for many epochs, and you watch the loss fall exactly as it did in your hand-written gradient descent in section 2, just across thousands of weights instead of two.
\nThe digits dataset holds 1,797 images of handwritten digits, each 8×8 pixels flattened into 64 numbers. scikit-learn’s MLPClassifier (multi-layer perceptron) is a small fully connected network.
from sklearn.datasets import load_digits\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.neural_network import MLPClassifier\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nX, y = load_digits(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)\n\nnet = make_pipeline(StandardScaler(),\n MLPClassifier(hidden_layer_sizes=(64,), max_iter=500, random_state=42))\nnet.fit(X_train, y_train)\nmlp = net[-1]\nprint(\"weight matrices:\", [w.shape for w in mlp.coefs_])\nprint(\"parameters:\", sum(w.size for w in mlp.coefs_) + sum(b.size for b in mlp.intercepts_))\nprint(\"epochs run:\", mlp.n_iter_, \"| loss first, 10th, last:\",\n [round(float(mlp.loss_curve_[i]), 3) for i in (0, 9, -1)])\nprint(\"network test accuracy: \", round(net.score(X_test, y_test), 3))\n\nlinear = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)).fit(X_train, y_train)\nprint(\"logistic test accuracy:\", round(linear.score(X_test, y_test), 3))One hidden layer of 64 neurons means two weight matrices: 64×64 from pixels to hidden units and 64×10 from hidden units to digits, 4,810 parameters with the biases. The loss falls from 2.31 (a uniform guess over ten classes) to under 0.01, and training stops on its own after 164 epochs because the improvement has stalled. Test accuracy: 98%.
\nAnd logistic regression? Also about 98%. On 8×8 images with a few hundred examples per class, a straight boundary is already enough. The network’s extra power has almost nothing to work on. Try it on churn and the result is the same, slightly worse: around 0.79 ROC AUC against 0.81 for logistic regression.
\n\nThe networks behind modern AI differ from MLPClassifier in scale and in architecture. Convolutional networks reuse the same small filters across an image, so they learn edges, then shapes, then objects. Transformers, behind today’s language models, let every word in a text weigh every other word. Both are trained on millions or billions of examples, on GPUs, with libraries such as PyTorch, Keras or JAX, not scikit-learn.
What they share with your 4,810-weight network is everything conceptual: layers, activations, a loss, backpropagation, epochs, overfitting, a validation set. When a deep learning paper says “we trained for 20 epochs with early stopping on validation loss”, you now know what every word means.
\n\nPretrained models are the bridge between the two worlds, and the most visible of them are large language models. Next lesson explains what they do, how embeddings turn text into the kind of vectors you’ve been feeding models all course, and where they fit next to a churn model.
\n","toc":[{"id":"sec-1","text":"From one neuron to a network","level":2},{"id":"sec-2","text":"How a network learns","level":2},{"id":"sec-3","text":"A network on handwritten digits","level":2},{"id":"sec-4","text":"Where deep learning wins","level":2}],"takeaways":["A neuron computes a weighted sum of its inputs plus a bias and passes it through an activation function such as ReLU; a layer is many neurons side by side.","Without non-linear activations, any stack of layers collapses into a single linear model.","Training repeats forward pass, loss, backpropagation (gradients for every weight) and an optimiser step; one pass over the training data is an epoch.","MLPClassifier is a small fully connected network for tabular-sized problems; real deep learning uses PyTorch, Keras or JAX on GPUs.","Neural networks shine on large amounts of raw, unstructured data such as images, audio and text; on small tables, linear models and boosted trees usually match or beat them."],"further":[{"title":"Neural network models (supervised)","url":"https://scikit-learn.org/stable/modules/neural_networks_supervised.html"},{"title":"MLPClassifier","url":"https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html"}],"quiz":[{"q":"Why do neural networks need non-linear activation functions between layers?
","options":[{"text":"They make training faster on GPUs.","why":"Speed isn’t the reason; the reason is what the network can represent at all."},{"text":"They keep the outputs between 0 and 1.","why":"That’s only true of some activations (sigmoid); ReLU outputs can be any non-negative number."},{"text":"They stop the weights from growing too large.","why":"That’s regularisation’s job (alpha in MLPClassifier), not the activation’s."},{"text":"Without them, several linear layers multiply out to one linear function, so depth adds nothing.","why":"Correct. The non-linearity is what lets stacked layers model curves and interactions."}],"answer":3},{"q":"A network trains on 1,347 images with max_iter=500, and n_iter_ reports 164. What happened?
adam and sgd solvers, max_iter caps the number of epochs, and training stops early once progress stalls."},{"text":"It saw only 164 of the 1,347 images.","why":"An epoch is a full pass over all training rows; each of the 164 epochs used every image."},{"text":"It crashed at epoch 164 and kept the last good weights.","why":"Stopping early is normal convergence, not a crash; a failure to converge would raise a ConvergenceWarning at 500."}],"answer":0},{"q":"On the churn table, a small MLP reaches CV ROC AUC 0.79 while logistic regression reaches 0.81. What’s the most sensible reading?
","options":[{"text":"The MLP needs more layers; deep learning always wins with enough depth.","why":"With 298 rows and eight features, extra capacity mostly adds variance. Depth isn’t free."},{"text":"On a small table of informative features, a simple model is hard to beat; the network adds complexity without signal to use.","why":"Correct. Networks earn their keep when there’s a lot of data and raw inputs whose features are hard to design by hand."},{"text":"Neural networks can’t be used for binary classification.","why":"They handle binary classification routinely; the issue is data size and type."}],"answer":1}],"exercise":{"kind":"python","starter":"from sklearn.datasets import load_digits\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.neural_network import MLPClassifier\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nX, y = load_digits(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)\n\n# 1. Pipeline: StandardScaler, then an MLPClassifier with ONE hidden layer of 32 neurons,\n# max_iter=500, random_state=42. Fit it on the training data.\nnet = MLPClassifier(hidden_layer_sizes=(32, 32), max_iter=500, random_state=42)\n\n# 2. Facts about the trained network\nn_params = None # total number of weights + biases\nepochs = None # epochs actually run\ntest_acc = None # test accuracy\n","solution":"from sklearn.datasets import load_digits\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.neural_network import MLPClassifier\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nX, y = load_digits(return_X_y=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)\n\nnet = make_pipeline(StandardScaler(),\n MLPClassifier(hidden_layer_sizes=(32,), max_iter=500, random_state=42))\nnet.fit(X_train, y_train)\n\nmlp = net[-1]\nn_params = sum(w.size for w in mlp.coefs_) + sum(b.size for b in mlp.intercepts_)\nepochs = mlp.n_iter_\ntest_acc = net.score(X_test, y_test)\nprint(n_params, epochs, round(test_acc, 3))\n","checks":[{"id":"pipeline","test":"hasattr(net, 'steps') and type(net[0]).__name__ == 'StandardScaler' and type(net[-1]).__name__ == 'MLPClassifier'"},{"id":"architecture","test":"tuple(net[-1].hidden_layer_sizes) == (32,) and [w.shape for w in net[-1].coefs_] == [(64, 32), (32, 10)]"},{"id":"params","test":"n_params is not None and int(n_params) == 64 * 32 + 32 + 32 * 10 + 10"},{"id":"trained","test":"epochs is not None and int(epochs) == net[-1].n_iter_ and net[-1].loss_curve_[-1] < net[-1].loss_curve_[0]"},{"id":"accuracy","test":"test_acc is not None and 0.95 < float(test_acc) <= 1.0"}],"title":"Train and inspect a small network","prompt":"net: a pipeline of StandardScaler then MLPClassifier with one hidden layer of 32 neurons (max_iter=500, random_state=42), fitted on the digits training data.n_params (all weights plus all biases), epochs (how many epochs training actually ran) and test_acc (accuracy on the test set).hidden_layer_sizes=(32,); the trailing comma makes it a tuple.","net[-1] is the fitted MLPClassifier. Its coefs_ and intercepts_ are lists of NumPy arrays; add up their .size.","n_iter_ holds the number of epochs run."],"explanation":"The network has a 64×32 weight matrix into the hidden layer and a 32×10 matrix out to the ten digits, plus 32 and 10 biases: 2,410 parameters. Training stops by itself once the loss stops improving, well before 500 epochs, and the test accuracy lands around 97–98%, about the same as logistic regression on these small images. Without the scaler, pixel values from 0 to 16 make training slower and less stable.
\n","checkLabels":{"pipeline":"net scales, then runs an MLPClassifier","architecture":"One hidden layer of 32 neurons","params":"n_params counts every weight and bias","trained":"epochs matches the epochs run, and the loss fell","accuracy":"test_acc is above 0.95"}},"words":803,"readMinutes":4}