Neural Networks and Deep Learning

“Deep learning” gets used as a synonym for AI, and it powers image recognition, speech-to-text and the large language models in the next lesson. Underneath, it’s built from pieces you already know: weighted sums like linear regression, a squashing function like the sigmoid, and gradient descent. This lesson assembles them, trains a small network, and then puts it next to logistic regression to see what it’s really worth.

From one neuron to a network

A neuron does what logistic regression does without the final squash: multiply each input by a weight, add them up, add a bias. Then it applies an activation function. The most common is ReLU, which outputs the sum if it’s positive and 0 otherwise. A layer is a set of neurons that all read the same inputs, each with its own weights. A network stacks layers: the outputs of one become the inputs of the next.

The activation is the whole trick. Without it, a layer is a linear function, and a linear function of a linear function is still linear: ten layers would be no more powerful than one logistic regression. With ReLU in between, each layer can bend and combine what the previous one produced, so the network can represent curves and interactions you never wrote down. In effect, the hidden layers learn their own features.

For classification, the output layer has one neuron per class and a softmax, which turns the outputs into probabilities that sum to 1, the multi-class cousin of the sigmoid.

How a network learns

Training repeats four steps:

1. Forward pass: run a batch of examples through the network to get predictions.
2. Loss: measure how wrong they are, with log loss for classification.
3. Backpropagation: compute the gradient of the loss with respect to every weight, working backwards from the output layer with the chain rule.
4. Optimiser step: nudge every weight against its gradient. adam, the default, is gradient descent with a per-weight adaptive learning rate.

One pass through the whole training set is an epoch. Training runs for many epochs, and you watch the loss fall exactly as it did in your hand-written gradient descent in section 2, just across thousands of weights instead of two.

A network on handwritten digits

The digits dataset holds 1,797 images of handwritten digits, each 8×8 pixels flattened into 64 numbers. scikit-learn’s MLPClassifier (multi-layer perceptron) is a small fully connected network.

from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)

net = make_pipeline(StandardScaler(),
                    MLPClassifier(hidden_layer_sizes=(64,), max_iter=500, random_state=42))
net.fit(X_train, y_train)
mlp = net[-1]
print("weight matrices:", [w.shape for w in mlp.coefs_])
print("parameters:", sum(w.size for w in mlp.coefs_) + sum(b.size for b in mlp.intercepts_))
print("epochs run:", mlp.n_iter_, "| loss first, 10th, last:",
      [round(float(mlp.loss_curve_[i]), 3) for i in (0, 9, -1)])
print("network test accuracy: ", round(net.score(X_test, y_test), 3))

linear = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)).fit(X_train, y_train)
print("logistic test accuracy:", round(linear.score(X_test, y_test), 3))

One hidden layer of 64 neurons means two weight matrices: 64×64 from pixels to hidden units and 64×10 from hidden units to digits, 4,810 parameters with the biases. The loss falls from 2.31 (a uniform guess over ten classes) to under 0.01, and training stops on its own after 164 epochs because the improvement has stalled. Test accuracy: 98%.

And logistic regression? Also about 98%. On 8×8 images with a few hundred examples per class, a straight boundary is already enough. The network’s extra power has almost nothing to work on. Try it on churn and the result is the same, slightly worse: around 0.79 ROC AUC against 0.81 for logistic regression.

Where deep learning wins

The networks behind modern AI differ from MLPClassifier in scale and in architecture. Convolutional networks reuse the same small filters across an image, so they learn edges, then shapes, then objects. Transformers, behind today’s language models, let every word in a text weigh every other word. Both are trained on millions or billions of examples, on GPUs, with libraries such as PyTorch, Keras or JAX, not scikit-learn.

What they share with your 4,810-weight network is everything conceptual: layers, activations, a loss, backpropagation, epochs, overfitting, a validation set. When a deep learning paper says “we trained for 20 epochs with early stopping on validation loss”, you now know what every word means.

Pretrained models are the bridge between the two worlds, and the most visible of them are large language models. Next lesson explains what they do, how embeddings turn text into the kind of vectors you’ve been feeding models all course, and where they fit next to a churn model.