Logistic Regression: Predicting Probabilities

The depth-3 tree from section 1 answered “churn” or “stay”. Marketing doesn’t really want that. They have budget for a few dozen win-back calls a week and want to know who to call first. That needs a score that ranks customers by risk, and the classic model for it is logistic regression.

From a line to a probability

Logistic regression starts like linear regression: multiply each feature by a coefficient, add them up, add an intercept. That gives a score z that can be any number. Then it squashes the score through the sigmoid function, 1 / (1 + e^-z), which maps any number to a value between 0 and 1. Large positive scores give probabilities near 1, large negative scores near 0, and a score of exactly 0 gives 0.5.

During training, the model adjusts the coefficients so that churners get high probabilities and non-churners low ones. The loss it minimises is log loss, which punishes confident wrong answers hard: predicting 0.99 for a customer who stayed costs far more than predicting 0.6. There’s no exact formula for the minimum, so scikit-learn finds it with an iterative optimiser, a refined cousin of the gradient descent you wrote last section.

Fitting it on churn

Because the optimiser is iterative and the model has a penalty (more on that below), scale the features first. A pipeline keeps the scaler and the model together.

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])

model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
print("test accuracy:", model.score(X_test, y_test))
print(model.predict_proba(X_test)[:4].round(2))
print(model.predict(X_test)[:4])

predict_proba returns two columns, one per class in the order of classes_: probability of staying, probability of churning. They add up to 1. predict is a shortcut that labels a row 1 when the churn probability is at least 0.5. That 0.5 is a default, not a law, and section 3’s last lesson is about choosing it on purpose.

Accuracy is 0.72, level with the 80-day rule. Don’t stop reading there: the probabilities carry information accuracy throws away.

Reading the coefficients

With standardised features, each coefficient says how much one standard deviation of that feature moves the score z (the log-odds of churn). That makes them comparable across features.

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)

coefs = pd.Series(model[-1].coef_[0], index=features).sort_values()
print(coefs.round(2))

ranked = X_test.assign(churn_prob=model.predict_proba(X_test)[:, 1], churned=y_test)
top10 = ranked.sort_values("churn_prob", ascending=False).head(10)
print(top10[["days_since_last_order", "orders", "churn_prob", "churned"]].round(2))
print("share of the top 10 who really churned:", top10["churned"].mean())

Recency (days_since_last_order, +0.93) dominates, which matches what the rule baseline told you. Longer tenure also pushes towards churn, while more orders, more spend, coupon use and even more support tickets push away from it. Customers who contact support are engaged customers; it’s the silent ones who leave. That’s the kind of finding worth taking back to the business, with the caveat that it’s an association, not a cause.

Then the payoff: of the ten customers the model ranks riskiest, nine really churned. A yes/no rule can’t produce that list; a probability can.

The penalty you didn’t ask for

LogisticRegression is regularised by default with an L2 penalty, the same idea as Ridge. Its strength is set by C, which is the inverse of Ridge’s alpha: small C means a strong penalty, large C means a weak one. The default C=1.0 is a reasonable start. On this data, values from 0.01 to 100 all land within a couple of points of each other, a sign that eight well-behaved features don’t give the model much room to overfit. With hundreds of features, C matters a lot, and you’d tune it with the tools in section 4.

Logistic regression draws one straight boundary through the feature space. Next lesson’s model makes no such assumption: it asks a series of yes/no questions and can carve the space into boxes.