Thresholds and Class Imbalance

predict has been quietly making a decision for you: label a customer a churner when the probability is at least 0.5. That cut-off assumes a missed churner and a wasted offer cost the same. They almost never do. Choosing the threshold on purpose is one of the cheapest, highest-value improvements you can make to a classifier, and it doesn’t require retraining anything.

The threshold is a dial

Probabilities come from the model; the threshold is your policy. Sweep it and watch precision and recall pull against each other.

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
prob = model.predict_proba(X_test)[:, 1]

for t in [0.3, 0.4, 0.5, 0.6, 0.7, 0.8]:
    flagged = (prob >= t).astype(int)
    print(f"threshold {t:.1f}: flags {flagged.sum():2}  precision {precision_score(y_test, flagged):.2f}"
          f"  recall {recall_score(y_test, flagged):.2f}")

At 0.3 the model flags 74 customers and catches 93% of churners, but 31% of the flags are wrong. At 0.8 it flags 21, and 86% of them are right, but it catches only a third of the churners. Neither end is “better”. The right point depends on money.

Let the costs pick the threshold

Cartwheel’s numbers: a win-back offer costs $20 in discount and handling. Contacting a real churner saves them 30% of the time, and a saved customer is worth $100 in margin, so each contacted churner is worth 0.3 * 100 - 20 = +$10 on average. Contacting a loyal customer wastes the $20. Profit is 10 * TP - 20 * FP.

If the probabilities were perfectly calibrated, you could solve for the break-even point: contact when p * 10 > (1 - p) * 20, which is when p > 2/3. Real probabilities are rarely that well calibrated, so the more reliable approach is to search for the threshold that maximises profit, using cross-validation on the training data. TunedThresholdClassifierCV does exactly that.

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import make_scorer
from sklearn.model_selection import TunedThresholdClassifierCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])

def profit(y_true, y_pred):
    y_true, y_pred = np.asarray(y_true), np.asarray(y_pred)
    tp = ((y_pred == 1) & (y_true == 1)).sum()
    fp = ((y_pred == 1) & (y_true == 0)).sum()
    return 10 * tp - 20 * fp

base = make_pipeline(StandardScaler(), LogisticRegression())
tuned = TunedThresholdClassifierCV(base, scoring=make_scorer(profit), cv=5).fit(X_train, y_train)
print("chosen threshold:", round(tuned.best_threshold_, 2))

default = base.fit(X_train, y_train)
print("test profit, contact everyone: ", profit(y_test, np.ones(len(y_test))))
print("test profit, threshold 0.5:    ", profit(y_test, default.predict(X_test)))
print("test profit, tuned threshold:  ", profit(y_test, tuned.predict(X_test)))

Contacting all 100 test customers loses $350. The default threshold makes $170; the tuned one, about 0.58, makes $180 while contacting fewer people. The theoretical 0.67 would have made less, because the model’s probabilities are a little off: a reminder to trust measured results over tidy formulas. make_scorer turns any (y_true, y_pred) function into something scikit-learn’s tuning tools can maximise.

A probability is calibrated when, among all customers scored at 0.7, about 70% really churn. Logistic regression is often close; boosted trees and forests are often further off. When you need the probabilities themselves to be right (for an expected-revenue forecast rather than a ranking), scikit-learn’s CalibratedClassifierCV can rescale them. For choosing a threshold, tuning directly on the business metric sidesteps the question.

When the positive class is rare

Churn is 55% of customers, so both classes are well represented. Many real problems aren’t like that: fraud, defects, rare diseases. Take the digits dataset and ask “is this a 9?”. Only 10% of images are, so a model that always answers “no” scores 90% accuracy.

from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, digit = load_digits(return_X_y=True)
y = (digit == 9).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
print("share of 9s:", round(y.mean(), 3))

for weights in [None, "balanced"]:
    m = make_pipeline(StandardScaler(), LogisticRegression(class_weight=weights, max_iter=1000))
    m.fit(X_train, y_train)
    pred, prob = m.predict(X_test), m.predict_proba(X_test)[:, 1]
    print(f"class_weight={weights!s:9} accuracy {m.score(X_test, y_test):.3f}  "
          f"precision {precision_score(y_test, pred):.2f}  recall {recall_score(y_test, pred):.2f}  "
          f"avg precision {average_precision_score(y_test, prob):.3f}")

Accuracy barely moves, from 0.98 to 0.97, and tells you nothing useful. Precision and recall tell the real story. class_weight="balanced" weights each class inversely to its frequency during training, so a missed 9 costs about nine times as much as a false alarm. Recall rises from 0.82 to 0.91, precision falls from 0.97 to 0.82. Average precision (the area under the precision-recall curve) summarises ranking quality with a focus on the rare class; it’s usually more informative than ROC AUC when positives are scarce.

You’ve been comparing models and thresholds on one 100-customer test set and hedging every gap as “maybe noise”. Section 4 replaces that single split with cross-validation, so you can finally say how big a difference has to be before it’s real.