Classification Metrics Beyond Accuracy

Logistic regression and the 80-day rule both score 0.72 accuracy on the churn test set. Are they equally good? Accuracy says yes. Marketing would disagree as soon as they see who each one flags. Accuracy blends two very different mistakes into one number, and throws away the ranking that made the call list useful. This lesson takes them apart.

Four outcomes

Every prediction on a yes/no problem lands in one of four cells. Call churners positives (they’re the class you’re trying to find).

• True positive (TP): predicted churn, did churn. Marketing reached the right person.
• False positive (FP): predicted churn, stayed. An offer wasted on a loyal customer.
• False negative (FN): predicted stay, churned. A customer lost without a fight.
• True negative (TN): predicted stay, stayed. Correctly left alone.

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
pred = model.predict(X_test)
rule = (X_test["days_since_last_order"] > 80).astype(int)

print("model:\n", confusion_matrix(y_test, pred))
print("rule:\n", confusion_matrix(y_test, rule))

scikit-learn lays the matrix out with actual classes as rows and predicted classes as columns, both in sorted order (0 then 1). So the model’s [[35, 10], [18, 37]] reads like this:

The rule’s matrix is [[31, 14], [14, 41]]. Same 72 correct predictions, different mistakes: the model wastes fewer offers (10 against 14) and misses more churners (18 against 14). Which is better depends on what each mistake costs, and accuracy can’t tell you.

Precision, recall and F1

Two ratios turn the matrix into questions a manager asks:

• Precision = TP / (TP + FP): of the customers we flagged, how many were really leaving? The model: 37 / 47 = 0.79. The rule: 0.75.
• Recall = TP / (TP + FN): of the customers who left, how many did we flag? The model: 37 / 55 = 0.67. The rule: 0.75.

There’s a built-in tension. Flag more people and recall rises while precision usually falls; flag fewer and the reverse. F1 is the harmonic mean of the two, 2 * P * R / (P + R). It’s high only when both are, so a model can’t game it by maximising one. Use it when you need a single number and both errors matter roughly equally.

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)

print(classification_report(y_test, model.predict(X_test), target_names=["stayed", "churned"]))
print("model ROC AUC:", round(roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]), 3))
print("rule  ROC AUC:", round(roc_auc_score(y_test, X_test["days_since_last_order"]), 3))

classification_report prints precision, recall and F1 for each class, with support (the number of true cases). Read the “churned” row for the business question; the “stayed” row is the same analysis with the roles swapped.

ROC AUC: grading the ranking

Precision and recall depend on the 0.5 cut-off. ROC AUC doesn’t use a cut-off at all. It asks: if you pick one churner and one stayer at random, how often does the model give the churner the higher score? 0.5 means the scores are no better than a coin; 1.0 means every churner outranks every stayer.

roc_auc_score takes scores, not labels: pass predict_proba(...)[:, 1]. Any score that orders customers works, so you can grade the rule by its raw days_since_last_order. The rule gets 0.77; logistic regression gets 0.81. That gap is the real difference between them, hidden by the accuracy tie: the model’s ranking is meaningfully better, which is why its top-10 list in the logistic regression lesson was so accurate.

Choosing a metric

Start from the cost of each mistake, then pick:

Precision and recall moved in opposite directions between the model and the rule because they effectively use different cut-offs. You can move that cut-off yourself, on purpose, and that’s next lesson.