Logistic regression and the 80-day rule both score 0.72 accuracy on the churn test set. Are they equally good? Accuracy says yes. Marketing would disagree as soon as they see who each one flags. Accuracy blends two very different mistakes into one number, and throws away the ranking that made the call list useful. This lesson takes them apart.
\nEvery prediction on a yes/no problem lands in one of four cells. Call churners positives (they’re the class you’re trying to find).
\nimport pandas as pd\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\nmodel = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)\npred = model.predict(X_test)\nrule = (X_test[\"days_since_last_order\"] > 80).astype(int)\n\nprint(\"model:\\n\", confusion_matrix(y_test, pred))\nprint(\"rule:\\n\", confusion_matrix(y_test, rule))scikit-learn lays the matrix out with actual classes as rows and predicted classes as columns, both in sorted order (0 then 1). So the model’s [[35, 10], [18, 37]] reads like this:
The rule’s matrix is [[31, 14], [14, 41]]. Same 72 correct predictions, different mistakes: the model wastes fewer offers (10 against 14) and misses more churners (18 against 14). Which is better depends on what each mistake costs, and accuracy can’t tell you.
Two ratios turn the matrix into questions a manager asks:
\nThere’s a built-in tension. Flag more people and recall rises while precision usually falls; flag fewer and the reverse. F1 is the harmonic mean of the two, 2 * P * R / (P + R). It’s high only when both are, so a model can’t game it by maximising one. Use it when you need a single number and both errors matter roughly equally.
import pandas as pd\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.metrics import classification_report, roc_auc_score\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\nmodel = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)\n\nprint(classification_report(y_test, model.predict(X_test), target_names=[\"stayed\", \"churned\"]))\nprint(\"model ROC AUC:\", round(roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]), 3))\nprint(\"rule ROC AUC:\", round(roc_auc_score(y_test, X_test[\"days_since_last_order\"]), 3))classification_report prints precision, recall and F1 for each class, with support (the number of true cases). Read the “churned” row for the business question; the “stayed” row is the same analysis with the roles swapped.
Precision and recall depend on the 0.5 cut-off. ROC AUC doesn’t use a cut-off at all. It asks: if you pick one churner and one stayer at random, how often does the model give the churner the higher score? 0.5 means the scores are no better than a coin; 1.0 means every churner outranks every stayer.
\nroc_auc_score takes scores, not labels: pass predict_proba(...)[:, 1]. Any score that orders customers works, so you can grade the rule by its raw days_since_last_order. The rule gets 0.77; logistic regression gets 0.81. That gap is the real difference between them, hidden by the accuracy tie: the model’s ranking is meaningfully better, which is why its top-10 list in the logistic regression lesson was so accurate.
Start from the cost of each mistake, then pick:
\n| Situation | \nWatch | \nWhy | \n
|---|---|---|
| Missing a case is costly (cancer screening, fraud) | \nRecall | \nEvery false negative is the expensive error | \n
| Acting on a case is costly (pricey offers, account bans) | \nPrecision | \nEvery false positive wastes money or trust | \n
| You’ll act on the top N by score | \nROC AUC, or the hit rate in the top N | \nRanking quality is what matters | \n
| Both errors matter, one number needed | \nF1 | \nPunishes neglecting either | \n
Precision and recall moved in opposite directions between the model and the rule because they effectively use different cut-offs. You can move that cut-off yourself, on purpose, and that’s next lesson.
\n","toc":[{"id":"sec-1","text":"Four outcomes","level":2},{"id":"sec-2","text":"Precision, recall and F1","level":2},{"id":"sec-3","text":"ROC AUC: grading the ranking","level":2},{"id":"sec-4","text":"Choosing a metric","level":2}],"takeaways":["A confusion matrix counts true negatives, false positives, false negatives and true positives; in scikit-learn rows are actual classes and columns are predicted classes.","Precision is the share of flagged cases that are real; recall is the share of real cases that get flagged.","F1 is the harmonic mean of precision and recall, useful when you need one number that punishes neglecting either.","ROC AUC measures ranking quality across all thresholds: the chance that a random positive gets a higher score than a random negative.","Always check which class is the positive one: inload_breast_cancer, malignant is 0, so recall for malignant needs pos_label=0."],"further":[{"title":"Classification metrics","url":"https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics"},{"title":"confusion_matrix","url":"https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html"},{"title":"roc_auc_score","url":"https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html"}],"quiz":[{"q":"confusion_matrix(y_test, pred) returns [[35, 10], [18, 37]] for churn (1 = churned). How many churners did the model miss?
Each win-back offer costs $40, so marketing only wants to contact customers who are very likely to churn. Which metric should you watch most closely?
","options":[{"text":"Precision, because it measures how many contacted customers really were churning.","why":"Correct. Every false positive is $40 spent on a customer who would have stayed anyway."},{"text":"Recall, because it counts how many churners you reach.","why":"Recall rewards contacting more people, which is exactly what an expensive offer argues against."},{"text":"Accuracy, because it balances both kinds of error.","why":"Accuracy weighs both errors equally, but here false positives cost money and false negatives cost nothing extra."}],"answer":0},{"q":"Model A has ROC AUC 0.81 and model B has 0.77, but both have accuracy 0.72 at the default threshold. What does that tell you?
","options":[{"text":"The AUC difference must be a calculation error, since accuracy is equal.","why":"The two metrics measure different things. Accuracy looks at one threshold; AUC looks at the whole ranking."},{"text":"B is better because it reaches the same accuracy with a weaker ranking.","why":"A weaker ranking is a disadvantage; it means B’s scores separate churners from stayers less well."},{"text":"A ranks customers better, so it will give a better call list at most list sizes.","why":"Correct. AUC summarises ranking quality, which is what a top-N list depends on."}],"answer":2},{"q":"A model on load_breast_cancer reports recall_score(y_test, pred) = 0.99. Why might that number mislead a doctor?
pos_label=0 (or read the per-class report) to get recall for malignant."},{"text":"Recall ignores the false negatives, which are what matter in medicine.","why":"Recall is built from false negatives; the problem is which class it’s computed for."}],"answer":1}],"exercise":{"kind":"python","starter":"from sklearn.datasets import load_breast_cancer\nfrom sklearn.metrics import confusion_matrix, precision_score, recall_score, roc_auc_score\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.tree import DecisionTreeClassifier\n\ndata = load_breast_cancer()\nprint(data.target_names) # index 0 = malignant, 1 = benign\nX, y = load_breast_cancer(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)\n\nmodel = DecisionTreeClassifier(max_depth=2, random_state=42).fit(X_train, y_train)\npred = model.predict(X_test)\n\n# 1. Confusion matrix (rows = actual, columns = predicted)\ncm = None\n\n# 2. Recall and precision for MALIGNANT tumours\nrecall_malignant = recall_score(y_test, pred)\nprecision_malignant = precision_score(y_test, pred)\n\n# 3. ROC AUC of the model's scores\nauc = None\n","solution":"from sklearn.datasets import load_breast_cancer\nfrom sklearn.metrics import confusion_matrix, precision_score, recall_score, roc_auc_score\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.tree import DecisionTreeClassifier\n\ndata = load_breast_cancer()\nprint(data.target_names) # index 0 = malignant, 1 = benign\nX, y = load_breast_cancer(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)\n\nmodel = DecisionTreeClassifier(max_depth=2, random_state=42).fit(X_train, y_train)\npred = model.predict(X_test)\n\ncm = confusion_matrix(y_test, pred)\n\nrecall_malignant = recall_score(y_test, pred, pos_label=0)\nprecision_malignant = precision_score(y_test, pred, pos_label=0)\n\nauc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])\nprint(cm, recall_malignant, precision_malignant, auc)\n","checks":[{"id":"cm","test":"cm is not None and [list(map(int, r)) for r in cm] == [[50, 3], [10, 80]]"},{"id":"recall","test":"abs(float(recall_malignant) - 50 / 53) < 1e-6"},{"id":"precision","test":"abs(float(precision_malignant) - 50 / 60) < 1e-6"},{"id":"auc","test":"auc is not None and 0.9 < float(auc) < 0.97"}],"title":"Metrics for the class that matters","prompt":"A depth-2 tree screens breast tumours. In this dataset 0 means malignant and 1 means benign. Measure the model from a doctor’s point of view.
\ncm: the confusion matrix of y_test against pred.recall_malignant and precision_malignant: recall and precision with malignant as the positive class.auc: ROC AUC from the model’s predicted probabilities.confusion_matrix(y_test, pred): true labels first.","The starter computes the metrics for label 1 (benign). Add pos_label=0.","roc_auc_score wants scores for the positive label of y_test (1). Use model.predict_proba(X_test)[:, 1], not predict."],"explanation":"The matrix [[50, 3], [10, 80]] shows 3 malignant tumours called benign: the dangerous error. Recall for malignant is 50/53 ≈ 0.94; precision is 50/60 ≈ 0.83, because 10 benign tumours were flagged. With the default pos_label=1 you’d have reported benign recall (0.89) as if it were about cancer. AUC is about 0.94: the tree’s four leaves give a coarse but useful ranking.