{"id":"l-ml-1-4","courseId":"ml-crash-course","lang":"en","kind":"lesson","summary":"Build the two baselines every model must beat, a majority-class dummy and a one-line business rule, and judge a model by its lift over them instead of its raw score.","html":"

Your depth-3 tree predicts churn with 73% accuracy on unseen customers. Before you show that number to anyone, ask the question every experienced data scientist asks first: compared to what? Without a comparison, 73% is neither good nor bad. It’s a number.

\n

The dumbest possible model

\n

The first baseline ignores the features entirely. For classification, it predicts the most common class for everyone. For regression, it predicts the average. scikit-learn ships both as real estimators, so they fit into the same code as any model.

\n
Python
import pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.dummy import DummyClassifier\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n            \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n    churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n    stratify=churn[\"churned\"])\n\ndummy = DummyClassifier(strategy=\"most_frequent\").fit(X_train, y_train)\nprint(\"dummy predicts:\", dummy.predict(X_test)[:5])\nprint(\"dummy accuracy:\", dummy.score(X_test, y_test))
\n

Predicting “churned” for every customer is right 55% of the time, because 55% of customers churned. That’s the floor. Your tree’s 73% is 18 points above it, which is real progress, but you’re not done asking.

\n\n

The rule a person would write

\n

The second baseline is the one that hurts. Before machine learning, someone at Cartwheel would have written a rule: “if the customer hasn’t ordered in N days, they’re churning”. You choose N the way you’d train a model: on the training data only.

\n
Python
import pandas as pd\nfrom sklearn.model_selection import train_test_split\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n            \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n    churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n    stratify=churn[\"churned\"])\n\ndef rule_accuracy(days, X, y):\n    predicted = (X[\"days_since_last_order\"] > days).astype(int)\n    return (predicted == y).mean()\n\ncandidates = range(30, 301, 10)\nbest_days = max(candidates, key=lambda d: rule_accuracy(d, X_train, y_train))\nprint(\"best cut-off on training data:\", best_days, \"days\")\nprint(\"rule accuracy on test data:\", rule_accuracy(best_days, X_test, y_test))
\n

The rule gets 72% on the test set. The tree gets 73%. On 100 test customers that’s a difference of one person.

\n
\n Bar chart of test accuracy: majority-class dummy 55%, 80-day rule 72%, depth-3 tree 73%.\n \n Dummy\n \n 55%\n 80-day rule\n \n 72%\n Depth-3 tree\n \n 73%\n bar length = accuracy (0% at the axis, 100% at 610)\n
Accuracy on the same 100 test customers
\n

That isn’t a failure. It’s the most useful thing you’ve learned so far. It tells you that most of the signal for churn lives in one column, recency, and that an eight-feature tree has so far added almost nothing on top of it. It also gives you a target: from here on, a model has to beat 72%, not 55%, to be worth deploying, explaining and monitoring.

\n\n

Baselines for regression

\n

The regression counterpart is DummyRegressor, which predicts the training mean for every row. Its score (R²) on test data is close to 0, because R² is defined as improvement over predicting the mean. You’ll use it in section 2.

\n
Python
from sklearn.datasets import load_diabetes\nfrom sklearn.dummy import DummyRegressor\nfrom sklearn.model_selection import train_test_split\n\nX, y = load_diabetes(return_X_y=True, as_frame=True)\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\ndummy = DummyRegressor(strategy=\"mean\").fit(X_train, y_train)\nprint(\"predicts\", round(dummy.predict(X_test)[0], 1), \"for everyone\")\nprint(\"R² on test:\", round(dummy.score(X_test, y_test), 3))
\n

Lift, not score

\n

From now on, report a model as lift over its baselines: “73% accuracy, against 55% for the majority class and 72% for the 80-day rule”. That one sentence tells a reader more than any single number, and it protects you from fooling yourself.

\n

Keep the baselines in your code, next to the models, and rerun them whenever the data or the split changes. A baseline computed once on an old split and quoted forever is worse than none, because it looks like evidence. In practice the comparison table at the end of a notebook should have the dummy at the top, the best rule below it, and every model after that, all scored on exactly the same test rows.

\n

Be honest about noise, too. With 100 test customers, one customer is one percentage point, so differences of two or three points can appear or vanish with a different shuffle. Section 4 replaces the single split with cross-validation, which tells you how much a score wobbles. Until then, treat small gaps as ties.

\n

Accuracy is also not the only way to compare. A model that ranks customers by churn risk can hand marketing the 30 most likely churners, which a yes/no rule can’t do. Section 3 measures exactly that. First, section 2 opens up the simplest model of all, a straight line, to see what “learning” means in numbers.

\n","toc":[{"id":"sec-1","text":"The dumbest possible model","level":2},{"id":"sec-2","text":"The rule a person would write","level":2},{"id":"sec-3","text":"Baselines for regression","level":2},{"id":"sec-4","text":"Lift, not score","level":2}],"takeaways":["A score means nothing until you compare it with a baseline built on the same split.","DummyClassifier predicts the most common class and DummyRegressor predicts the mean; they set the floor.","A simple rule tuned on the training data is the baseline that keeps you honest about whether a model is worth its cost.","When one class dominates, accuracy rewards ignoring the rare class; a 95% score can mean the model found nothing."],"further":[{"title":"Dummy estimators","url":"https://scikit-learn.org/stable/modules/model_evaluation.html#dummy-estimators"},{"title":"DummyClassifier","url":"https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html"}],"quiz":[{"q":"

A fraud model scores 97% accuracy. 3% of transactions are fraud. What should you check first?

","options":[{"text":"Whether the model beats a dummy that predicts “not fraud” for everything.","why":"Correct. That dummy also scores 97%, so the model may have learned nothing about fraud."},{"text":"Whether 97% is higher than the industry average for fraud models.","why":"Other companies’ numbers come from other data and class balances. The comparison that matters is on your own split."},{"text":"Whether the training accuracy is also around 97%.","why":"Comparing train and test checks overfitting, but both can be 97% while the model never catches a single fraud."}],"answer":0},{"q":"

Your rule baseline uses a cut-off of 80 days. How should that 80 have been chosen?

","options":[{"text":"By trying several cut-offs and keeping the one with the best test accuracy.","why":"That tunes the baseline on the test set, making it look better than it would on new data, which is the same mistake as tuning a model on it."},{"text":"By trying several cut-offs on the training data and keeping the best one there.","why":"Correct. The baseline gets the same treatment as a model; the test set only measures the final result."},{"text":"It doesn’t matter for a baseline, since baselines aren’t shipped.","why":"Rules often are shipped, and an unfairly tuned baseline gives you a wrong verdict on whether the model is worth it."}],"answer":1},{"q":"

DummyRegressor() on the diabetes data scores an R² of about 0 on the test set. Why?

","options":[{"text":"The dummy failed to converge.","why":"There’s nothing to converge. The dummy computes one number, the training mean."},{"text":"The diabetes target has no pattern in it.","why":"The dummy never looks at the features, so its score says nothing about whether a pattern exists."},{"text":"R² measures improvement over predicting the mean, and the dummy predicts the mean.","why":"Correct. R² is 0 for the mean predictor by design; a tiny negative number appears because the training mean differs slightly from the test mean."}],"answer":2},{"q":"

A tree beats the majority-class dummy by 18 points but a one-line rule by 1 point. What is the most defensible next step?

","options":[{"text":"Ship the tree, since it has the highest score.","why":"One point on 100 test customers is one customer. That isn’t evidence the tree is better, and the rule is easier to run and explain."},{"text":"Report that the tree is 18 points better than the baseline.","why":"Quoting only the weakest baseline hides that a much simpler option does as well. That’s the hype this lesson warns about."},{"text":"Keep the rule as the benchmark and only switch when a model beats it clearly and repeatably.","why":"Correct. Section 4’s cross-validation tells you whether a gap is real or noise."}],"answer":2}],"exercise":{"kind":"python","files":["churn.csv"],"starter":"import pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.dummy import DummyClassifier\nfrom sklearn.neighbors import KNeighborsClassifier\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\n\n# 1. Majority-class baseline: test accuracy\ndummy_acc = None\n\n# 2. Rule baseline: churned if days_since_last_order > best_days.\n# Pick best_days from range(30, 301, 10) using the TRAINING data only.\nbest_days = None\nrule_acc = None # the rule's accuracy on the test data\n\n# 3. A k-nearest-neighbours model (default settings): test accuracy\nknn_acc = None\n","solution":"import pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.dummy import DummyClassifier\nfrom sklearn.neighbors import KNeighborsClassifier\n\nchurn = pd.read_csv(\"churn.csv\")\nfeatures = [\"tenure_days\", \"orders\", \"total_spent\", \"avg_order_value\",\n \"days_since_last_order\", \"used_coupon\", \"support_tickets\", \"mobile_share\"]\nX_train, X_test, y_train, y_test = train_test_split(\n churn[features], churn[\"churned\"], test_size=0.25, random_state=42,\n stratify=churn[\"churned\"])\n\ndummy_acc = DummyClassifier(strategy=\"most_frequent\").fit(X_train, y_train).score(X_test, y_test)\n\ndef rule_accuracy(days, X, y):\n return ((X[\"days_since_last_order\"] > days).astype(int) == y).mean()\n\nbest_days = max(range(30, 301, 10), key=lambda d: rule_accuracy(d, X_train, y_train))\nrule_acc = rule_accuracy(best_days, X_test, y_test)\n\nknn_acc = KNeighborsClassifier().fit(X_train, y_train).score(X_test, y_test)\nprint(dummy_acc, best_days, rule_acc, knn_acc)\n","checks":[{"id":"dummy","test":"dummy_acc is not None and abs(float(dummy_acc) - 0.55) < 0.005"},{"id":"best-days","test":"best_days in range(30, 301, 10) and abs(((X_train['days_since_last_order'] > best_days).astype(int) == y_train).mean() - max(((X_train['days_since_last_order'] > d).astype(int) == y_train).mean() for d in range(30, 301, 10))) < 1e-9"},{"id":"rule-acc","test":"rule_acc is not None and abs(float(rule_acc) - ((X_test['days_since_last_order'] > best_days).astype(int) == y_test).mean()) < 1e-9"},{"id":"knn","test":"knn_acc is not None and abs(float(knn_acc) - 0.72) < 0.005"}],"title":"Beat the baselines, or don't","prompt":"

Put a k-nearest-neighbours model next to its two baselines on the same split.

\n
    \n
  1. dummy_acc: test accuracy of a DummyClassifier that predicts the most frequent class.
    2. \n
  2. best_days: the cut-off from range(30, 301, 10) that gives the best accuracy on the training data for the rule “churned if days_since_last_order > cut-off”. Then rule_acc: that rule’s accuracy on the test data.
    3. \n
  3. knn_acc: test accuracy of KNeighborsClassifier() with default settings.
    4. \n
\n","hints":["DummyClassifier(strategy="most_frequent").fit(X_train, y_train).score(X_test, y_test)","Write a helper rule_accuracy(days, X, y) that compares (X["days_since_last_order"] > days).astype(int) with y and takes the mean.","max(range(30, 301, 10), key=lambda d: rule_accuracy(d, X_train, y_train)) picks the cut-off using training data only."],"explanation":"

The dummy scores 0.55, the 80-day rule 0.72 and k-NN 0.72. k-nearest neighbours, with eight features, does no better than one line of logic on one column. That’s a real finding: either the other features carry little extra signal, or this model can’t use it (k-NN is distorted by features on very different scales, which you’ll fix in section 2). Choosing the cut-off on the training data keeps the comparison fair.

\n","checkLabels":{"dummy":"dummy_acc is the majority-class test accuracy","best-days":"best_days is the best cut-off on the training data","rule-acc":"rule_acc is that rule’s test accuracy","knn":"knn_acc is the k-NN model’s test accuracy"}},"words":668,"readMinutes":3}