Baselines: The Model to Beat

Your depth-3 tree predicts churn with 73% accuracy on unseen customers. Before you show that number to anyone, ask the question every experienced data scientist asks first: compared to what? Without a comparison, 73% is neither good nor bad. It’s a number.

The dumbest possible model

The first baseline ignores the features entirely. For classification, it predicts the most common class for everyone. For regression, it predicts the average. scikit-learn ships both as real estimators, so they fit into the same code as any model.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])

dummy = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
print("dummy predicts:", dummy.predict(X_test)[:5])
print("dummy accuracy:", dummy.score(X_test, y_test))

Predicting “churned” for every customer is right 55% of the time, because 55% of customers churned. That’s the floor. Your tree’s 73% is 18 points above it, which is real progress, but you’re not done asking.

The rule a person would write

The second baseline is the one that hurts. Before machine learning, someone at Cartwheel would have written a rule: “if the customer hasn’t ordered in N days, they’re churning”. You choose N the way you’d train a model: on the training data only.

import pandas as pd
from sklearn.model_selection import train_test_split

churn = pd.read_csv("churn.csv")
features = ["tenure_days", "orders", "total_spent", "avg_order_value",
            "days_since_last_order", "used_coupon", "support_tickets", "mobile_share"]
X_train, X_test, y_train, y_test = train_test_split(
    churn[features], churn["churned"], test_size=0.25, random_state=42,
    stratify=churn["churned"])

def rule_accuracy(days, X, y):
    predicted = (X["days_since_last_order"] > days).astype(int)
    return (predicted == y).mean()

candidates = range(30, 301, 10)
best_days = max(candidates, key=lambda d: rule_accuracy(d, X_train, y_train))
print("best cut-off on training data:", best_days, "days")
print("rule accuracy on test data:", rule_accuracy(best_days, X_test, y_test))

The rule gets 72% on the test set. The tree gets 73%. On 100 test customers that’s a difference of one person.

That isn’t a failure. It’s the most useful thing you’ve learned so far. It tells you that most of the signal for churn lives in one column, recency, and that an eight-feature tree has so far added almost nothing on top of it. It also gives you a target: from here on, a model has to beat 72%, not 55%, to be worth deploying, explaining and monitoring.

Baselines for regression

The regression counterpart is DummyRegressor, which predicts the training mean for every row. Its score (R²) on test data is close to 0, because R² is defined as improvement over predicting the mean. You’ll use it in section 2.

from sklearn.datasets import load_diabetes
from sklearn.dummy import DummyRegressor
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
dummy = DummyRegressor(strategy="mean").fit(X_train, y_train)
print("predicts", round(dummy.predict(X_test)[0], 1), "for everyone")
print("R² on test:", round(dummy.score(X_test, y_test), 3))

Lift, not score

From now on, report a model as lift over its baselines: “73% accuracy, against 55% for the majority class and 72% for the 80-day rule”. That one sentence tells a reader more than any single number, and it protects you from fooling yourself.

Keep the baselines in your code, next to the models, and rerun them whenever the data or the split changes. A baseline computed once on an old split and quoted forever is worse than none, because it looks like evidence. In practice the comparison table at the end of a notebook should have the dummy at the top, the best rule below it, and every model after that, all scored on exactly the same test rows.

Be honest about noise, too. With 100 test customers, one customer is one percentage point, so differences of two or three points can appear or vanish with a different shuffle. Section 4 replaces the single split with cross-validation, which tells you how much a score wobbles. Until then, treat small gaps as ties.

Accuracy is also not the only way to compare. A model that ranks customers by churn risk can hand marketing the 30 most likely churners, which a yes/no rule can’t do. Section 3 measures exactly that. First, section 2 opens up the simplest model of all, a straight line, to see what “learning” means in numbers.