Checkpoint: Classification

Five questions on this section. Answer all of them, then see your score and why each answer is right or wrong.

A fraud model must catch at least 90% of fraud; analysts can review up to 500 alerts a day. Which pair of numbers do you need from validation to choose a threshold?

Accuracy and ROC AUC.
Training and test accuracy.
Precision and F1 at the default threshold.
Recall and the number of flagged cases at each candidate threshold.

A random forest uses 500 fully grown trees. Which change most directly reduces its training time without usually hurting accuracy much?

Set max_depth=1 so each tree is a single split.
Remove the bootstrap sampling.
Use fewer trees, such as 200, since the gains from extra trees flatten out.
Scale the features first.

Python

prob = model.predict_proba(X_test)
print(roc_auc_score(y_test, prob))

For a binary problem this raises an error. What’s the fix?

Use model.predict(X_test) instead.
Pass the probability of the positive class: prob[:, 1].
Sort y_test before calling the metric.
Convert prob to a list.

A tree-based churn model and a logistic regression both have ROC AUC 0.80 on the same folds. The business needs probabilities that mean what they say, to forecast how many customers will leave. Which is the safer first choice?

Logistic regression, whose probabilities are usually closer to calibrated out of the box.
The tree model, because trees always produce calibrated probabilities.
Either; equal AUC means equal probabilities.
Neither; probabilities can’t be used for counts.

In a dataset where 2% of loans default, a model with class_weight="balanced" raises recall from 0.40 to 0.75 and lowers precision from 0.60 to 0.20. What happened?

The model got worse at everything and should be discarded.
The weights leaked the test labels into training.
Mistakes on defaults now cost more during training, so the model flags many more loans: it catches more defaults but with more false alarms.
Accuracy must have risen, since recall rose.

5 questions unanswered