Final assessment

Twelve questions across the whole course. Score 70% or more to pass and unlock your certificate. You can retake it as often as you like.

Finish every lesson to earn the certificate — you can take the assessment any time.

A teammate shows a fraud model with 99.1% accuracy. Fraud is 0.8% of transactions. What’s your first question?

How long did it take to train?
Which algorithm did you use?
Was the data shuffled before splitting?
What does always predicting ‘not fraud’ score, and what are the precision and recall on fraud?

Which pipeline is leak-free for cross-validating a model on data with missing values and text categories?

make_pipeline(ColumnTransformer([...imputer, scaler..., ...OneHotEncoder...]), model) passed to cross_val_score with the raw DataFrame.
Impute and encode the whole DataFrame, then pass the result and the model to cross_val_score.
Impute the training set and test set separately with their own medians.
Drop every row with a missing value, then encode the whole DataFrame.

Your churn model’s ROC AUC is 0.81 and the business wants to contact 200 customers a month. Which output should you hand over?

The 0/1 labels from predict, for every customer.
The model’s coefficients, so marketing can score customers themselves.
A list of customers ranked by churn probability, so they take the top 200.
The confusion matrix at threshold 0.5.

Python

search = GridSearchCV(pipe, {"model__max_depth": [2, 4, 8]}, cv=5)
search.fit(X_train, y_train)
print(search.score(X_test, y_test))

What does the printed number estimate?

The average validation score of the best setting across the five folds.
Performance on new data of the best setting, refit on all training rows.
The training accuracy of the deepest tree.
An average over all three depths on the test set.

A model predicts house prices. Which situation calls for RMSE rather than MAE as the main metric?

Stakeholders want a number in the target’s units.
The data contains no outliers.
You want every error, large or small, to count in proportion to its size.
Large errors are disproportionately costly, so they should weigh more than small ones.

Why does a fully grown decision tree usually score worse on new data than a random forest of such trees?

The forest’s trees are each shallower.
The forest sees more training data.
Averaging many trees with partly independent errors cancels much of each tree’s variance.
The forest uses gradient descent to tune its trees.

Logistic regression on churn: C=0.01 gives CV AUC 0.797, C=1 gives 0.807, C=100 gives 0.806, with fold spreads around 0.035. What should you conclude?

Regularisation strength barely matters here; keep the default and look for gains elsewhere.
C=1 is clearly optimal and must be documented as essential.
Small C overfits, so avoid it.
The model is broken, since tuning should change the score a lot.

Which feature is safe to use when predicting, on 30 September, whether a customer orders in the next 90 days?

Number of support tickets opened in October.
Whether the customer received a win-back email in November.
Total refunds issued in Q4.
Average days between orders over the customer’s history up to 30 September.

You need to group 50,000 free-text support tickets into topics with no labels. Which approach fits best?

Train a logistic regression on the tickets with churned as the target.
Embed the tickets with an embedding model, then cluster the vectors with k-means and profile the clusters.
Count words with TF-IDF and pick the most frequent word as each ticket’s topic.
Send all 50,000 tickets to an LLM one by one and trust its labels.

Which statement about neural networks is accurate?

They always outperform gradient boosting on tabular data.
They don’t need feature scaling because they learn their own features.
Without non-linear activations, stacking layers adds no expressive power.
An epoch is one update of the weights.

A model’s recall is 0.85 for one region and 0.65 for another, with 900 and 40 customers respectively. What’s the most responsible next step?

Conclude the model is biased against the smaller region and drop it from the rollout.
Ignore the gap because the smaller region is a small share of revenue.
Remove region from the features.
Report the gap with its uncertainty, monitor it as data accumulates, and decide with the business whether a threshold or data fix is needed.

A churn model goes live with offers to every flagged customer, and churn among flagged customers falls from 70% to 55%. What’s needed to claim the model and offer caused the drop?

A randomly held-out group of flagged customers who got no offer, compared over the same period.
A higher test ROC AUC.
Retraining on the new data.
A longer outcome window.

12 questions unanswered