Machine Learning

A Gentle Introduction to Probability Scoring Methods in Python

Last Updated on December 31, 2019

How to Score Probability Predictions in Python and
Develop an Intuition for Different Metrics.

Predicting probabilities instead of class labels for a classification problem can provide additional nuance and uncertainty for the predictions.

The added nuance allows more sophisticated metrics to be used to interpret and evaluate the predicted probabilities. In general, methods for the evaluation of the accuracy of predicted probabilities are referred to as scoring rules or scoring functions.

In this tutorial, you will discover three scoring methods that you can use to evaluate the predicted probabilities on your classification predictive modeling problem.

After completing this tutorial, you will know:

  • The log loss score that heavily penalizes predicted probabilities far away from their expected value.
  • The Brier score that is gentler than log loss but still penalizes proportional to the distance from the expected value.
  • The area under ROC curve that summarizes the likelihood of the model predicting a higher probability for true positive cases than true negative cases.

Kick-start your project with my new book Probability for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.