Probabilistic Model Selection with AIC, BIC, and MDL

Last Updated on August 28, 2020

Model selection is the problem of choosing one from among a set of candidate models.

It is common to choose a model that performs the best on a hold-out test dataset or to estimate model performance using a resampling technique, such as k-fold cross-validation.

An alternative approach to model selection involves using probabilistic statistical measures that attempt to quantify both the model performance on the training dataset and the complexity of the model. Examples include the Akaike and Bayesian Information Criterion and the Minimum Description Length.

The benefit of these information criterion statistics is that they do not require a hold-out test set, although a limitation is that they do not take the uncertainty of the models into account and may end-up selecting models that are too simple.

In this post, you will discover probabilistic statistics for machine learning model selection.

After reading this post, you will know:

• Model selection is the challenge of choosing one among a set of candidate models.
• Akaike and Bayesian Information Criterion are two ways of scoring a model based on its log-likelihood and complexity.
• Minimum Description Length provides another scoring method from information theory that
  To finish reading, please visit source site