Machine Learning

Essence of Boosting Ensembles for Machine Learning

Boosting is a powerful and popular class of ensemble learning techniques. Historically, boosting algorithms were challenging to implement, and it was not until AdaBoost demonstrated how to implement boosting that the technique could be used effectively. AdaBoost and modern gradient boosting work by sequentially adding models that correct the residual prediction errors of the model. As such, boosting methods are known to be effective, but constructing models can be slow, especially for large datasets. More recently, extensions designed for computational […]

Read more
Machine Learning

A Gentle Introduction to Multiple-Model Machine Learning

An ensemble learning method involves combining the predictions from multiple contributing models. Nevertheless, not all techniques that make use of multiple machine learning models are ensemble learning algorithms. It is common to divide a prediction problem into subproblems. For example, some problems naturally subdivide into independent but related subproblems and a machine learning model can be prepared for each. It is less clear whether these represent examples of ensemble learning, although we might distinguish these methods from ensembles given the […]

Read more
Machine Learning

A Gentle Introduction to Ensemble Diversity for Machine Learning

Ensemble learning combines the predictions from machine learning models for classification and regression. We pursue using ensemble methods to achieve improved predictive performance, and it is this improvement over any of the contributing models that defines whether an ensemble is good or not. A property that is present in a good ensemble is the diversity of the predictions made by contributing models. Diversity is a slippery concept as it has not been precisely defined; nevertheless, it provides a useful practical […]

Read more
Machine Learning

Essence of Bootstrap Aggregation Ensembles

Bootstrap aggregation, or bagging, is a popular ensemble method that fits a decision tree on different bootstrap samples of the training dataset. It is simple to implement and effective on a wide range of problems, and importantly, modest extensions to the technique result in ensemble methods that are among some of the most powerful techniques, like random forest, that perform well on a wide range of predictive modeling problems. As such, we can generalize the bagging method to a framework […]

Read more
Machine Learning

A Gentle Introduction to the BFGS Optimization Algorithm

The Broyden, Fletcher, Goldfarb, and Shanno, or BFGS Algorithm, is a local search optimization algorithm. It is a type of second-order optimization algorithm, meaning that it makes use of the second-order derivative of an objective function and belongs to a class of algorithms referred to as Quasi-Newton methods that approximate the second derivative (called the Hessian) for optimization problems where the second derivative cannot be calculated. The BFGS algorithm is perhaps one of the most widely used second-order algorithms for […]

Read more
Machine Learning

Dual Annealing Optimization With Python

Dual Annealing is a stochastic global optimization algorithm. It is an implementation of the generalized simulated annealing algorithm, an extension of simulated annealing. In addition, it is paired with a local search algorithm that is automatically performed at the end of the simulated annealing procedure. This combination of effective global and local search procedures provides a powerful algorithm for challenging nonlinear optimization problems. In this tutorial, you will discover the dual annealing global optimization algorithm. After completing this tutorial, you […]

Read more
Machine Learning

Gradient Descent With RMSProp from Scratch

Gradient descent is an optimization algorithm that follows the negative gradient of an objective function in order to locate the minimum of the function. A limitation of gradient descent is that it uses the same step size (learning rate) for each input variable. AdaGrad, for short, is an extension of the gradient descent optimization algorithm that allows the step size in each dimension used by the optimization algorithm to be automatically adapted based on the gradients seen for the variable […]

Read more
Machine Learning

Line Search Optimization With Python

The line search is an optimization algorithm that can be used for objective functions with one or more variables. It provides a way to use a univariate optimization algorithm, like a bisection search on a multivariate objective function, by using the search to locate the optimal step size in each dimension from a known point to the optima. In this tutorial, you will discover how to perform a line search optimization in Python. After completing this tutorial, you will know: […]

Read more
Machine Learning

One-Dimensional (1D) Test Functions for Function Optimization

Function optimization is a field of study that seeks an input to a function that results in the maximum or minimum output of the function. There are a large number of optimization algorithms and it is important to study and develop intuitions for optimization algorithms on simple and easy-to-visualize test functions. One-dimensional functions take a single input value and output a single evaluation of the input. They may be the simplest type of test function to use when studying function […]

Read more
Machine Learning

A Gentle Introduction to Function Optimization

Function optimization is a foundational area of study and the techniques are used in almost every quantitative field. Importantly, function optimization is central to almost all machine learning algorithms, and predictive modeling projects. As such, it is critical to understand what function optimization is, the terminology used in the field, and the elements that constitute a function optimization problem. In this tutorial, you will discover a gentle introduction to function optimization. After completing this tutorial, you will know: The three […]

Read more
1 … 36 37 38 39 40 … 907