Why Optimization Is Important in Machine Learning
Machine learning involves using an algorithm to learn and generalize from historical data in order to make predictions on new data.
This problem can be described as approximating a function that maps examples of inputs to examples of outputs. Approximating a function can be solved by framing the problem as function optimization. This is where a machine learning algorithm defines a parameterized mapping function (e.g. a weighted sum of inputs) and an optimization algorithm is used to fund the values of the parameters (e.g. model coefficients) that minimize the error of the function when used to map inputs to outputs.
This means that each time we fit a machine learning algorithm on a training dataset, we are solving an