Last Updated on August 6, 2019 Deep learning neural networks are challenging to configure and train. There are decades of tips and tricks spread across hundreds of research papers, source code, and in the heads of academics and practitioners. The book “Neural Networks: Tricks of the Trade” originally published in 1998 and updated in 2012 at the cusp of the deep learning renaissance ties together the disparate tips and tricks into a single volume. It includes advice that is required […]

Last Updated on August 6, 2019 Neural network models are trained using stochastic gradient descent and model weights are updated using the backpropagation algorithm. The optimization solved by training a neural network model is very challenging and although these algorithms are widely used because they perform so well in practice, there are no guarantees that they will converge to a good model in a timely manner. The challenge of training neural networks really comes down to the challenge of configuring […]

Last Updated on August 6, 2019 Deep learning neural networks are relatively straightforward to define and train given the wide adoption of open source libraries. Nevertheless, neural networks remain challenging to configure and train. In his 2012 paper titled “Practical Recommendations for Gradient-Based Training of Deep Architectures” published as a preprint and a chapter of the popular 2012 book “Neural Networks: Tricks of the Trade,” Yoshua Bengio, one of the fathers of the field of deep learning, provides practical recommendations […]

Last Updated on August 6, 2019 A learning curve is a plot of model learning performance over experience or time. Learning curves are a widely used diagnostic tool in machine learning for algorithms that learn from a training dataset incrementally. The model can be evaluated on the training dataset and on a hold out validation dataset after each update during training and plots of the measured performance can created to show learning curves. Reviewing learning curves of models during training […]

Last Updated on August 6, 2019 Or, Why Stochastic Gradient Descent Is Used to Train Neural Networks. Fitting a neural network involves using a training dataset to update the model weights to create a good mapping of inputs to outputs. This training process is solved using an optimization algorithm that searches through a space of possible values for the neural network model weights for a set of weights that results in good performance on the training dataset. In this post, […]

Last Updated on August 6, 2019 Skills in deep learning are in great demand, although these skills can be challenging to identify and to demonstrate. Explaining that you are familiar with a technique or type of problem is very different to being able to use it effectively with open source APIs on real datasets. Perhaps the most effective way of demonstrating skill as a deep learning practitioner is by developing models. A practitioner can practice on standard publicly available machine […]