Using Learning Rate Schedule in PyTorch Training

Training a neural network or large deep learning model is a difficult optimization task.

The classical algorithm to train neural networks is called stochastic gradient descent. It has been well established that you can achieve increased performance and faster training on some problems by using a learning rate that changes during training.

In this post, you will discover what is learning rate schedule and how you can use different learning rate schedules for your neural network models in PyTorch.

After reading this post, you will know:

• The role of learning rate schedule in model training
• How to use learning rate schedule in PyTorch training loop
• How to set up your own learning rate schedule