Bunch of optimizer implementations in PyTorch
Bunch of optimizer implementations in PyTorch with clean-code, strict types. Also, including useful optimization ideas.
Most of the implementations are based on the original paper, but I added some tweaks.
Documentation
https://pytorch-optimizers.readthedocs.io/en/latest/
Usage
Install
$ pip3 install pytorch-optimizer
Simple Usage
from pytorch_optimizer import Ranger21 ... model = YourModel() optimizer = Ranger21(model.parameters()) ... for input, output in data: optimizer.zero_grad() loss = loss_function(output, model(input)) loss.backward() optimizer.step()
Supported Optimizers
Useful Resources
Several optimization ideas to regularize & stabilize the training. Most of the ideas are applied in Ranger21 optimizer.
Also, most