Image classification with synthetic gradient in Pytorch
I implement the Decoupled Neural Interfaces using Synthetic Gradients in pytorch. The paper uses synthetic gradient to decouple the layers among the network, which is pretty interesting since we won’t suffer from update lock anymore. I test my model in mnist and almost the same performance, compared to the model updated with backpropagation.
Requirement
- pytorch
- python 3.5
- torchvision
- seaborn (optional)
- matplotlib (optional)
TODO
- use multi-threading on gpu to analyze the speed
What’s synthetic gradients?
We ofter optimize NN by backpropogation, which is usually implemented in some well-known framework. However, is there another way for the layers in NN to communicate with other layers? Here comes the synthetic gradients! It gives us a way to allow neural networks to communicate, to learn to