Perform low-rank neural network reparameterization and its stable training in a compressed form
Spectral Tensor Train Parameterization of Deep Learning Layers
This repository is the official implementation of our AISTATS 2021 paper titled “Spectral Tensor Train Parameterization of Deep Learning Layers” by Anton Obukhov, Maxim Rakhuba, Alexander Liniger, Zhiwu Huang, Stamatios Georgoulis, Dengxin Dai, and Luc Van Gool [arXiv] [PMLR].
It demonstrates how to perform low-rank neural network reparameterization and its stable training in a compressed form. The code provides all experiments (GAN and Image Classification) from the paper (see configs/aistats21 directory) with the following types of reparameterizations: SNGAN, SRGAN, SVDP, or STTP.
Installation
All experiments can be reproduced on a single 11Gb GPU.
Clone the repository, then create a new virtual environment, and install python dependencies into it:
python3 -m venv venv_sttp
source venv_sttp/bin/activate
pip3