Official Python package for Deep Kernel Shaping (DKS) and Tailored Activation Transformations (TAT)
This Python package implements the activation function transformations andweight initializations used Deep Kernel Shaping (DKS) and Tailored ActivationTransformations (TAT). DKS and TAT, which were introduced in the DKS paper andTAT paper, are methods constructing/transforming neural networks to make themmuch easier to train. For example, these methods can be used in conjunction withK-FAC to train deep vanilla deep convnets (without skip connections ornormalization layers) as fast as standard ResNets of the same depth. The package supports the JAX, PyTorch, and TensorFlow […]
Read more