This is reimplementation of Trajectory Transformer, introduced in Offline Reinforcement Learningas One Big Sequence Modeling Problem paper. The original implementation has few problems with inference speed, namely quadratic attention duringinference and sequential rollouts. The former slows down planning a lot, while the latter does notallow to do rollouts in parallel and utilize GPU to the full. Still, even after all changes, it is not that fast compared to traditional methods such as PPO or SAC/DDPG.However, the gains are huge, what […]

This is the artifact for the ICSE 2022 paper “ReMoS: Reducing Defect Inheritance in Transfer Learning via Relevant Model Slicing”. Transfer learning is a popular software reuse technique in the deep learning community that enables developers to build custom models (students) based on sophisticated pretrained models (teachers). However, like vulnerability inheritance in traditional software reuse, some defects in the teacher model may also be inherited by students, such as well-known adversarial vulnerabilities and backdoors. Reducing such defects is challenging since […]

This Python package implements the activation function transformations andweight initializations used Deep Kernel Shaping (DKS) and Tailored ActivationTransformations (TAT). DKS and TAT, which were introduced in the DKS paper andTAT paper, are methods constructing/transforming neural networks to make themmuch easier to train. For example, these methods can be used in conjunction withK-FAC to train deep vanilla deep convnets (without skip connections ornormalization layers) as fast as standard ResNets of the same depth. The package supports the JAX, PyTorch, and TensorFlow […]

LayerNorm(SmallInit(Embedding)) in a Transformer I find that when training a transformer, the embedding matrix moves slowly, hence it’s difficult for the model to jump out of the initial noisy embedding. (initial embedding) [[-0.0073 0.0062 -0.0261 … 0.0086 0.0107 -0.008 ] … ] (after 1 step, the directions of the embedding vectors are not moved much because the numbers change by ~LR = ~4e-4) [[-0.0069 0.0066 -0.0265 … 0.009 0.0111 -0.0084] … ] So I propose initializing the embedding matrix to […]

This repository contains PyTorch evaluation code, training code and pretrained EViT models for the ICLR 2022 Spotlight paper: Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, Pengtao Xie The proposed EViT models obtain competitive tradeoffs in terms of speed / precision: If you use this code for a paper please cite: @inproceedings{liang2022evit, title={Not All Patches are What You    

This repo is the official implementation of “Uniformer: Unified Transformer for Efficient Spatiotemporal Representation Learning”. It currently includes code and models for the following tasks: Updates 01/13/2022 [Initial commits]: Pretrained models on ImageNet-1K, Kinetics-400, Kinetics-600, Something-Something V1&V2 The supported code and models for image classification and video classification are provided. Introduction UniFormer (Unified transFormer) is introduce in arxiv, which effectively unifies 3D convolution and spatiotemporal self-attention in a concise transformer format. We adopt local MHRA in shallow layers to largely […]