COCO LM Pretraining (wip) Implementation of COCO-LM, Correcting and Contrasting Text Sequences for Language Model Pretraining, in Pytorch. They were able to make contrastive learning work in a self-supervised manner for language model pretraining. Seems like a solid successor to Electra. Install $ pip install coco-lm-pytorch Usage An example using the x-transformers library $ pip install x-transformers Then import torch from coco_lm_pytorch import COCO # (1) instantiate the generator and discriminator, making sure that the generator is roughly a quarter […]

Omninet – Pytorch Implementation of OmniNet, Omnidirectional Representations from Transformers, in Pytorch. The authors propose that we should be attending to all the tokens of the previous layers, leveraging recent efficient attention advances to achieve this goal. Install $ pip install omninet-pytorch Usage import torch from omninet_pytorch import Omninet omninet = Omninet( dim = 512, # model dimension depth = 6, # depth dim_head = 64, # dimension per head heads =