Implementation of OmniNet, Omnidirectional Representations from Transformers in Pytorch

Omninet – Pytorch

Implementation of OmniNet, Omnidirectional Representations from Transformers, in Pytorch. The authors propose that we should be attending to all the tokens of the previous layers, leveraging recent efficient attention advances to achieve this goal.

Install

$ pip install omninet-pytorch

Usage

import torch

from omninet_pytorch import Omninet

omninet = Omninet(

    dim = 512,                     # model dimension

    depth = 6,                     # depth

    dim_head = 64,                 # dimension per head

    heads = 
 
To finish reading, please visit source site