Issue #63 – Neuron Interaction Based Representation Composition for Neural Machine Translation

05 Dec19

Author: Dr. Rohit Gupta, Sr. Machine Translation Scientist @ Iconic

Transformer models are state of the art in Neural Machine Translation. In this blog post, we will take a look at a recently proposed approach by Li et al (2019) which further improves upon the transformer model by modeling more neuron interactions. Li et al (2019) claim that their approach models better encoder representation and captures semantic and syntactic properties better than the baseline transformer. 

Bilinear Pooling:

Bilinear pooling (Tenenbaum and Freeman (2000)) is defined as an outer product of two representation vectors followed by a linear projection. For the transformer model we have multi-layer and multi-head representation vectors. These vectors are concatenated to form the input to bilinear pooling. 

Bilinear pooling only encodes second-order (i.e., multiplicative) interactions among individual neurons, therefore, they proposed the extended bilinear pooling which also includes first-order representations. The figure below illustrates bilinear and extended bilinear pooling. 

To finish reading, please visit source site