Machine Translation Weekly 91: Zero-Shot Machine Translation with a Universal Encoder from Pre-trained Representations
How many times have you heard someone saying that multilingual BERT or similar
models could be used as a universal encoder in machine translation? I heard
that (and said that) many times, but never heard about someone who actually did
that, until now. Folks from The University of Hong Kong, Mircosoft Research,
Shanghai University, and Texas A&M University published their preprint on this
topic last Thursday on arXiv. The title of the paper is Towards Making the
Most of Multilingual Pretraining for Zero-Shot Neural Machine
Translation. They actually published a large
deal in this direction already in April, but at that time I did not notice, and
this time their method works much better.
The idea is simple. Multilingual BERT and XLM-RoBERTa provide sentence
representation for more than 100