Issue #82 – Constrained Decoding using Levenshtein Transformer

14 May20

Author: Raj Patel, Machine Translation Scientist @ Iconic

Introduction

In constrained decoding, we force in-domain terminology to appear in the final translation. We have previously discussed constrained decoding in earlier blog posts (#7, #9, #79). In this blog post, we will discuss a simple and effective algorithm for incorporating lexical constraints in Neural Machine Translation (NMT) proposed by Susanto et al. (2020) and try to understand how it is better than the existing techniques.

Levenshtein Transformer (LevT)

Levenshtein Transformer is based on an encoder-decoder framework using Transformer blocks. Unlike the token generation in a typical Transformer model, LevT decoder is based on a Markov Decision Process (MDP) that iteratively refines the generated token with a sequence of insertion and deletion operations. The deletion and insertion operations are performed via three classifiers that run sequentially:

1. 1. 1. 1. Deletion Classifier, which predicts for each token position, whether they should be “kept”or “deleted”,
         2. Placeholder Classifier, which predicts the number of tokens to be inserted
            To finish reading, please visit source site