Issue #104 – Using Test Sets to Evaluate Machine Translation
22 Oct20
Issue #104 – Using Test Sets to Evaluate Machine Translation
Author: Dr. Karin Sim, Machine Translation Scientist @ Iconic
Introduction
There is finally a growing acceptance in some circles that evaluation of Machine Translation (MT) is lagging behind progress in Neural MT (NMT). Especially with regards to metrics such as BLEU, there is a recognition that “as NMT continues to improve, these metrics will inevitably lose their effectiveness” (Isabelle et al., 2017). In today’s blog post, we look at another method of evaluation, that of test sets. These are common in software engineering, and have been used previously in MT some time ago (King and Falkedal (1990)). More recently, Isabelle et al. (2017) used this same approach in