Highlights from Machine Translation and Multilinguality in February 2025
WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects
Folks from Google and Unbable extended the WMT24 test sets from 8 to 55 languages by adding more human references. They evaluated LLMs and commercial MT services on them. The winner is OpenAI’s o1, followed by Claude and Gemini. The best open-source model is Unbabel’s Tower, which outperforms all standard commercial translation services (Google Translate, DeepL, and Microsoft Translator).
SMOL: Professionally translated parallel data for 115 under-represented languages
Speaking of machine translation datasets, Google opened-sourced a parallel dataset for 115 languages with some non-English-centric pairs. The data was not crawled from the Internet but manually translated. They show that the data are useful by fine-tuning Gemini and improving translation quality. Well, nice,