I finally got around to submitting my thesis. The thesis touches on the four areas of transfer learning that are most prominent in current Natural Language Processing (NLP): domain adaptation, multi-task learning, cross-lingual learning, and sequential transfer learning. Most of the work in the thesis has been previously presented (see Publications). Nevertheless, there are some new parts as well. The most notable are: a background chapter (§2) that lays out key concepts in terms of probability and information theory, machine […]

Update 16.10.2020: Added Chinese and Spanish translations. This post expands on the NAACL 2019 tutorial on Transfer Learning in NLP. The tutorial was organized by Matthew Peters, Swabha Swayamdipta, Thomas Wolf, and me. In this post, I highlight key insights and takeaways and provide updates based on recent work. You can see the structure of this post below: The slides, a Colaboratory notebook, and code of the tutorial are available online. For an overview of what transfer learning is, have […]

This post expands on the ACL 2019 tutorial on Unsupervised Cross-lingual Representation Learning. The tutorial was organised by Ivan Vulić, Anders Søgaard, and me. In this post, I highlight key insights and takeaways and provide additional context and updates based on recent work. In particular, I cover unsupervised deep multilingual models such as multilingual BERT. You can see the structure of this post below: The slides of the tutorial are available online. Cross-lingual representation learning can be seen as an […]

This post gathers ten ML and NLP research directions that I found exciting and impactful in 2019. For each highlight, I summarise the main advances that took place this year, briefly state why I think it is important, and provide a short outlook to the future. The full list of highlights is here: Universal unsupervised pretraining Lottery tickets The Neural Tangent Kernel Unsupervised multilingual learning More robust benchmarks ML and NLP for science Fixing decoding errors in NLG Augmenting pretrained […]

This advice should be most relevant to people studying machine learning (ML) and natural language processing (NLP) as that is what I did in my PhD. Having said that, this advice is not just limited to PhD students. If you are an independent researcher, want to start a PhD in the future or simply want to learn, then you will find most of this advice applicable. Pick and choose.  Everyone is different. You will have the most success if you […]

Natural language processing (NLP) research predominantly focuses on developing methods that work well for English despite the many positive benefits of working on other languages. These benefits range from an outsized societal impact to modelling a wealth of linguistic features to avoiding overfitting as well as interesting challenges for machine learning (ML). There are around 7,000 languages spoken around the world. The map above (see the interactive version at Langscape) gives an overview of languages spoken around the world, with […]

The selection of areas and methods is heavily influenced by my own interests; the selected topics are biased towards representation and transfer learning and towards natural language processing (NLP). I tried to cover the papers that I was aware of but likely missed many relevant ones—feel free to highlight them in the comments below. In all, I discuss the following highlights: Scaling up—and down Retrieval augmentation Few-shot learning Contrastive learning Evaluation beyond accuracy Practical concerns of large LMs Multilinguality Image […]

Fine-tuning a pre-trained language model (LM) has become the de facto standard for doing transfer learning in natural language processing. Over the last three years (Ruder, 2018), fine-tuning (Howard & Ruder, 2018) has superseded the use of feature extraction of pre-trained embeddings (Peters et al., 2018) while pre-trained language models are favoured over models trained on translation (McCann et al., 2018), natural language inference (Conneau et al., 2017), and other tasks due to their increased sample efficiency and performance (Zhang […]