The post discusses highlights of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018). This post originally appeared at the AYLIEN blog. You can find past highlights of conferences here. You can find all 549 accepted papers in the EMNLP proceedings. In this review, I will focus on papers that relate to the following topics: Inductive bias The inductive bias of a machine learning algorithm is the set of assumptions that the model makes in order to […]

This post gathers 10 ideas that I found exciting and impactful this year—and that we’ll likely see more of in the future. For each idea, I will highlight 1-2 papers that execute them well. I tried to keep the list succinct, so apologies if I did not cover all relevant work. The list is necessarily subjective and covers ideas mainly related to transfer learning and generalization. Most of these (with some exceptions) are not trends (but I suspect that some […]

This post discusses 4 major open problems in NLP based on an expert survey and a panel discussion at the Deep Learning Indaba. This is the second blog post in a two-part series. The series expands on the Frontiers of Natural Language Processing session organized by Herman Kamper, Stephan Gouws, and me at the Deep Learning Indaba 2018. Slides of the entire session can be found here. The first post discussed major recent advances in NLP focusing on neural network-based […]

I finally got around to submitting my thesis. The thesis touches on the four areas of transfer learning that are most prominent in current Natural Language Processing (NLP): domain adaptation, multi-task learning, cross-lingual learning, and sequential transfer learning. Most of the work in the thesis has been previously presented (see Publications). Nevertheless, there are some new parts as well. The most notable are: a background chapter (§2) that lays out key concepts in terms of probability and information theory, machine […]

Update 16.10.2020: Added Chinese and Spanish translations. This post expands on the NAACL 2019 tutorial on Transfer Learning in NLP. The tutorial was organized by Matthew Peters, Swabha Swayamdipta, Thomas Wolf, and me. In this post, I highlight key insights and takeaways and provide updates based on recent work. You can see the structure of this post below: The slides, a Colaboratory notebook, and code of the tutorial are available online. For an overview of what transfer learning is, have […]

This post expands on the ACL 2019 tutorial on Unsupervised Cross-lingual Representation Learning. The tutorial was organised by Ivan Vulić, Anders Søgaard, and me. In this post, I highlight key insights and takeaways and provide additional context and updates based on recent work. In particular, I cover unsupervised deep multilingual models such as multilingual BERT. You can see the structure of this post below: The slides of the tutorial are available online. Cross-lingual representation learning can be seen as an […]

This post gathers ten ML and NLP research directions that I found exciting and impactful in 2019. For each highlight, I summarise the main advances that took place this year, briefly state why I think it is important, and provide a short outlook to the future. The full list of highlights is here: Universal unsupervised pretraining Lottery tickets The Neural Tangent Kernel Unsupervised multilingual learning More robust benchmarks ML and NLP for science Fixing decoding errors in NLG Augmenting pretrained […]

This advice should be most relevant to people studying machine learning (ML) and natural language processing (NLP) as that is what I did in my PhD. Having said that, this advice is not just limited to PhD students. If you are an independent researcher, want to start a PhD in the future or simply want to learn, then you will find most of this advice applicable. Pick and choose.  Everyone is different. You will have the most success if you […]

Natural language processing (NLP) research predominantly focuses on developing methods that work well for English despite the many positive benefits of working on other languages. These benefits range from an outsized societal impact to modelling a wealth of linguistic features to avoiding overfitting as well as interesting challenges for machine learning (ML). There are around 7,000 languages spoken around the world. The map above (see the interactive version at Langscape) gives an overview of languages spoken around the world, with […]

The selection of areas and methods is heavily influenced by my own interests; the selected topics are biased towards representation and transfer learning and towards natural language processing (NLP). I tried to cover the papers that I was aware of but likely missed many relevant ones—feel free to highlight them in the comments below. In all, I discuss the following highlights: Scaling up—and down Retrieval augmentation Few-shot learning Contrastive learning Evaluation beyond accuracy Practical concerns of large LMs Multilinguality Image […]