Language Modelling as a Multi-Task Problem
Abstract
In this paper, we propose to study language modeling as a multi-task problem, bringing together three strands of research: multitask learning, linguistics, and interpretability. Based on hypotheses derived from linguistic theory, we investigate whether language models adhere to learning principles of multi-task learning during training. We showcase the idea by analysing the generalization behavior of language models during learning of the linguistic concept of Negative Polarity Items (NPIs). Our experiments demonstrate that a multi-task setting naturally emerges within the objective of the