Not All Memories are Created Equal: Learning to Forget by Expiring
Abstract
Attention mechanisms have shown promising results in sequence modeling tasks that require long term memory. Recent work investigated mechanisms to reduce the computational cost of preserving and storing memories (Rae et al., 2020). However, not all content in the past is equally important to remember. We propose Expire-Span, a method that learns to retain the most important information and expire the irrelevant information. This forgetting of memories enables Transformers to scale to attend over tens of thousands of previous timesteps efficiently, as not all states from previous timesteps are preserved. We demonstrate that Expire-Span can help models identify