Not All Memories are Created Equal: Learning to Forget by Expiring
Abstract Attention mechanisms have shown promising results in sequence modeling tasks that require long term memory. Recent work investigated mechanisms to reduce the computational cost of preserving and storing memories (Rae et al., 2020). However, not all content in the past is equally important to remember. We propose Expire-Span, a method that learns to retain the most important information and expire the irrelevant information. This forgetting of memories enables Transformers to scale to attend over tens of thousands of previous […]
Read more