Large language models (LLMs) are routinely pre-trained on billions of tokens, only to start
the process over again once new data becomes available. A much more efficient solution is …

Deep representation learning methods struggle with continual learning, suffering from both
catastrophic forgetting of useful units and loss of plasticity, often due to rigid and unuseful …

The recent increase in data and model scale for language model pre-training has led to
huge training costs. In scenarios where new data become available over time, updating a …

Modern representation learning methods often struggle to adapt quickly under non-
stationarity because they suffer from catastrophic forgetting and decaying plasticity. Such …

Continual learning research has shown that neural networks suffer from catastrophic
forgetting" at the output level", but it is debated whether this is also the case at the level of …

Language models (LMs) are known to suffer from forgetting of previously learned examples
when fine-tuned, breaking stability of deployed LM systems. Despite efforts on mitigating …

Machine learning (ML) models achieve remarkable performance on tasks they are trained
for. However, they often are sensitive to shifts in the data distribution, which may lead to …

Large Language models (LLMs) suffer from forgetting of upstream data when fine-tuned.
Despite efforts on mitigating forgetting, few have investigated whether, and how forgotten …