Nader Asadi

Featured Publications

1. 

   Prototype-Sample Relation Distillation: Towards Replay-Free Continual Learning

   Nader Asadi, MohammadReza Davari, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky

   Preprint 2023
2. 

   Probing Representation Forgetting in Supervised and Unsupervised Continual Learning

   MohammadReza Davari*,  Nader Asadi*, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky

   CVPR 2022

   PDF Abs arXiv Bib

   Continual Learning (CL) research typically focuses on tackling the phenomenon of catastrophic forgetting in neural networks. Catastrophic forgetting is associated with an abrupt loss of knowledge previously learned by a model when the task, or more broadly the data distribution, being trained on changes. In supervised learning problems this forgetting, resulting from a change in the model’s representation, is typically measured or observed by evaluating the decrease in old task performance. However, a model’s representation can change without losing knowledge about prior tasks. In this work we consider the concept of representation forgetting, observed by using the difference in performance of an optimal linear classifier before and after a new task is introduced. Using this tool we revisit a number of standard continual learning benchmarks and observe that, through this lens, model representations trained without any explicit control for forgetting often experience small representation forgetting and can sometimes be comparable to methods which explicitly control for forgetting, especially in longer task sequences. We also show that representation forgetting can lead to new insights on the effect of model capacity and loss function used in continual learning. Based on our results, we show that a simple yet competitive approach is to learn representations continually with standard supervised contrastive learning while constructing prototypes of class samples when queried on old samples.

   @inproceedings{davari2022probing,
     title = {Probing Representation Forgetting in Supervised and Unsupervised Continual Learning},
     author = {Davari, MohammadReza and Asadi, Nader and Mudur, Sudhir and Aljundi, Rahaf and Belilovsky, Eugene},
     booktitle = {CVPR},
     year = {2022},
     equal_first_authors = {Davari, MohammadReza and Asadi, Nader}
   }

3. 

   New Insights on Reducing Abrupt Representation Change in Online Continual Learning

   Lucas Caccia, Rahaf Aljundi,  Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky

   ICLR 2022

   PDF Code Abs arXiv Bib

   In the online continual learning paradigm, agents must learn from a changing distribution while respecting memory and compute constraints. Experience Replay (ER), where a small subset of past data is stored and replayed alongside new data, has emerged as a simple and effective learning strategy. In this work, we focus on the change in representations of observed data that arises when previously unobserved classes appear in the incoming data stream, and new classes must be distinguished from previous ones. We shed new light on this question by showing that applying ER causes the newly added classes’ representations to overlap significantly with the previous classes, leading to highly disruptive parameter updates. Based on this empirical analysis, we propose a new method which mitigates this issue by shielding the learned representations from drastic adaptation to accommodate new classes. We show that using an asymmetric update rule pushes new classes to adapt to the older ones (rather than the reverse), which is more effective especially at task boundaries, where much of the forgetting typically occurs. Empirical results show significant gains over strong baselines on standard continual learning benchmarks.

   @inproceedings{caccia2021reducing,
     title = {New Insights on Reducing Abrupt Representation Change in Online Continual Learning},
     author = {Caccia, Lucas and Aljundi, Rahaf and Asadi, Nader and Tuytelaars, Tinne and Pineau, Joelle and Belilovsky, Eugene},
     booktitle = {ICLR},
     year = {2022},
   }

4. 

   Tackling Online One-Class Incremental Learning by Removing Negative Contrasts

   Nader Asadi, Sudhir Mudur, and Eugene Belilovsky

   NeurIPS Workshop on Distribution Shifts 2021

   PDF Abs arXiv Bib

   Recent work studies the supervised online continual learning setting where a learner receives a stream of data whose class distribution changes over time. Distinct from other continual learning settings the learner is presented new samples only once and must distinguish between all seen classes. A number of successful methods in this setting focus on storing and replaying a subset of samples alongside incoming data in a computationally efficient manner. One recent proposal ER-AML achieved strong performance in this setting by applying an asymmetric loss based on contrastive learning to the incoming data and replayed data. However, a key ingredient of the proposed method is avoiding contrasts between incoming data and stored data, which makes it impractical for the setting where only one new class is introduced in each phase of the stream. In this work we adapt a recently proposed approach (\textitBYOL) from self-supervised learning to the supervised learning setting, unlocking the constraint on contrasts. We then show that supplementing this with additional regularization on class prototypes yields a new method that achieves strong performance in the one-class incremental learning setting and is competitive with the top performing methods in the multi-class incremental setting.

   @inproceedings{asadi2021tackling,
     title = {Tackling Online One-Class Incremental Learning by Removing Negative Contrasts},
     author = {Asadi, Nader and Mudur, Sudhir and Belilovsky, Eugene},
     booktitle = {NeurIPS Workshop on Distribution Shifts},
     year = {2021},
   }