Energy Alignment for Bias Rectification in Class Incremental Learning

Abstract

In class incremental learning (CIL), models are expected to be able to learn new categories continuously. However, the standard DNNs suffer from catastrophic forgetting. Recent studies show class imbalance is an essential factor that causes catastrophic forgetting in CIL. In this paper, from the perspective of energy-based model, we demonstrate that the free energies of categories are aligned with the label distribution theoretically, thus the energies of different classes are expected to be close to each other when aiming for "balanced" performance. However, we discover a severe energy-bias phenomenon in the models trained in CIL. To eliminate the bias, we propose a simple and effective method named Energy Alignment by merely adding the calculated shift scalars onto the output logits, which does not require to (i) modify the network architectures, (ii) intervene the standard learning paradigm. Experimental results show that energy alignment can achieve good performance on several CIL benchmarks.