Abstract
One major challenge for modern artificial neural networks (ANNs) is that they typically does not handle incremental learning well. In other words, while learning the new features, the performances of existing features usually deteriorate. This phenomenon is called catastrophic forgetting, which causes great problems for continuous, incremental, and intelligent learning. In this work, we propose a dynamic correction vector based algorithm to address both the bias problem from knowledge distillation and the overfitting problem. Specifically, we have made the following contributions: 1) we have designed a novel dynamic correction vector based algorithm; 2) we have proposed new loss functions accordingly. Experimental results on MNIST and CIFAR-100 datasets demonstrate that our technique can outperform state-of-the-art incremental learning methods by 4% on large datasets.
Highlights
Human learning process is intrinsically incremental, whereas new information is assimilated to improve the learning performance gradually
Cauwenberghs and Poggio [15] proposed an alternative method that preserves the Karush-Kuhn-Tucker condition for all previously trained old data and updates the weight based on the new data. These early strategies show some success, they are confined by specific classifiers, e.g., support vector machines
2) DATASETS In this work, we evaluate our techniques on two datasets, which are MNIST and CIFAR-100 [33], respectively
Summary
Human learning process is intrinsically incremental, whereas new information is assimilated to improve the learning performance gradually. To mimic the performance of the human neural networks, the concept of incremental learning techniques is introduced [3]. To address these problems, researchers have proposed various solutions [4]–[7]. The existing incremental learning techniques can be categorized into four parts, which are 1) representative memory based [4], 2) context neuron gating based [9], 3) limiting weight updating direction and sizes [10], and 4) training a specific network for each task [11]. 1) We have designed a dynamic correction vector algorithm, which combines the representative memory and knowledge distillation loss techniques to solve the catastrophic forgetting problem. V concludes the paper and discuss the possible future research directions
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
