A three-way decision approach for dynamically expandable networks

Abstract

Conventional deep learning models are designed to work on a single task. They are required to be trained from scratch each time new tasks are added. This leads to overhead in training time. Continual deep learning models with dynamically expandable network architecture aim to handle this issue. The key idea in these models is to find a balance between the properties of stability (preserving the learned information) and plasticity (updating and accommodating the new information) also sometimes referred to as the stability-plasticity dilemma. The stability and plasticity of the model critically depends on three-way division of nodes into freeze, partially regularize and duplicate nodes. Freezing more nodes result in high stability but typically low plasticity. On the other hand, duplicating more nodes result in high plasticity but may not have an effective stability. In this paper, we introduce an approach called three-way decisions based dynamically expandable networks or 3WDDEN and its memory-based version called 3WDDEN-replay. The proposed approaches use game-theoretic rough sets to determine effective thresholds for three-way division of nodes by considering a tradeoff game between stability and plasticity. Experimental results of 3WDDEN on MNIST variant datasets show an overall improvement of 3.8% in accuracy compared to standard dynamically expandable network approach or DEN. 3WDDEN-replay further adds to accuracy with additional memory cost.