Abstract
Incremental learning means the methodology that continuously uses sequential input data to extend the existing network’s knowledge. The layer sharing algorithm is one of the representative methods which leverages general knowledge by sharing some initial layers of the existing network. In this algorithm, estimating how much initial convolutional layers of the existing network can be shared as the fixed feature extractors for incremental learning should be solved. However, the existing algorithm selects the sharing configurations through not a proper optimization strategy but a brute force manner. Accordingly, it has to search for all possible sharing layer cases, leading to high computational complexity. To solve this problem, we firstly define this problem as a discrete combinatorial optimization problem. However, this problem is a non-convex and non-differential optimization problem which can not be solved using the gradient descent algorithm or other convex optimization methods, even though these are the powerful optimization techniques. Thus, we propose a novel efficient incremental learning algorithm based on Bayesian optimization, which guarantees the global convergence in a non-convex and non-differential optimization problem. And the proposed algorithm can adaptively find the optimal number of sharing layers via adjusting the threshold accuracy parameter in the proposed loss function. The proposed method produces the global optimal sharing layer number in only 6 iterations without searching for all possible layer cases in experimental results. Hence, the proposed method can find the global optimal sharing layer and achieve both high combined accuracy and low computational complexity.
Highlights
Computer vision technologies, including image recognition and object detection, have developed rapidly in the field of deep learning
This form can not be applied to BayesOpt, because the Bayesian optimization guarantees the global convergence in a continuous but non-derivative combinatorial optimization problem
Because the proposed loss function is shaped in continuous function, the BayesOpt in the proposed algorithm can converge to the global optimal sharing layer number, meeting incremental learning conditions
Summary
Computer vision technologies, including image recognition and object detection, have developed rapidly in the field of deep learning Despite these remarkable achievements, one of the significant challenges in neural network-based computer vision algorithms is learning new tasks incrementally, like the cognitive process of human learning [1,2]. [4] requires high computational complexity and time consumption to train all possible cases To solve this limitation, the proposed method utilizes a Bayesian optimization (BayesOpt) to get the optimal number of sharing layer without considering all possible cases. The proposed algorithm can find a global optimal sharing layer for layer sharing-based incremental learning. By utilizing BayesOpt, the proposed method effectively computes the number of global optimal sharing layer without computing all possible cases. To employ BayesOpt, the proposed objective function, which is a discrete function due to the number of layers, is designed to represent the combinatorial optimization problem with a step function as a continuous function
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have