Concurrent optimization of multiple base learners in neural network ensembles: An adaptive niching differential evolution approach

Abstract

Neural network ensemble (NNE) exhibits improved performance when compared with a single neural network (NN) in most cases. Traditionally, each base network in an NNE is trained individually, which may result in network redundancy and expensive training overhead. This paper proposes a new adaptive niching evolutionary algorithm, which possesses promising performance in finding multiple optima in terms of good accuracy and diversity. By means of this algorithm, all NNs in an NNE can be trained simultaneously. In particular, the proposed algorithm is named adaptive niching differential evolution (ANDE), which is characterized by a heuristic clustering method to enable iteratively cluster subpopulations that track and locate multiple optima, a parameter adaptation strategy to adaptively adjust parameters according to the subpopulation states, and an auxiliary movement scheme to promote the equilibrium between exploration and exploitation. Experimental results validate the efficiency and effectiveness of the proposed ANDE on the benchmark test suite of multimodal optimization. Furthermore, ANDE is extended to concurrently train multiple base NNs for ensemble and the experiments show a promising performance of ANDE-NNE.