Adaptive Ensemble Construction based on Progressive Interactive Training of Neural Networks

Abstract

The goal of an ensemble construction with several neural networks (NNs) is to achieve better generalization ability over a single neural network. In a Neural Network Ensemble (NNE), component networks solve the desired problem independently and combine their outputs for NNE's output. Therefore, the size of an NNE plays an important role in determining the performance for a particular problem. Although there has been much work for constructing ensembles, most of the works trains predefined number of NNs for an NNE. This study presents a problem dependent adaptive NNE construction method based on Progressive Interactive Training Scheme (PITS) of NNs. In PITS, a predefined number of NNs are trained one by one in a progressive manner, where each NN is concerned with a specific task that has not been solved by any previously trained NN. Proposed Adaptive PITS (APITS) is an extension of PITS utilizing inherited benefit of its sequential style training. APITS employs a simple overhead in PITS while training NNs one after another sequentially. The overhead determines more NNs training will be beneficial or not for a given problem and therefore, return different number of NNs for different problems. The proposed method has been evaluated on several benchmark problems, and the method (with concise NNE) is shown competitive generalization ability with the popular NNE methods.