A constructive approach to data-driven randomized learning for feedforward neural networks

Abstract

There is an issue with the way in which feedforward neural networks with random hidden nodes generate random parameters in order to obtain a good projection space. Typically, random weights and biases are both drawn from the same interval, which is misguided as they have different functions. Recently, more sophisticated methods of random parameters generation have been developed, such as the data-driven approach, where the sigmoids are placed in randomly selected regions of the input space and then their slopes are adjusted to the local fluctuations of the target function. In this work, we propose a new constructive data-driven method that builds iteratively the network architecture. This method successively generates new candidate hidden nodes and accepts them when the training error falls significantly. The threshold of acceptance is adapted throughout training, accepting at the beginning of the training process only those nodes which lead to the largest reductions in error. In the next stages, the threshold is successively reduced to accept only those nodes which model the target function details more accurately. This leads to a more compact network architecture, as it includes only ”significant” nodes. It is worth noting that redundant, random nodes, which are usually generated by existing randomized learning methods, are not accepted by the proposed method. We empirically compared our approach with several alternative methods, including its predecessor, competitive randomized learning solutions, a gradient-based network and a generalized additive model. We found that our proposed approach outperformed its competitors in terms of fitting accuracy.