State-of-the-art Multivariate Regression with a General Nk Hidden Multi-Layer Feed Forward Neural Network Model

Abstract

We have introduced a novel multivariate regression model, called General N <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</inf> Hidden Multi-Layer Feed Forward Neural Network Model. The model prediction power comes essentially from a combination of Universal Approximation theorem, the Stochastic Gradient Descent convergence, and finally the use of ADAM optimizer which is also (at least) locally convergent. This N <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</inf> -architecture is proposed to fit any multivariate regression task, as well as any classification task when a softmax activation gate function is applied to the output layer nodes. The model can easily be augmented to thousands of possible layers without loss of predictive power, and has the potential to overcome our difficulties simultaneously in building a model that has a good fit on the test data, and don’t overfit. Its hyper-parameters, the learning rate, the batch size, the number of training times (epochs), the size of each layer, the number of hidden layers, all can be chosen experimentally with cross-validation methods. We have run some experiments with the Mulan Project Datasets [29] to illustrate the performance of the model against Random Forest with a number of estimators from 5 to 10, and a maximum depth from 10 to 30. Not only has the model surpasses the Random Forest model in all tested configurations, but, we have also found this General N <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</inf> Hidden Multi-layer Feed Forward Neural Network to be so effective as it reaches state-of-the-art performance for multivariate regression in terms of Mean Squared Error.