Empirical modeling of very large data sets using neural networks

Abstract

Building empirical predictive models from very large data sets is challenging. One has to deal both with the 'curse of dimensionality' (hundreds or thousands of variables) and with 'too many records' (many thousands of instances). While neural networks [Rumelhart, et al., 1986] are widely recognized as universal function approximators [Cybenko, 1989], their training time rapidly increases with the number of variables and instances. I discuss practical methods for overcoming this problem so that neural network models can be developed for very large databases. The methods include: Dimensionality reduction with neural net modeling, PLS modeling, and bottleneck neural networks; Sub-sampling and re-sampling with many smaller data sets to reduce training time; Committee of networks to make the final prediction more robust and to estimate its uncertainty.