Neural networks with sigmoidal activation functions—dimension reduction using normal random projection

Abstract

The dimensionality and the amount of data that needs to be processed when intensive data streams are observed grows rapidly, together with the development of sensors arrays, CCD and CMOS cameras and other devices. The aim of this paper is to analyze an approach to dimensionality reduction as a first stage of the multi-layer feed-forward neural networks with sigmoidal activation functions. The lower bound on dimension of the output of dimensionality reduction layer is established. Tail bounds for inner products–similar to the known Johnson–Lindenstrauss results, concerning squared distance preservation–are given.