Abstract
Artificial neural networks (ANNs) have emerged as hot topics in the research community. Despite the success of ANNs, it is challenging to train and deploy modern ANNs on commodity hardware due to the ever-increasing model size and the unprecedented growth in the data volumes. Particularly for microarray data, the very high dimensionality and the small number of samples make it difficult for machine learning techniques to handle. Furthermore, specialized hardware such as graphics processing unit (GPU) is expensive. Sparse neural networks are the leading approaches to address these challenges. However, off-the-shelf sparsity-inducing techniques either operate from a pretrained model or enforce the sparse structure via binary masks. The training efficiency of sparse neural networks cannot be obtained practically. In this paper, we introduce a technique allowing us to train truly sparse neural networks with fixed parameter count throughout training. Our experimental results demonstrate that our method can be applied directly to handle high-dimensional data, while achieving higher accuracy than the traditional two-phase approaches. Moreover, we have been able to create truly sparse multilayer perceptron models with over one million neurons and to train them on a typical laptop without GPU (https://github.com/dcmocanu/sparse-evolutionary-artificial-neural-networks/tree/master/SET-MLP-Sparse-Python-Data-Structures), this being way beyond what is possible with any state-of-the-art technique.
Highlights
In the past decades, artificial neural networks (ANNs) have become an active area of current research due to state-ofthe-art performance they have achieved in a variety of domains, including image recognition, text classification, and speech recognition
We can observe that sparse evolutionary training (SET)-Multilayer perceptron (MLP) consistently outperform MLPFixProb on all datasets, which means that the adaptive sparse connectivity associated with SET-MLP helps to find better sparse structures
To understand better the connections reduction made by the SET procedure in a SET-MLP model in comparison with a fully connected MLP (FC-MLP) which has the same amount of neurons, Fig. 4 and Table 7 provide the number of connections for the SET-MLP models discussed above and their FC-MLP counterparts on all four datasets
Summary
Artificial neural networks (ANNs) have become an active area of current research due to state-ofthe-art performance they have achieved in a variety of domains, including image recognition, text classification, and speech recognition. GPU is expensive and the explosive increase of model size leads to prohibitive memory requirements. The required resources to train and employ the modern ANNs are at odds with commodity hardware where the resources are very limited. Motivated by these challenges, sparse neural networks [9, 12] have been introduced to effectively reduce the Neural Computing and Applications (2021) 33:2589–2604 memory requirements to deploy ANN models. While achieving a high level of sparsity and preserving competitive performance, these methods usually involve a pretrained model and a retraining process, which makes the training process remain inefficient
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
