Abstract
Neural network forms the foundation of deep learning and numerous AI applications. Classical neural networks are fully connected, expensive to train and prone to overfitting. Sparse networks tend to have convoluted structure search, suboptimal performance and limited usage. We proposed the novel uniform sparse network (USN) with even and sparse connectivity within each layer. USN has one striking property that its performance is independent of the substantial topology variation and enormous model space, thus offers a search-free solution to all above mentioned issues of neural networks. USN consistently and substantially outperforms the state-of-the-art sparse network models in prediction accuracy, speed and robustness. It even achieves higher prediction accuracy than the fully connected network with only 0.55% parameters and 1/4 computing time and resources. Importantly, USN is conceptually simple as a natural generalization of fully connected network with multiple improvements in accuracy, robustness and scalability. USN can replace the latter in a range of applications, data types and deep learning architectures. We have made USN open source at https://github.com/datapplab/sparsenet.
Highlights
Neural network (NN) or artificial neural network (ANN) is one of the most popular machine learning (ML) frameworks, and form the foundation of most artificial intelligence (AI) and deep learning (DL) methods emerged in the past decades[1,2,3,4,5,6]
SPARSE ADVANTAGES HOLD IN VARIOUS DEEP NEURAL NETWORK ARCHITECTURES We have focused on multi-layer perceptrons (MLPs) or the basic deep NN architecture so far to show the sparse advantages (Figure 4-7)
uniform sparse network (USN) exhibited some striking properties that its performance is closely related to connection density, but independent of its substantial topology variation and enormous model space
Summary
Neural network (NN) or artificial neural network (ANN) is one of the most popular machine learning (ML) frameworks, and form the foundation of most artificial intelligence (AI) and deep learning (DL) methods emerged in the past decades[1,2,3,4,5,6]. Density and hidden layer size are the primary determining factors of USN model performance (loss and accuracy) and its variance (Figure 2 row 2-3 and Table 2). USN is more accurate and faster, and more robust or reproducible than SET and DSR (Figure 3 upper panel) The latter two methods had big fluctuations (shown by IQR or range between 10 and 90 quantiles, the shaded regions in Figure 3) in accuracy throughout the training process especially at high sparsity levels or in BSEQ data, but USN had smooth accuracy curves in all benchmark experiments. USN had little or the smallest performance variations between different sparsity levels and hidden layer sizes for both datasets (Figure 3 upper panel)
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
