Efficient Neural Network Model Research Articles

Efficient neural network-aided seismic life-cycle cost optimization of steel moment frames

In this paper, a novel and efficient neural network-based methodology is proposed to achieve seismic total cost optimization of steel moment-resisting frames in a timely manner. The computational burden of an optimization process based on performing nonlinear time-history analysis is prohibitively high. To address this crucial issue, a new and efficient neural network model is proposed in this paper to accurately predict the nonlinear time history response of steel frames during the optimization process. In the proposed neural network model, an ensemble of parallel neural networks is used to provide excellent prediction accuracy. In addition, a new repairability constraint is proposed to check the seismic damage level of structures during the optimization process with the aid of the proposed neural network model. Moreover, an efficient metaheuristic algorithm is used to achieve the optimization task. Two numerical examples are illustrated to demonstrate the efficiency of the proposed methodology. The results show that the proposed neural network model outperforms the existing standard models in terms of prediction accuracy. Furthermore, it is shown that by using the proposed methodology, the optimal seismic total cost of steel frames increases by less than 2.5%, yet their seismic collapse capacity increases by at least 30%.

Real-time segmentation of short videos under VR technology in dynamic scenes

Abstract This work addresses the challenges of scene segmentation and low segmentation accuracy in short videos by employing virtual reality (VR) technology alongside a 3D DenseNet model for real-time segmentation in dynamic scenes. First, this work extracted short videos by frame and removed redundant background information. Then, the volume rendering algorithm in VR technology was used to reconstruct short videos in dynamic scenes in 3D. It enriched the detailed information of short videos, and finally used the 3D DenseNet model for real-time segmentation of short videos in dynamic scenes, improving the accuracy of segmentation. The experiment compared the performance of High resolution network, Mask region based convolutional neural network, 3D U-Net, Efficient neural network models on the Densely annotation video segmentation dataset. The experimental results showed that the segmentation accuracy of the 3D DenseNet model has reached 99.03%, which was 15.11% higher than that of the ENet model. The precision rate reached 98.33%, and the average segmentation time reached 0.64 s, improving the segmentation accuracy and precision rate. It can adapt to various scene situations and has strong robustness. The significance of this research lies in its innovative approach in tackling these issues. By integrating VR technology with advanced deep learning models, we can achieve more precise segmentation of dynamic scenes in short videos, enabling real-time processing. This has significant practical implications for fields such as video editing, VR applications, and intelligent surveillance. Furthermore, the outcomes of this research contribute to advancing computer vision in video processing, providing valuable insights for the development of future intelligent video processing systems.

Improved weight initialization for deep and narrow feedforward neural network

Appropriate weight initialization settings, along with the ReLU activation function, have become cornerstones of modern deep learning, enabling the training and deployment of highly effective and efficient neural network models across diverse areas of artificial intelligence. The problem of “dying ReLU,” where ReLU neurons become inactive and yield zero output, presents a significant challenge in the training of deep neural networks with ReLU activation function. Theoretical research and various methods have been introduced to address the problem. However, even with these methods and research, training remains challenging for extremely deep and narrow feedforward networks with ReLU activation function. In this paper, we propose a novel weight initialization method to address this issue. We establish several properties of our initial weight matrix and demonstrate how these properties enable the effective propagation of signal vectors. Through a series of experiments and comparisons with existing methods, we demonstrate the effectiveness of the novel initialization method.

Evolutionary Perspectives on Neural Network Generations: A Critical Examination of Models and Design Strategies

Abstract: In the last few years, Neural Networks have become more common in different areas due to their ability to learn intricate patterns and provide precise predictions. Nonetheless, creating an efficient neural network model is a difficult task that demands careful thought of multiple factors, such as architecture, optimization method, and regularization technique. This paper aims to comprehensively overview the state-of-the-art artificial neural network (ANN) generation and highlight key challenges and opportunities in machine learning applications. It provides a critical analysis of current neural network model design methodologies, focusing on the strengths and weaknesses of different approaches. Also, it explores the use of different deep neural networks (DNN) in image recognition, natural language processing, and time series analysis. In addition, the text explores the advantages of selecting optimal values for various components of an Artificial Neural Network (ANN). These components include the number of input/output layers, the number of hidden layers, the type of activation function used, the number of epochs, and the model type selection. Setting these components to their ideal values can help enhance the model's overall performance and generalization. Furthermore, it identifies some common pitfalls and limitations of existing design methodologies, such as overfitting, lack of interpretability, and computational complexity. Finally, it proposes some directions for future research, such as developing more efficient and interpretable neural network architectures, improving the scalability of training algorithms, and exploring the potential of new paradigms, such as Spiking Neural Networks, quantum neural networks, and neuromorphic computing.

Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models

Current AI training infrastructure is dominated by single instruction multiple data (SIMD) and systolic array architectures, such as Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs), that excel at accelerating parallel workloads and dense vector matrix multiplications. Potentially more efficient neural network models utilizing sparsity and recurrence cannot leverage the full power of SIMD processor and are thus at a severe disadvantage compared to today's prominent parallel architectures like Transformers and CNNs, thereby hindering the path towards more sustainable AI. To overcome this limitation, we explore sparse and recurrent model training on a massively parallel multiple instruction multiple data (MIMD) architecture with distributed local memory. We implement a training routine based on backpropagation though time (BPTT) for the brain-inspired class of Spiking Neural Networks (SNNs) that feature binary sparse activations. We observe a massive advantage in using sparse activation tensors with a MIMD processor, the Intelligence Processing Unit (IPU) compared to GPUs. On training workloads, our results demonstrate 5-10x throughput gains compared to A100 GPUs and up to 38x gains for higher levels of activation sparsity, without a significant slowdown in training convergence or reduction in final model performance. Furthermore, our results show highly promising trends for both single and multi IPU configurations as we scale up to larger model sizes. Our work paves the way towards more efficient, non-standard models via AI training hardware beyond GPUs, and competitive large scale SNN models.

Design of an efficient neural network model for detection and classification of phase loss faults for three-phase induction motor

AbstractIn industrial applications, three-phase induction motors (IMs) are widely used, and they are subjected to many types of faults. One such fault is the loss of a motor phase caused by a blown fuse, a broken wire, mechanical damage, etc. When this fault occurs during motor operation, it continues to rotate but experiences rapid heating, which can ultimately lead to motor failure. Therefore, various protection devices are available to protect the motor against this fault, but most traditional protection devices do not offer a comprehensive classification of such a fault. So, in this paper, an efficient neural network model is presented for detecting and classifying 12 types of phase loss faults for a three-phase induction motor (IM) based on factors such as the unhealthy phase, fault location, and motor action modes (standstill, transient, and steady-state modes). Thus, the main goal of this work is to determine the motor mode during the fault, the defective phase, and its location to help the maintenance team repair the fault quickly. The system is simulated and tested using the “MATLAB/Simulink” software, employing a feed-forward neural network. The simulation results demonstrate that the proposed network achieves correct detection and classification of phase loss faults within a short time frame from the occurrence of the fault. Therefore, the proposed network model proves to be a simple and reliable solution for integration into the protection system of a three-phase IM, enabling the detection and classification of various phase loss faults.

Privacy-Preserving Decentralized Inference With Graph Neural Networks in Wireless Networks

As an efficient neural network model for graph data, graph neural networks (GNNs) recently find successful applications for various wireless optimization problems. Given that the inference stage of GNNs can be naturally implemented in a decentralized manner, GNN is a potential enabler for decentralized control/management in the next-generation wireless communications. Privacy leakage, however, may occur due to the information exchanges among neighbors during decentralized inference with GNNs. To deal with this issue, in this paper, we analyze and enhance the privacy of decentralized inference with GNNs in wireless networks. Specifically, we adopt local differential privacy as the metric, and design novel privacy-preserving signals as well as privacy-guaranteed training algorithms to achieve privacy-preserving inference. We also define the SNR-privacy trade-off function to analyze the performance upper bound of decentralized inference with GNNs in wireless networks. To further enhance the communication and computation efficiency, we adopt the over-the-air computation technique and theoretically demonstrate its advantage in privacy preservation. Through extensive simulations on the synthetic graph data, we validate our theoretical analysis, verify the effectiveness of proposed privacy-preserving wireless signaling and privacy-guaranteed training algorithm, and offer some guidance on practical implementation.

Kernel-wise difference minimization for convolutional neural network compression in metaverse.

Convolutional neural networks have achieved remarkable success in computer vision research. However, to further improve their performance, network models have become increasingly complex and require more memory and computational resources. As a result, model compression has become an essential area of research in recent years. In this study, we focus on the best-case scenario for Huffman coding, which involves data with lower entropy. Building on this concept, we formulate a compression with a filter-wise difference minimization problem and propose a novel algorithm to solve it. Our approach involves filter-level pruning, followed by minimizing the difference between filters. Additionally, we perform filter permutation to further enhance compression. Our proposed algorithm achieves a compression rate of 94× on Lenet-5 and 50× on VGG16. The results demonstrate the effectiveness of our method in significantly reducing the size of deep neural networks while maintaining a high level of accuracy. We believe that our approach holds great promise in advancing the field of model compression and can benefit various applications that require efficient neural network models. Overall, this study provides important insights and contributions toward addressing the challenges of model compression in deep neural networks.

An efficient neural network model to determine maximum swelling pressure of clayey soils

Expansive soils exhibit excessive volume increases upon contact with water, which can pose a serious threat to stability of structures and foundations. Therefore, it is essential to determine the swelling properties, e.g. maximum swelling pressure, of these problematic soils. We employed a feed-forward neural network algorithm trained with Levenberg–Marquardt, Bayesian regularization, scaled conjugate gradient, and genetic algorithm to build a network model capable of determining the maximum swelling pressure of clayey soils over a wide range of conditions. The models were developed based on a sufficiently large experimental dataset that takes into account key factors that influence the soil swelling. The results show that the feed-forward neural network algorithm trained with Bayesian regularization has the highest overall accuracy, as its predictions agree well with the experimental data. Besides, a simplified network model was developed to be used in cases of limited data availability. The developed model provides accurate predictions over a wide range of conditions and can serve as a valuable tool for researchers and engineers dealing with expansive soils.

Neural architecture search for resource constrained hardware devices: A survey

AbstractWith the emergence of powerful and low‐energy Internet of Things devices, deep learning computing is increasingly applied to resource‐constrained edge devices. However, the mismatch between hardware devices with low computing capacity and the increasing complexity of Deep Neural Network models, as well as the growing real‐time requirements, bring challenges to the design and deployment of deep learning models. For example, autonomous driving technologies rely on real‐time object detection of the environment, which cannot tolerate the extra latency of sending data to the cloud, processing and then sending the results back to edge devices. Many studies aim to find innovative ways to reduce the size of deep learning models, the number of Floating‐point Operations per Second, and the time overhead of inference. Neural Architecture Search (NAS) makes it possible to automatically generate efficient neural network models. The authors summarise the existing NAS methods on resource‐constrained devices and categorise them according to single‐objective or multi‐objective optimisation. We review the search space, the search algorithm and the constraints of NAS on hardware devices. We also explore the challenges and open problems of hardware NAS.

Decentralized Inference With Graph Neural Networks in Wireless Communication Systems

Graph neural network (GNN) is an efficient neural network model for graph data and is widely used in different fields, including wireless communications. Different from other neural network models, GNN can be implemented in a decentralized manner with information exchanges among neighbors, making it a potentially powerful tool for decentralized control in wireless communication systems. The main bottleneck, however, is wireless channel impairments that deteriorate the prediction robustness of GNN. To overcome this obstacle, we analyze and enhance the robustness of the decentralized GNN in different wireless communication systems in this paper. Specifically, using a GNN binary classifier as an example, we first develop a methodology to verify whether the predictions are robust. Then, we analyze the performance of the decentralized GNN binary classifier in both uncoded and coded wireless communication systems. To remedy imperfect wireless transmission and enhance the prediction robustness, we further propose novel retransmission mechanisms for the above two communication systems, respectively. Through simulations on the synthetic graph data, we validate our analysis, verify the effectiveness of the proposed retransmission mechanisms, and provide some insights for practical implementation.

Few-shot learning using explainable Siamese twin network for the automated classification of blood cells.

Automated classification of blood cells from microscopic images is an interesting research area owing to advancements of efficient neural network models. The existing deep learning methods rely on large data for network training and generating such large data could be time-consuming. Further, explainability is required via class activation mapping for better understanding of the model predictions. Therefore, we developed a Siamese twin network (STN) model based on contrastive learning that trains on relatively few images for the classification of healthy peripheral blood cells using EfficientNet-B3 as the base model. Hence, in this study, a total of 17,092 publicly accessible cell histology images were analyzed from which 6% were used for STN training, 6% for few-shot validation, and the rest 88% for few-shot testing. The proposed architecture demonstrates percent accuracies of 97.00, 98.78, 94.59, 95.70, 98.86, 97.09, 99.71, and 96.30 during 8-way 5-shot testing for the classification of basophils, eosinophils, immature granulocytes, erythroblasts, lymphocytes, monocytes, platelets, and neutrophils, respectively. Further, we propose a novel class activation mapping scheme that highlights the important regions in the test image for the STN model interpretability. Overall, the proposed framework could be used for a fully automated self-exploratory classification of healthy peripheral blood cells. The whole proposed framework demonstrates the Siamese twin network training and 8-way k-shot testing. The values indicate the amount of dissimilarity.

Numerical‐discrete‐scheme‐incorporated recurrent neural network for tasks in natural language processing

AbstractA variety of neural networks have been presented to deal with issues in deep learning in the last decades. Despite the prominent success achieved by the neural network, it still lacks theoretical guidance to design an efficient neural network model, and verifying the performance of a model needs excessive resources. Previous research studies have demonstrated that many existing models can be regarded as different numerical discretizations of differential equations. This connection sheds light on designing an effective recurrent neural network (RNN) by resorting to numerical analysis. Simple RNN is regarded as a discretisation of the forward Euler scheme. Considering the limited solution accuracy of the forward Euler methods, a Taylor‐type discrete scheme is presented with lower truncation error and a Taylor‐type RNN (T‐RNN) is designed with its guidance. Extensive experiments are conducted to evaluate its performance on statistical language models and emotion analysis tasks. The noticeable gains obtained by T‐RNN present its superiority and the feasibility of designing the neural network model using numerical methods.

Storm Surge Forecast Using an Encoder–Decoder Recurrent Neural Network Model

This study presents an encoder–decoder neural network model to forecast storm surges on the US North Atlantic Coast. The proposed multivariate time-series forecast model consists of two long short-term memory (LSTM) models. The first LSTM model encodes the input sequence, including storm position, central pressure, and the radius of the maximum winds to an internal state. The second LSTM model decodes the internal state to forecast the storm surge water level and velocity. The neural network model was developed based on a storm surge dataset generated by the North Atlantic Comprehensive Coastal Study using a physics-based storm surge model. The neural network model was trained to predict storm surges at three forecast lead times ranging from 3 h to 12 h by learning the correlation between the past storm conditions and future storm hazards. The results show that the computationally efficient neural network model can forecast a storm in a fraction of one second. The neural network model not only forecasts peak surges, but also predicts the time-series profile of a storm. Furthermore, the model is highly versatile, and it can forecast storm surges generated by different sizes and strengths of bypassing and landfalling storms. Overall, this work demonstrates the success of data-driven approaches to improve coastal hazard research.

Deep learning tomographic reconstruction through hierarchical decomposition of domain transforms.

Deep learning (DL) has shown unprecedented performance for many image analysis and image enhancement tasks. Yet, solving large-scale inverse problems like tomographic reconstruction remains challenging for DL. These problems involve non-local and space-variant integral transforms between the input and output domains, for which no efficient neural network models are readily available. A prior attempt to solve tomographic reconstruction problems with supervised learning relied on a brute-force fully connected network and only allowed reconstruction with a 1284 system matrix size. This cannot practically scale to realistic data sizes such as 5124 and 5126 for three-dimensional datasets. Here we present a novel framework to solve such problems with DL by casting the original problem as a continuum of intermediate representations between the input and output domains. The original problem is broken down into a sequence of simpler transformations that can be well mapped onto an efficient hierarchical network architecture, with exponentially fewer parameters than a fully connected network would need. We applied the approach to computed tomography (CT) image reconstruction for a 5124 system matrix size. This work introduces a new kind of data-driven DL solver for full-size CT reconstruction without relying on the structure of direct (analytical) or iterative (numerical) inversion techniques. This work presents a feasibility demonstration of full-scale learnt reconstruction, whereas more developments will be needed to demonstrate superiority relative to traditional reconstruction approaches. The proposed approach is also extendable to other imaging problems such as emission and magnetic resonance reconstruction. More broadly, hierarchical DL opens the door to a new class of solvers for general inverse problems, which could potentially lead to improved signal-to-noise ratio, spatial resolution and computational efficiency in various areas.

Image-based malware representation approach with EfficientNet convolutional neural networks for effective malware classification

The targeted malware attacks are usually created by few crime groups. They may essentially use their existing malware sample malicious code to rebuild the variants for sophistication and evade the malware detection. This trend emphasizes the importance of performing the malware family classification for applying the effective malware mitigation and prevention strategies. In this paper, we propose an efficient neural network model EfficientNetB1 to perform the malware family classification using the malware byte level image representation technique. To alleviate the computation resource consumption caused by deep learning (DL) models training and testing the various Convolutional Neural Network (CNN) based models, we have performed the performance and computational efficiency evaluation of the various CNN pretrained models to select the best CNN network architecture for malware classification. Additionally, the CNN pretrained models are evaluated against the different types of malware image representation methods, which are distinguished based on selection of the image width size. Our evaluation of the proposed model EfficientNetB1 shows that it has achieved an accuracy of 99% to classify the Microsoft Malware Classification Challenge (MMCC) malware classes using the malware image representation with fixed image width and also require fewer network parameters compared to other pretrained models to achieve the performance accuracy. Furthermore, various visualization techniques were used to compare the performances of the various CNN pretrained models.

Invariant data-driven subgrid stress modeling in the strain-rate eigenframe for large eddy simulation

We present a new approach for constructing data-driven subgrid stress models for large eddy simulation of turbulent flows. The key to our approach is representation of model input and output tensors in the filtered strain rate eigenframe. Provided inputs and outputs are selected and non-dimensionalized in a suitable manner, this yields a model form that is symmetric, Galilean invariant, rotationally invariant, reflectionally invariant, and unit invariant. We use this model form to train a simple and efficient neural network model using only one time step of filtered direct numerical simulation data from a forced homogeneous isotropic turbulence simulation. We demonstrate the accuracy of this model as well as the model’s ability to generalize to previously unseen filter widths, Reynolds numbers, and flow physics using a priori and a posteriori tests.

Efficient Neural Network model for Cyber Crime Foreca

Efficient Neural Network model for Cyber Crime Foreca

Recognition of Edge Coherent Mode in EAST with Convolutional Neural Network

Edge-coherent mode (ECM) is one of the most promising modes in the tokamak fusion experiment, such as the Experimental Advanced Superconducting Tokamak (EAST). This paper presents an efficient convolution neural network model called NoiseNet for ECM recognition from the cross-power spectral data. NoiseNet suppresses the overfitting by applying noise in both the horizontal and vertical directions to the output of each layer of the convolution. And the improvement of the receptive field enables the convolution layer to better learn the difference between the ECM and the turbulence in the data. Experiments show that NoiseNet has better performance in ECM recognition with fewer parameters, and thus improved efficiency, than other major models, such as AlexNet, ResNet, and DenseNet. NoiseNet achieves a test accuracy of 93.94% on the ECM data sets. In addition, compared with the traditional method, this method does not depend on the empirical threshold and its generalization ability will improve with the increase in the amount of data.

Designing efficient convolutional neural network structure: A survey

As a powerful machine learning method, deep learning has attracted the attention of numerous researchers. While exploring a high-performance neural network model, the floating-point operations of a neural network model are also increasing. In recent years, many researchers have noticed that efficiency is also one of important indicators to measure the property of neural network models. Obviously, the efficient neural network model is more helpful to deploy on mobile and embedded devices. Therefore, the efficient neural network model becomes a hot research spot. In this paper, we review the methods related to the structural design of efficient convolution neural networks in recent years. According to the characteristics of these methods, we divide them into three kinds of methods: model pruning, efficient architecture, and neural architecture search. Detailed analyses of each method are presented to demonstrate their advantages and disadvantages. Then, we comprehensively compare them in detail and propose many suggestions about the design of the efficient convolution neural network model structure. Inspired by these suggestions, we built a new efficient neural network model, SharedNet. And the SharedNet obtains the best accuracy of manually-designed efficient CNN models on the ImageNet dataset.