Popular Neural Network Architectures Research Articles

Hardware Trojan Attacks on the Reconfigurable Interconnections of Field-Programmable Gate Array-Based Convolutional Neural Network Accelerators and a Physically Unclonable Function-Based Countermeasure Detection Technique.

Convolutional neural networks (CNNs) have demonstrated significant superiority in modern artificial intelligence (AI) applications. To accelerate the inference process of CNNs, reconfigurable CNN accelerators that support diverse networks are widely employed for AI systems. Given the ubiquitous deployment of these AI systems, there is a growing concern regarding the security of CNN accelerators and the potential attacks they may face, including hardware Trojans. This paper proposes a hardware Trojan designed to attack a crucial component of FPGA-based CNN accelerators: the reconfigurable interconnection network. Specifically, the hardware Trojan alters the data paths during activation, resulting in incorrect connections in the arithmetic circuit and consequently causing erroneous convolutional computations. To address this issue, the paper introduces a novel detection technique based on physically unclonable functions (PUFs) to safeguard the reconfigurable interconnection network against hardware Trojan attacks. Experimental results demonstrate that by incorporating a mere 0.27% hardware overhead to the accelerator, the proposed hardware Trojan can degrade the inference accuracy of popular neural network architectures, including LeNet, AlexNet, and VGG, by a significant range of 8.93% to 86.20%. The implemented arbiter-PUF circuit on a Xilinx Zynq XC7Z100 platform successfully detects the presence and location of hardware Trojans in a reconfigurable interconnection network. This research highlights the vulnerability of reconfigurable CNN accelerators to hardware Trojan attacks and proposes a promising detection technique to mitigate potential security risks. The findings underscore the importance of addressing hardware security concerns in the design and deployment of AI systems utilizing FPGA-based CNN accelerators.

Developing an optimized recurrent neural network model for air quality prediction using K-means clustering and PCA dimension reduction

Prediction is a means of forecasting a future value by using and analyzing historical or current data. A popular neural network architecture used as a prediction model is the Recurrent Neural Network (RNN) because of its wide application and very high generalization performance. This study aims to improve the RNN prediction model’s accuracy using k-means grouping and PCA dimension reduction methods by comparing the five distance functions. Data were processed using Python software and the results obtained from the PCA calculation yielded three new variables or principal components out of the five examined. This study used an optimized RNN prediction model with k-means clustering by comparing the Euclidean, Manhattan, Canberra, Average, and Chebyshev distance functions as a measure of data grouping similarity to avoid being trapped in the local optimal solution. In addition, PCA dimension reduction was also used in facilitating multivariate data analysis. The k-means grouping showed that the most optimal distance is the average function producing a DBI value of 0.60855 and converging at the 9th iteration. The RNN prediction model results evaluated based on the number of RMSE errors which was 0.83, while that of MAPE was 8.62%. Therefore, it was concluded that the K-means and PCA methods generated a more optimal prediction model for the RNN method.

A Novel Semantic Segmentation Algorithm for RGB-D Images Based on Non-Symmetry and Anti-Packing Pattern Representation Model

With the rapid development of deep learning technology, the accuracy of image semantic segmentation tasks has been greatly improved. However, indoor RGB-D semantic segmentation remains a challenging problem because of the complexity of indoor environments. The emergence of depth sensors makes depth information gradually used to improve the effect of semantic segmentation. The splicing of weights such as between the RGB features and the depth features which are used as the input features of the neural network can effectively improve the accuracy of the indoor semantic segmentation tasks. Most previous researches focused on improving the performance of semantic segmentation by adjusting the structure of convolutional neural network. These researches have either added attention mechanism or performed data augmentation on input features, but they didn’t make full use of the boundary information and the texture information of the original RGB image. In this paper, we propose a semantic segmentation algorithm for RGB-D images based on Non-symmetry and Anti-packing pattern representation Model (NAM). The core idea of the proposed algorithm is to take the channel-wise concatenation of pre-segmentation labels provided by the traditional hierarchical image segmentation and RGB-D features as the input of the neural network so as to guide the semantic segmentation tasks. The extensive experiments are conducted on the popular indoor RGB-D semantic segmentation datasets. When compared with the state-of-art algorithms, the experimental results presented in this paper show that our proposed method has improved the performance of image semantic segmentation networks on several popular neural network architectures.

Network Architecture Influence on Facial Emotion Recognition

Abstract Artificial Intelligence has been blending into daily life by means of many useful applications from voice command to facial recognition. One therapeutic application to be supported by AI solutions is treatment of people with Autism Spectrum Disorder. A closed loop feedback system is planned in conjunction with a novel reward system that will encourage the user to express emotions and be rewarded for it in a virtual environment. In this work five popular neural network architectures of VGG16, ResNet50, GoogleNet, ShuffleNet and EfficientNetb0 are studied and compared, with the aim of finding a relation between accuracy and developed features based on the architecture, for the application in Facial Emotion Recognition (FER). Three datasets were used, the OULU-CASIA for training and validation, alongside FACES and JAFFE for robustness analysis. The images were first preprocessed to eliminate background noise. The performance of the model was based on the true positive predictions with Grad-CAM prediction visualizations to visualize the focus of the networks in making decisions for classification. Results showed that deep network architectures with high parameter space performed best, with architecture design showing more influence on the region of focus than on classification results. This is attributed to the different layer combinations as well as parameters used for feature extraction. Shallow depth networks with high parameter space performed better than deep networks with low parameter space for FER application.

FORMULA: A Deep Learning Approach for Rare Alarms Predictions in Industrial Equipment

Predictive Maintenance technologies are particularly appealing for Industrial Equipment producers, as they pave the way to the selling of high added-value services and customized maintenance plans. However, standard Predictive Maintenance approaches assume the availability of sensor measurements, and the costs associated with adding sensors or remotely accessing sensor readings may discourage the development of such technologies. In this context, Alarm Forecasting can be very useful as it represents a low-cost alternative or helpful support to sensor-based Predictive Maintenance. In this work, we propose a new formulation for the Alarm Forecasting problem, framed as a multi-label classification task. We present a novel deep learning-based approach called FORMULA (alarm FORecasting in MUlti-LAbel setting). FORMULA leverages Transformer, a popular Neural Network architecture in the field of Natural Language Processing. To cope with alarm imbalance, we draw inspiration from Segmentation and Object Detection. Thus, FORMULA is trained by minimizing the Weighted Focal Loss, which turns out to be very effective in predicting rare alarms. These alarms, even if they are difficult to predict by nature, often are business-critical. We assess the proposed approach on a representative real-world problem from the packaging industry. In particular, we show that it outperforms not only classic multilabel techniques but also models based on recurrent neural networks. As regards the latter, the proposed approach also exhibits a lower computational burden, both in terms of training time and model size. To foster research in the field and reproducibility, we also publicly share the alarm logs dataset and the code used to perform the experiments. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper was motivated by the problem of monitoring equipment in the scenario of dairy products packaging, under the mild assumption that logs of the alarm generated by the packaging machines are available. This paper proposes an alarm forecasting algorithm. Its goal is to predict if any alarm will occur in the future, based only on past alarm logs. The limits of the considered future window can be defined arbitrarily, so there is enough time to perform corrective actions. Thus, the proposed approach aims to prevent unexpected downtime that would not only hinder productivity but also imply significant material waste. The proposed approach leverages methodologies from Natural Language Processing and Object Detection to deal with rare alarms that are often very informative in the industrial scenario. Besides, both the code and the real-world industrial datasets used to evaluate the methodology are available publicly. Currently, the proposed approach only uses alarm logs. Especially in the context of Industry 4.0, where many sensory data may be available, this is a limitation. Thus, the described approach might be extended by integrating alarm logs with sensory data. This integration is expected to improve the estimation of equipment health state. The results described in this paper may find application not only in the manufacturing sector but also in different areas such as Cyber Security, where log files keep activity records of each process performed.

CutResize: Improved data augmentation method for RGB-D Object Recognition

RGB-D object recognition is an essential but challenging task in computer vision. Most existing methods often exploit popular neural network architectures and various multi-modal fusion manners, which fail to explore the relation between low-level and high-level feature representations. To address this issue, we propose a convolutional recurrent fusion method for RGB-D object recognition. Firstly, a novel regularization strategy, called CutResize, is utilized to generate more reliable multi-scale RGB-D fusion input. Secondly, a convolutional attention gate recurrent unit (ConvAGRU) is proposed to fuse multi-modal features, which can capture the relation between low-level and high-level feature representations. Finally, the network recognizes RGB-D objects in an end-to-end manner. Especially, the CutResize uses the scaled image from another patch to mix with the input image, which avoids the mixed image only contains background and improves the localization ability of the model for small-scale features. The proposed ConvAGRU enhances the context correlation of the network and produces more compact and robust multi-modal fusion features. The proposed method is evaluated on several public datasets (including CIFAR-10, CIFAR-100, JHUIT-50, and OCID datasets). The experiment results show that the proposed method obtains better recognition performance than existing methods and achieves state-of-the-art results.

Multiclass Classification of UML Diagrams from Images Using Deep Learning

Unified Modeling Language (UML) diagrams are a recognized standard modeling language for representing design of software systems. For academic research, large cases containing UML diagrams are needed. One of the challenges in collecting such datasets is automatically determining whether an image is a UML diagram or not and what type of UML diagram an image contains. In this work, we collected UML diagrams from open datasets and manually labeled them into 10 types of UML diagrams (i.e. class diagrams, activity diagrams, use case diagrams, sequence diagrams, communication diagrams, component diagrams, deployment diagrams, object diagrams, package diagrams, and state machine diagrams) and non-UML images. We evaluated the performance of seven popular neural network architectures using transfer learning on the dataset of 4706 images, including 700 class diagrams, 454 activity diagrams, 651 use case diagrams, 706 sequence diagrams, 204 communication diagrams, 208 component diagrams, 287 deployment diagrams, 207 object diagrams, 246 package diagrams, 323 state machine diagrams, and 720 non-UML images, respectively. We also proposed our neural network architecture for multiclass classification of UML diagrams. The experiment results show that Xception achieved the best performance amongst the algorithms we evaluated with a precision of 93.03%, a recall of 92.44%, and an F1-score of 92.73%. Moreover, it is possible to develop small and almost the same efficient neural network architectures, that our proposed architecture has the least parameters (around 2.4 millions) and spends the least time per image (0.0135[Formula: see text]s per image using graphics processing unit) for classifying UML diagrams with a precision of 91.25%, a recall of 90.34%, and an F1-score of 90.79%.

Kenaf plant pest and disease detection using faster regional based convolutional neural network

&lt;span&gt;Kenaf plant is a fibre plant whose stem bark is taken to be used as raw material for making geo-textile, particleboard, pulp, fiber drain, fiber board, and paper. The presence of plant pests and diseases that attack causes crop production to decrease. The detection of pests and diseases by farmers may be a challenging task. The detection can be done using artificial intelligence-based method. Convolutional neural networks (CNNs) are one of the most popular neural network architectures and have been successfully implemented for image classification. However, the CNN method is still considered a long time in the process, so this method was developed into namely faster regional based convolution neural network (RCNN). As the selection of the input features largely determines the accuracy of the results, a pre-processing procedure is developed to transform the kenaf plant image into input features of faster RCNN. A computational experiment proves that the faster RCNN has a very short computation time by completing 10000 iterations in 3 hours compared to convolutional neural network (CNN) completing 100 iterations at the same time. Furthermore, Faster RCNN gets 77.50% detection accuracy and bounding box accuracy 96.74% while CNN gets 72.96% detection accuracy at 400 epochs. The results also prove that the selection of input features and its pre-processing procedure could produce a high accuracy of detection. &lt;/span&gt;

SmaQ: Smart Quantization for DNN Training by Exploiting Value Clustering

Advancements in modern deep learning have shown that deeper networks with larger datasets can achieve state of the art results in many different tasks. As networks become deeper, the memory requirement of neural network training proves to be the primary bottleneck of single-machine training. In this letter, we first study the characteristics of neural network weight, gradient, feature map, gradient map, and optimizer state distributions for some popular neural network architectures. Our investigation shows that the majority of the data structures used by neural networks can have their value distributions be approximated with normal distributions. We then introduce Smart Quantization (SmaQ), a quantization scheme that exploits this observed normal distribution to quantize the data structures. Our dynamic quantization method calculates the sampled mean and standard deviation of tensors and quantizes each tensor element to 6 or 8 bits based on the z-score of that value. Our scheme reduces the memory usage during training by up to 6.7x with minor losses in accuracy.

A method for reducing the impact of information risks on a megaproject life cycle based on a semantic information field

This paper consider a comprehensive method for devising loyalty programs based on the stages of a life cycle of an international megaproject. The method is based on the analysis of information risks and their management. The method is focused on aggregation and processing of data from various sources of textual information, which demonstrates the attitudes of key categories of individuals regarding the implementation of a megaproject at its numerous life cycle stages. The semantic information field is formed using hardware and software based on Neural Network Technologies. Authors examine the most popular neural network architectures that are used in the sentiment analysis. The paper describes a comparative analysis of classification accuracy of neural network architectures based on volume of texts and neural network profitability to sentiment analysis of large and small volumes of text. The method is aimed at managing and influencing information flow that accompanies the implementation of a megaproject stages. The application of semantic information field makes it possible to account for the informational risks of a megaproject and to prepare an effective set of measures to counteract these risks in a timely manner. This work was supported by RFBR grant № 20-010-00708\\20.

Drop-Activation: Implicit Parameter Reduction and Harmonious Regularization

Overfitting frequently occurs in deep learning. In this paper, we propose a novel regularization method called drop-activation to reduce overfitting and improve generalization. The key idea is to drop nonlinear activation functions by setting them to be identity functions randomly during training time. During testing, we use a deterministic network with a new activation function to encode the average effect of dropping activations randomly. Our theoretical analyses support the regularization effect of drop-activation as implicit parameter reduction and verify its capability to be used together with batch normalization (Ioffe and Szegedy in Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167 , 2015). The experimental results on CIFAR10, CIFAR100, SVHN, EMNIST, and ImageNet show that drop-activation generally improves the performance of popular neural network architectures for the image classification task. Furthermore, as a regularizer drop-activation can be used in harmony with standard training and regularization techniques such as batch normalization and AutoAugment (Cubuk et al. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 113–123, 2019). The code is available at https://github.com/LeungSamWai/Drop-Activation .

Data Integration Using Advances in Machine Learning in Drug Discovery and Molecular Biology.

While the term artificial intelligence and the concept of deep learning are not new, recent advances in high-performance computing, the availability of large annotated data sets required for training, and novel frameworks for implementing deep neural networks have led to an unprecedented acceleration of the field of molecular (network) biology and pharmacogenomics. The need to align biological data to innovative machine learning has stimulated developments in both data integration (fusion) and knowledge representation, in the form of heterogeneous, multiplex, and biological networks or graphs. In this chapter we briefly introduce several popular neural network architectures used in deep learning, namely, the fully connected deep neural network, recurrent neural network, convolutional neural network, and the autoencoder. Deep learning predictors, classifiers, and generators utilized in modern feature extraction may well assist interpretability and thus imbue AI tools with increased explication, potentially adding insights and advancements in novel chemistry and biology discovery.The capability of learning representations from structures directly without using any predefined structure descriptor is an important feature distinguishing deep learning from other machine learning methods and makes the traditional feature selection and reduction procedures unnecessary. In this chapter we briefly show how these technologies are applied for data integration (fusion) and analysis in drug discovery research covering these areas: (1) application of convolutional neural networks to predict ligand-protein interactions; (2) application of deep learning in compound property and activity prediction; (3) de novo design through deep learning. We also: (1) discuss some aspects of future development of deep learning in drug discovery/chemistry; (2) provide references to published information; (3) provide recently advocated recommendations on using artificial intelligence and deep learning in -omics research and drug discovery.

АНАЛИТИЧЕСКИЙ ОБЗОР НЕЙРОСЕТЕВЫХ АЛГОРИТМОВ ДЛЯ ОЦЕНКИ АРТЕРИАЛЬНОГО ДАВЛЕНИЯ НА ОСНОВЕ СИГНАЛОВ ЭЛЕКТРОКАРДИОГРАФИИ И ФОТОПЛЕТИЗМОГРАФИИ

Currently, the use of neural network algorithms for assessing health parameters is becoming more and more popular. The use of this mathematical apparatus in the processing and analysis of biomedical signals is of great interest to researchers. This article discusses the most popular neural network architectures used to determine human blood pressure, provides a comparative analysis of existing solutions, presents a summary table that reflects network parameters, training data and test results. In the conclusion of the work, conclusions are drawn about the prospects for using these algorithms to solve the problem of continuous monitoring of pressure.

Automated breast cancer lesion detection on breast MRI using artificial intelligence.

e14612 Background: Over 40,000 women in the US will die from breast cancer. Early detection of cancer is crucial and is a potential avenue to improve survival. The objective of this research study is to develop a convoluted neural network (CNN), a subset of artificial intelligence, in order to enhance computerized detection of breast lesions on MRIs. Methods: This is an institutional review board approved retrospective study with post contrast MRI data from 238 patients. Breast tumor segmentation was automated with a hybrid 3D/2D CNN designed adapted from U-net, a popular neural network architecture in biomedical image analysis. T1 post-contrast MRI volumes were used to train the network. The data set was separated into training (80%) and validation (20%) sets. Re-sampling and normalization using z-scores were applied to each volume before training. Contracting and expanding arms of the model consist of successive convolutions followed by batch normalization and ReLU operations. Ground truth was established through manual segmentation and previously conducted readings of the images used to train our network. Results: A 5-fold cross validation was performed for analysis. The Dice similarity coefficient was used to assess segmentation accuracy. The hybrid 3D/2D U-Net architecture yielded a Dice score of 0.753 and a Pearson correlation of 0.548 for the breast tumor segmentation. Conclusions: These results demonstrated the feasibility for artificial intelligence applications in accurately identifying the presence of lesions on breast MRI images.

Using Deep Neural Networks for Inverse Problems in Imaging: Beyond Analytical Methods

Traditionally, analytical methods have been used to solve imaging problems such as image restoration, inpainting, and superresolution (SR). In recent years, the fields of machine and deep learning have gained a lot of momentum in solving such imaging problems, often surpassing the performance provided by analytical approaches. Unlike analytical methods for which the problem is explicitly defined and domain-knowledge carefully engineered into the solution, deep neural networks (DNNs) do not benefit from such prior knowledge and instead make use of large data sets to learn the unknown solution to the inverse problem. In this article, we review deep-learning techniques for solving such inverse problems in imaging. More specifically, we review the popular neural network architectures used for imaging tasks, offering some insight as to how these deep-learning tools can solve the inverse problem. Furthermore, we address some fundamental questions, such as how deeplearning and analytical methods can be combined to provide better solutions to the inverse problem in addition to providing a discussion on the current limitations and future directions of the use of deep learning for solving inverse problem in imaging.

A Universal Approximation Theorem for Mixture-of-Experts Models.

The mixture-of-experts (MoE) model is a popular neural network architecture for nonlinear regression and classification. The class of MoE mean functions is known to be uniformly convergent to any unknown target function, assuming that the target function is from a Sobolev space that is sufficiently differentiable and that the domain of estimation is a compact unit hypercube. We provide an alternative result, which shows that the class of MoE mean functions is dense in the class of all continuous functions over arbitrary compact domains of estimation. Our result can be viewed as a universal approximation theorem for MoE models. The theorem we present allows MoE users to be confident in applying such models for estimation when data arise from nonlinear and nondifferentiable generative processes.

Training Algorithm Performance for Image Classification by Neural Networks

Adaptive training is critical for image classification by artificial neural networks (ANNS). While the machine learning community has been enthusiastic in developing various training algorithms, little research has been conducted to evaluate the performance of these algorithms in image classification by neural networks. We introduce and evaluate nine commonly-used training algorithms in terms of their performance in land-cover classification from remotely sensed data by the multi-layer-perceptron (MLP) neural networks. ALP has been considered as the most popular neural network architecture. The training algorithms we consider are Steepest Gradient Descent, Gradient Descent with Momentum, Resilient Propagation, Fletcher-Reeves, Polak-Ribiere, Powell-Beale, Scaled Conjugate Gradient, BFGS (Broyden, Fletcher, Goldfarb, and Shanno), and Levenberg-Marquardt. We use each algorithm to train the MLP networks multiple times using identical training samples, and then apply each of the resultant network models to derive land-cover information from a Landsat Enhanced Thematic Mapper Plus (ETM+) image. The training algorithms are further evaluated according to their training efficiency, capability of convergence, classification accuracy, and stability of the classification accuracy. It is found that the performance of these algorithms varies substantially and selecting an appropriate algorithm can lead to a fast and efficient training and an increase in land-cover classification accuracy by artificial neural networks.

Automatic Cluster Detection in Kohonen's SOM

Kohonen's self-organizing map (SOM) is a popular neural network architecture for solving problems in the field of explorative data analysis, clustering, and data visualization. One of the major drawbacks of the SOM algorithm is the difficulty for nonexpert users to interpret the information contained in a trained SOM. In this paper, this problem is addressed by introducing an enhanced version of the Clusot algorithm. This algorithm consists of two main steps: 1) the computation of the Clusot surface utilizing the information contained in a trained SOM and 2) the automatic detection of clusters in this surface. In the Clusot surface, clusters present in the underlying SOM are indicated by the local maxima of the surface. For SOMs with 2-D topology, the Clusot surface can, therefore, be considered as a convenient visualization technique. Yet, the presented approach is not restricted to a certain type of 2-D SOM topology and it is also applicable for SOMs having an n-dimensional grid topology.