Neural Network Operators Research Articles

Near-Data Processing in Memory Expander for DNN Acceleration on GPUs

We propose a near-data processing (NDP) architecture that exploits a memory expander with byte-addressable memory-semantic interconnect to accelerate memory-bound operations in deep neural networks (DNNs). Our architecture can execute NDP operations on the memory traffic from the GPU on-the-fly by employing bump-in-the-wire NDP logic between the off-chip link and memory controller. In addition, the memory-bound operations executed on the NDP unit can be effectively overlapped with compute-intensive operations executed on a GPU, even if the two operations have a dependency. Furthermore, the NDP offloading can be automatically done by the compiler without any code modification by deep learning practitioners. Our approach can achieve a 51% speedup for training VGG-16 with batch normalization.

Medical imaging deep learning with differential privacy

The successful training of deep learning models for diagnostic deployment in medical imaging applications requires large volumes of data. Such data cannot be procured without consideration for patient privacy, mandated both by legal regulations and ethical requirements of the medical profession. Differential privacy (DP) enables the provision of information-theoretic privacy guarantees to patients and can be implemented in the setting of deep neural network training through the differentially private stochastic gradient descent (DP-SGD) algorithm. We here present deepee, a free-and-open-source framework for differentially private deep learning for use with the PyTorch deep learning framework. Our framework is based on parallelised execution of neural network operations to obtain and modify the per-sample gradients. The process is efficiently abstracted via a data structure maintaining shared memory references to neural network weights to maintain memory efficiency. We furthermore offer specialised data loading procedures and privacy budget accounting based on the Gaussian Differential Privacy framework, as well as automated modification of the user-supplied neural network architectures to ensure DP-conformity of its layers. We benchmark our framework’s computational performance against other open-source DP frameworks and evaluate its application on the paediatric pneumonia dataset, an image classification task and on the Medical Segmentation Decathlon Liver dataset in the task of medical image segmentation. We find that neural network training with rigorous privacy guarantees is possible while maintaining acceptable classification performance and excellent segmentation performance. Our framework compares favourably to related work with respect to memory consumption and computational performance. Our work presents an open-source software framework for differentially private deep learning, which we demonstrate in medical imaging analysis tasks. It serves to further the utilisation of privacy-enhancing techniques in medicine and beyond in order to assist researchers and practitioners in addressing the numerous outstanding challenges towards their widespread implementation.

RISC-V Virtual Platform-Based Convolutional Neural Network Accelerator Implemented in SystemC

The optimization for hardware processor and system for performing deep learning operations such as Convolutional Neural Networks (CNN) in resource limited embedded devices are recent active research area. In order to perform an optimized deep neural network model using the limited computational unit and memory of an embedded device, it is necessary to quickly apply various configurations of hardware modules to various deep neural network models and find the optimal combination. The Electronic System Level (ESL) Simulator based on SystemC is very useful for rapid hardware modeling and verification. In this paper, we designed and implemented a Deep Learning Accelerator (DLA) that performs Deep Neural Network (DNN) operation based on the RISC-V Virtual Platform implemented in SystemC in order to enable rapid and diverse analysis of deep learning operations in an embedded device based on the RISC-V processor, which is a recently emerging embedded processor. The developed RISC-V based DLA prototype can analyze the hardware requirements according to the CNN data set through the configuration of the CNN DLA architecture, and it is possible to run RISC-V compiled software on the platform, can perform a real neural network model like Darknet. We performed the Darknet CNN model on the developed DLA prototype, and confirmed that computational overhead and inference errors can be analyzed with the DLA prototype developed by analyzing the DLA architecture for various data sets.

A pipelining strategy for accelerating convolution neural networks on ARM CPUs

AbstractConvolution is a primary operation in convolution neural networks. The speed of inference is mainly decided by the speed of the convolutional layer. Improving the performance of embedded processors makes it possible to process the inference on embedded devices. In this article, a pipelining strategy of single instruction and multiple data (SIMD) instructions is proposed to finely optimize the process of the 3 × 3 convolution on ARM‐based CPUs. We implement the SIMD group to improve the efficiency of the SIMD pipeline. A tiling method is exploited to increase data reuse during the process. An evaluation model is proposed to guide the design of the tiling method and register allocation. The speed of our implementation is 5.18 times of the GNU compiler collection compiled unoptimized version on RK3288. The effect of our optimizing method is measured by a performance profiling tool, the performance information suggests that the pipelining strategy has a significant effect for both normal and depthwise separable convolution. By implementing multithread processing, the speedup achieves 18.3 compared with the single thread unoptimized version.

Design Space Exploration of Matrix–Matrix Convolution Operation

Convolution is an important operation in neural networks which, in recent years, received significant attention from the researchers thanks to its ability to handle complex tasks such as image processing, computer vision in an efficient manner. In general, the convolution operation in neural networks considers two matrices as inputs: an image matrix representing an image and a kernel matrix required for necessary image processing operation and performs several multiplications and addition operations among the elements of image and kernel matrices. Realizing a circuit structure for matrix–matrix convolution is straightforward as each multiplication is realized by a multiplier, whereas an addition is carried out by an adder. However, the corresponding circuits result in large area, high power consumption and long delay because of the large number of multiplications and additions that are involved in the matrix–matrix convolution operations. While, the existing approaches focus on the accelerations of this computationally intensive tasks, they often do not guarantee minimality of area, power and delay. But we show that there exists design aspects through which the circuit structures for convolution operations can be realized with less area, power and delay. To do this, we consider the kernel definitions during the design of the circuit structures since the kernel matrices are often (pre)-determined based on the desired applications. Motivated by this, we first explore the design space of the convolution operation by introducing an alternative design scheme for realizing the respective operation between two matrices keeping the image processing/neural network applications in mind. Experimental evaluations confirm the potential benefits of the proposed design scheme and demonstrate that the reductions in the area and power by approximately [Formula: see text] and critical path delay by approximately [Formula: see text] can be achieved using the proposed design scheme. In addition, the FPGA implementations of the proposed scheme also show that the reductions of approximately [Formula: see text] and [Formula: see text] in the number of LUTs and in the number of pins, respectively, can be achieved. Compared to prior works, the proposed scheme allows higher parallelism with minimum LUT utilization.

Combining max-pooling and wavelet pooling strategies for semantic image segmentation

This paper presents a novel multi-pooling architecture generated by combining the advantages of wavelet and max-pooling operations in convolutional neural networks (CNNs), focusing on semantic segmentation tasks. CNNs often use pooling to reduce the number of parameters, improve invariance to certain distortions, and enlarge the receptive field. However, pooling can cause information loss and thus is detrimental to further operations such as feature extraction and analysis. This problem is particularly critical for semantic segmentation, where each pixel of an image is assigned to a specific class to divide the image into disjoint regions of interest. To address this problem, pooling strategies based on wavelets-operations have been proposed with the promise to achieve a better trade-off between receptive field size and computational efficiency. Previous works have confirmed the superiority of wavelet pooling over the traditional one in semantic segmentation tasks. However, we have observed in our computational experiments that the expressive gains reported from the use of wavelet pooling in other segmentation tasks were not observed in the scope of aerial imagery due to imprecision in the segmentation of image details. The combination of wavelet pooling and max-pooling, a solution not yet reported in the literature, can address that issue. Such gap observed in the pooling area motivated the two proposals that are the main contributions of this paper: (a) A new multi-pooling strategy combining wavelet and traditional pooling in a new network structure suitable for aerial image segmentation tasks; (b) Two-stream architectures using the traditional max-pooling and wavelet pooling as streams. These proposals were implemented using the Segnet, a known architecture for semantic segmentation. The computational experiments, based on the IRRG images from the Potsdam and Vaihingen data sets, demonstrated that the proposed architectures surpassed the original Segnet architecture’s performance with results comparable to state-of-the-art approaches.

The art of molecular computing: Whence and whither.

An astonishingly diverse biomolecular circuitry orchestrates the functioning machinery underlying every living cell. These biomolecules and their circuits have been engineered not only for various industrial applications but also to perform other atypical functions that they were not evolved for-including computation. Various kinds of computational challenges, such as solving NP-complete problems with many variables, logical computation, neural network operations, and cryptography, have all been attempted through this unconventional computing paradigm. In this review, we highlight key experiments across three different ''eras'' of molecular computation, beginning with molecular solutions, transitioning to logic circuits and ultimately, more complex molecular networks. We also discuss a variety of applications of molecular computation, from solving NP-hard problems to self-assembled nanostructures for delivering molecules, and provide a glimpse into the exciting potential that molecular computing holds for the future. Also see the video abstract here: https://youtu.be/9Mw0K0vCSQw.

Demand Response Package Model of Electric Vehicle Charging Station Based on LSTM Neural Network and Optimal Operation of Distribution Network

Demand Response Package Model of Electric Vehicle Charging Station Based on LSTM Neural Network and Optimal Operation of Distribution Network

End-point Temperature Preset of Molten Steel in the Final Refining Unit Based on an Integration of Deep Neural Network and Multi-process Operation Simulation

End-point Temperature Preset of Molten Steel in the Final Refining Unit Based on an Integration of Deep Neural Network and Multi-process Operation Simulation

Approximation by exponential sampling type neural network operators

Approximation by exponential sampling type neural network operators

Predicting mean ribosome load for 5'UTR of any length using deep learning.

The 5' untranslated region plays a key role in regulating mRNA translation and consequently protein abundance. Therefore, accurate modeling of 5'UTR regulatory sequences shall provide insights into translational control mechanisms and help interpret genetic variants. Recently, a model was trained on a massively parallel reporter assay to predict mean ribosome load (MRL)-a proxy for translation rate-directly from 5'UTR sequence with a high degree of accuracy. However, this model is restricted to sequence lengths investigated in the reporter assay and therefore cannot be applied to the majority of human sequences without a substantial loss of information. Here, we introduced frame pooling, a novel neural network operation that enabled the development of an MRL prediction model for 5'UTRs of any length. Our model shows state-of-the-art performance on fixed length randomized sequences, while offering better generalization performance on longer sequences and on a variety of translation-related genome-wide datasets. Variant interpretation is demonstrated on a 5'UTR variant of the gene HBB associated with beta-thalassemia. Frame pooling could find applications in other bioinformatics predictive tasks. Moreover, our model, released open source, could help pinpoint pathogenic genetic variants.

Fractional type multivariate neural network operators

In this paper, we introduce a novel family of multivariate neural network operators involving Riemann‐Liouville fractional integral operator of order α. Their pointwise and uniform approximation results are presented, and new results concerning the rate of convergence in terms of the modulus of continuity are estimated. Moreover, several graphical and numerical results are presented to demonstrate the accuracy, applicability, and efficiency of the operators through special activation functions. Finally, an illustrative real‐world example on the recent trend of novel corona virus Covid‐19 has been investigated in order to demonstrate the modeling capabilities of the proposed neural network operators.

Enhancing threshold neural network via suprathreshold stochastic resonance for pattern classification

Hard-threshold nonlinearities are of significant interest for neural-network information processing due to their simplicity and low-cost implementation. They however lack an important differentiability property. Here, hard-threshold nonlinearities receiving assistance from added noise are pooled into a large-scale summing array to approximate a neuron with a noise-smoothed activation function. Differentiability that facilitates gradient-based learning is restored for such neurons, which are assembled into a feed-forward neural network. The added noise components used to smooth the hard-threshold responses have adjustable parameters that are adaptively optimized during the learning process. The converged non-zero optimal noise levels establish a beneficial role for added noise in operation of the threshold neural network. In the retrieval phase the threshold neural network operating with non-zero optimal added noise, is tested for data classification and for handwritten digit recognition, which achieves state-of-the-art performance of existing backpropagation-trained analog neural networks, while requiring only simpler two-state binary neurons.

Ferroelectric Field Effect Transistors as a Synapse for Neuromorphic Application

In spite of the increasing use of machine learning techniques, in-memory computing and hardware have increased the interest to accelerate neural network operation. Henceforth, novel embedded nonvolatile memories (eNVMs) for highly scaled technology nodes, like ferroelectric field effect transistors (FeFETs), are heavily studied and very promising. Furthermore, inference and on-chip learning can be fostered by further eNVM technology options, such as multibit operation and linear switching. In this article, we present the advantages of hafnium oxide-based FeFETs for such purposes due to their basic three-terminal structure, which allows to selectively activate or deactivate selected devices as well as tune linearity and dynamic range for certain applications. Furthermore, we discuss the impact of the material properties of the ferroelectric layer, the interface layer thickness, and scaling on the device performance. Here, we demonstrate good device properties even for highly scaled devices ( $100\,\,nm \times 100$ nm).

Model of an Artificial Neural Network for Solving the Problem of Controlling a Genetic Algorithm Using the Mathematical Apparatus of the Theory of Petri Nets

The aim of the study was to increase the speed, quantity and quality of solutions in intelligent systems aimed at solving the problem of structural–parametric synthesis of models of large discrete systems with a given behavior. As a hypothesis, it was assumed that the adapted model of an artificial neural network is able to control changes in the parameters of the functioning of the operators of the genetic algorithm directly in the process of solving the problem of intelligent structural–parametric synthesis of models of large discrete systems. To solve the problem of managing the process of intelligent search for solutions based on a genetic algorithm, an artificial neural network, which is used as an add-in, must dynamically change the “destructive” ability of operators based on data about the current and/or historical state of the population. In the article, the theory of Petri nets is used as a single mathematical device capable of modeling the work of evolutionary procedures. This mathematical tool is able to simulate the operation of a genetic algorithm adapted to solving the problem of structural–parametric synthesis of models of large discrete systems with a given behavior; simulate the operation and training of an artificial neural network; combine the genetic algorithm with a control add-in based on an artificial neural network to prevent attenuation and premature convergence; simulate the process of recognizing the state of the population; and simulate the operation of the models obtained as a result of the synthesis. As an example of the functioning of the proposed approach, the article presents the results of a computational experiment, which considers the problem of structural–parametric synthesis of computer technology based on the developed models of the element base-RS, D and T triggers that are capable of processing a given input vector into the required (reference) output. In the software implementation of the proposed approach, calculations on the CPU and CPU+GPGPU technologies were used.

CRBA: A Competitive Rate-Based Algorithm Based on Competitive Spiking Neural Networks.

In this paper we present a Competitive Rate-Based Algorithm (CRBA) that approximates operation of a Competitive Spiking Neural Network (CSNN). CRBA is based on modeling of the competition between neurons during a sample presentation, which can be reduced to ranking of the neurons based on a dot product operation and the use of a discrete Expectation Maximization algorithm; the latter is equivalent to the spike time-dependent plasticity rule. CRBA's performance is compared with that of CSNN on the MNIST and Fashion-MNIST datasets. The results show that CRBA performs on par with CSNN, while using three orders of magnitude less computational time. Importantly, we show that the weights and firing thresholds learned by CRBA can be used to initialize CSNN's parameters that results in its much more efficient operation.

Radio-Frequency Multiply-and-Accumulate Operations with Spintronic Synapses

Exploiting the physics of nanoelectronic devices is a major lead for implementing compact, fast, and energy efficient artificial intelligence. In this work, we propose an original road in this direction, where assemblies of spintronic resonators used as artificial synapses can classify an-alogue radio-frequency signals directly without digitalization. The resonators convert the ra-dio-frequency input signals into direct voltages through the spin-diode effect. In the process, they multiply the input signals by a synaptic weight, which depends on their resonance fre-quency. We demonstrate through physical simulations with parameters extracted from exper-imental devices that frequency-multiplexed assemblies of resonators implement the corner-stone operation of artificial neural networks, the Multiply-And-Accumulate (MAC), directly on microwave inputs. The results show that even with a non-ideal realistic model, the outputs obtained with our architecture remain comparable to that of a traditional MAC operation. Us-ing a conventional machine learning framework augmented with equations describing the physics of spintronic resonators, we train a single layer neural network to classify radio-fre-quency signals encoding 8x8 pixel handwritten digits pictures. The spintronic neural network recognizes the digits with an accuracy of 99.96 %, equivalent to purely software neural net-works. This MAC implementation offers a promising solution for fast, low-power radio-fre-quency classification applications, and a new building block for spintronic deep neural net-works.

Cambricon-G: A Polyvalent Energy-Efficient Accelerator for Dynamic Graph Neural Networks

Graph neural networks (GNNs), which extend traditional neural networks for processing graph-structured data, have been widely used in many fields. The GNN computation mainly consists of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">edge processing</i> to generate messages by combining the edge/vertex features and the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">vertex processing</i> to update the vertex features with aggregated messages. In addition to nontrivial vector operations in the edge processing, huge random accesses and neural network operations in the vertex processing, the graph topology of GNNs may also vary during the computation (i.e., dynamic GNNs). The above characteristics pose significant challenges on existing architectures. In this article, we propose a novel accelerator named CAMBRICON-G for efficient processing of both dynamic and static GNNs. The key of CAMBRICON-G is to abstract the irregular computation of a broad range of GNN variants to the process of regularly tiled <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">adjacent cuboid</i> (which extends the traditional adjacent matrix of graph by adding the dimension of vertex features). The intuition is that the adjacent cuboid facilitates exploitation of both data locality and parallelism by offering <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multidimensional multilevel tiling</i> (including spatial and temporal tiling) opportunities. To perform the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multidimensional spatial tiling</i> , the CAMBRICON-G architecture mainly consists of the cuboid engine (CE) and hybrid on-chip memory. The CE has multiple vertex processing units (VPUs) working in a coordinated manner to efficiently process the sparse data and dynamically update the graph topology with dedicated instructions. The hybrid on-chip memory contains the topology-aware cache and multiple scratchpad memory to reduce off-chip memory access. To perform the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multidimensional temporal tiling</i> , an easy-to-use programming model is provided to flexibly explore different tiling options for large graphs. Experimental results show that compared against Nvidia P100 GPU, the performance and energy efficiency can be improved by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$7.14\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$20.18\times $ </tex-math></inline-formula> , respectively, on various GNNs, which validates both the versatility and energy efficiency of CAMBRICON-G.

Research on the Role of Influencing Factors on Hotel Customer Satisfaction Based on BP Neural Network and Text Mining

With the flourishing development of the hotel industry, the study of customer satisfaction based on online reviews and data has become a new model. In this paper, customer reviews and ratings on Ctrip.com are used, and TF-IDF and K-means algorithms are used to extract and cluster the keywords of reviews texts. Finally, 10 first-level influencing factors of hotel customer satisfaction are determined: epidemic prevention, consumption emotion, convenience, environment, facilities, catering, target group, perceived value, price, and service. Based on backpropagation neural network and weight matrix operation, an influencing factor analysis model of hotel customer satisfaction is constructed to explore the role of these factors. The results show that consumption emotion, perceived value, epidemic prevention, target group, and convenience would significantly affect customer satisfaction, among which epidemic prevention becomes a new factor affecting customer satisfaction. Environment, facilities, catering, and service have relatively little effect on customer satisfaction, while price has the least effect. This study provides a path and method for online reviews of hotel management to improve customer satisfaction and provides a theoretical basis for the study of online reviews of hotels.

Automatic Modulation Recognition Based on a DCN-BiLSTM Network.

Automatic modulation recognition (AMR) is a significant technology in noncooperative wireless communication systems. This paper proposes a deep complex network that cascades the bidirectional long short-term memory network (DCN-BiLSTM) for AMR. In view of the fact that the convolution operation of the traditional convolutional neural network (CNN) loses the partial phase information of the modulated signal, resulting in low recognition accuracy, we first apply a deep complex network (DCN) to extract the features of the modulated signal containing phase and amplitude information. Then, we cascade bidirectional long short-term memory (BiLSTM) layers to build a bidirectional long short-term memory model according to the extracted features. The BiLSTM layers can extract the contextual information of signals well and address the long-term dependence problems. Next, we feed the features into a fully connected layer. Finally, a softmax classifier is used to perform classification. Simulation experiments show that the performance of our proposed algorithm is better than that of other neural network recognition algorithms. When the signal-to-noise ratio (SNR) exceeds 4 dB, our model’s recognition rate for the 11 modulation signals can reach 90%.