Fully-connected Layers Research Articles

FPGA–accelerated CNN for real-time plant disease identification

Using convolutional neural network (CNN) to identify plant diseases in-situ is a hot research topic in smart agriculture. Due to the memory-intensive and compute-intensive characteristics of CNN algorithm, it is difficult to implement CNN on edge terminals with limited memory and computational resources. In this paper, Field Programmable Gate Array (FPGA) is used to accelerate CNN to identify plant diseases. First, a 7-layer simple-structured network called “LiteCNN”, with only 176 K parameters and 78.47 M floating point operations (FLOPs) was designed. And knowledge distillation method was used to train LiteCNN, making that the accuracy reaches 95.24 %. Secondly, the acceleration circuit of LiteCNN was designed and implemented on “ZYNQ Z7-Lite 7020″ FPGA board. To compress the network and speed up plant disease identification, the following methods were applied: 1) Separable convolution took place of regular convolution, and a low-redundancy block convolution approach was used to load data; 2) The Batch Normalization (BN) layer was fused into the previous convolutional layer (or fully-connected layer); 3) Feature data and model parameters were expressed by half float data. As the basic function of the circuit achieved, methods including unrolling the for-loop, pipelining the for-loop, loop flattening and array partitioning were used to optimize the parallelism of the circuit. Finally, LiteCNN on the FPGA board was verified. The plant disease identification accuracy was 95.71 %, the inference speed was 0.071 s per frame, and the power consumption was 2.41 W. The results show that this paper proposed a low-power, high-accuracy and fast-speed plant disease identification terminal, which can be well applied for real-time plant disease identification in the field.

Transfer learning-based crashworthiness prediction for the composite structure of a subway vehicle

Due to the lack of load/displacement sensors in a complex and uncertain crash test/accident of rail vehicles (e.g., vehicle-to-vehicle or train-to-train collision), only structural deformation images can be obtained while the crashworthiness indicators (e.g., force, displacement, energy absorption) cannot be measured directly. This paper aims to propose a transfer learning-based inverse method for extracting the structural parameters and crashworthiness characteristics by the deformed pictures of energy-absorbing structures. A finite element model of an energy-absorbing structure was firstly established and calibrated by experiments. Then, a number of deformation images were captured from the numerical design of experiment (DOE) through coding languages, which were saved as TFRecord format to reduce the computational time during the training of transfer learning models (i.e., VGG16, LetNet, AlexNet and ResNet50). The result showed that the transfer learning model, ResNet50, exhibited the best performance with R2 of 0.736 and 0.981, respectively, for predicting the structural parameters and crashworthiness characteristics. In addition, the number of full connection layers should be reasonably selected on the premise of maintaining accuracy and efficiency. A group of deformation pictures were randomly used as samples to validate the prediction of structural parameters and crashworthiness through the trained transfer learning model, where good consistence was observed. The proposed method is expected to bring the image recognition and big data prediction into the design and test of composite energy-absorbing structures, thus, auxiliary improve the crashworthiness of rail vehicles.

Research on Weld Defect Detection and Evaluation Technology based on Deep Learning

Aiming at the problems of low efficiency and strong subjectivity in the current detection of weld defects by radiographic imaging technology, an object detection method of weld defects based on multi-channel fusion convolutional neural network is proposed. In this method, the images of weld defects are encoded and input into multiple feature extraction channels formed by parallel fusion of CNN. After that, the extracted features are fused with full connection layer and the feature vectors are output. Finally, the final output is obtained by Softmax for classification. The proposed method is verified by weld defect images in actual production. The experimental results indicate that the mAP of the multi-channel fusion convolutional neural network reaches 76.37%, and the detection accuracy of weld defects is higher than that of other network such as ResNet-50 and VGG-16. The proposed method can be applied to X-ray intelligent detection of weld defects and other scenarios.

A concise self-adapting deep learning network for machine remaining useful life prediction

Most remaining useful life (RUL) prediction methods learn the feature using a single fixed pattern, resulting in a lack of self-adapting learning capability and a decrease in generalization and prediction accuracy. To address this issue, we propose a concise self-adapting deep learning network (CSDLN) for RUL prediction with fewer learnable parameters per sub-module. First, a multi-branch 1D involution neural network (MINN) is proposed to adaptively extract the hidden feature from the multi-input using the involution operation, which has inverse inherence with the convolution operation. Second, an adaptive learning algorithm called the multi-head gated recurrent unit (MGRU) is proposed to learn the hidden feature. Finally, the aero-engine RUL is determined by dimension reduction of the full connection (FC) layer and linear regression of the regression layer. Additionally, rather than using the ReLU activation function, the Mish activation function is used to strengthen the self-adapting deep learning ability in CSDLN. The performance in the Commercial Modular Aero Propulsion System Simulation (C-MAPSS) dataset and the real wind turbine gearbox bearing tests demonstrate the superiority of CSDLN over the state-of-the-art RUL prediction methods. Meanwhile, dropout is adopted in the model for avoiding overfitting and achieving the uncertainty quantification of RUL prediction in the two applications.

Power fingerprint identification based on the improved V-I trajectory with color encoding and transferred CBAM-ResNet.

In power fingerprint identification, feature information is insufficient when using a single feature to identify equipment, and small load data of specific customers, difficult to meet the refined equipment classification needs. A power fingerprint identification based on the improved voltage-current(V-I) trajectory with color encoding and transferred CBAM-ResNet34 is proposed. First, the current, instantaneous power, and trajectory momentum information are added to the original V-I trajectory image using color coding to obtain a color V-I trajectory image. Then, the ResNet34 model was pre-trained using the ImageNet dataset and a new fully-connected layer meeting the device classification goal was used to replace the fully-connected layer of ResNet34. The Convolutional Block Attention Module (CBAM) was added to each residual structure module of ResNet34. Finally, Class-Balanced (CB) loss is introduced to reweight the Softmax cross-entropy (SM-CE) loss function to solve the problem of data imbalance in V-I trajectory identification. All parameters are retrained to extract features from the color V-I trajectory images for device classification. The experimental results on the imbalanced PLAID dataset verify that the method in this paper has better classification capability in small sample imbalanced datasets. The experimental results show that the method effectively improves the identification accuracy by 4.4% and reduces the training time of the model by 14 minutes compared with the existing methods, which meets the accuracy requirements of fine-grained power fingerprint identification.

A fully-mapped and energy-efficient FPGA accelerator for dual-function AI-based analysis of ECG.

In this paper, a fully-mapped field programmable gate array (FPGA) accelerator is proposed for artificial intelligence (AI)-based analysis of electrocardiogram (ECG). It consists of a fully-mapped 1-D convolutional neural network (CNN) and a fully-mapped heart rate estimator, which constitute a complementary dual-function analysis. The fully-mapped design projects each layer of the 1-D CNN to a hardware module on an Intel Cyclone V FPGA, and a virtual flatten layer is proposed to effectively bridge the feature extraction layers and fully-connected layer. Also, the fully-mapped design maximizes computational parallelism to accelerate CNN inference. For the fully-mapped heart rate estimator, it performs pipelined transformations, self-adaptive threshold calculation, and heartbeat count on the FPGA, without multiplexed usage of hardware resources. Furthermore, heart rate calculation is elaborately analyzed and optimized to remove division and acceleration, resulting in an efficient method suitable for hardware implementation. According to our experiments on 1-D CNN, the accelerator can achieve 43.08× and 8.38× speedup compared with the software implementations on ARM-Cortex A53 quad-core processor and Intel Core i7-8700 CPU, respectively. For the heart rate estimator, the hardware implementations are 25.48× and 1.55× faster than the software implementations on the two aforementioned platforms. Surprisingly, the accelerator achieves an energy efficiency of 63.48 GOPS/W, which obviously surpasses existing studies. Considering its power consumption is only 67.74mW, it may be more suitable for resource-limited applications, such as wearable and portable devices for ECG monitoring.

НЕЙРОСЕТЕВОЙ АЛГОРИТМ ДЕТЕКТИРОВАНИЯ СИГНАЛОВ С ДВОИЧНОЙ ПОЗИЦИОННО-ИМПУЛЬСНОЙ МОДУЛЯЦИЕЙ

In the work, a neural network method for detecting signals with binary position-pulse modulation was studied. The detection process was reduced to distinguishing signals by a neural network, the architecture of which included convolutional and fully-connected layers. The network was trained on the basis of a set of labeled realizations of a sum of signals with noise, which were represented by discrete samples. The TensorFlow library was used to model and train the network. A bell video pulse and a quasi-radio signal with a bell modulating function were used as possible signal models. The functioning efficiency of the trained neural network was studied by the methods of statistical modeling on a computer. The detection quality of the neural network algorithm is compared with the classical one synthesized in the framework of the statistical decision theory. The possibility of training a neural network detector with an unknown duration of the received signals is investigated. The efficiency of the neural network detector is compared with the quasi-optimal one, which does not take into account the possible unknown of the signal duration. It is shown that the neural network detection algorithm is significantly inferior to the optimal one synthesized by the maximum likelihood method. However, in the presence of a priori uncertainty about the signal duration, the quasi-optimal detector is inferior in efficiency to the neural network one. Recommendations are formulated for a reasonable choice of the detector, depending on the required quality of its work.

Power and Area-Efficient XNOR-AND Hybrid Binary Neural Networks Using TFT-Type Synaptic Device

A novel XNOR- AND hybrid binary neural network (BNN) using a TFT-type synaptic device is proposed to reduce the area and power consumption of the synaptic array. Replacing some parts of the network from the XNOR operation to the AND operation results in expressing weight and input with a single cell and single word line. In the case of replacing the operation of the fully-connected (FC) layer with the AND operation, the accuracy of VGG9 BNN for CIFAR-10 datasets drops by about 1%, while the number of cells in the synaptic array decreases by 33.7%. Using the previously proposed TFT-type synaptic devices, the proposed method reduces power consumption by ~25%.

Development of a vertex finding algorithm using Recurrent Neural Network

Deep learning is a rapidly-evolving technology with possibility to significantly improve physics reach of collider experiments. In this study we developed a novel algorithm of vertex finding for future lepton colliders such as the International Linear Collider. We deploy two networks; one is simple fully-connected layers to look for vertex seeds from track pairs, and the other is a customized Recurrent Neural Network with an attention mechanism and an encoder-decoder structure to associate tracks to the vertex seeds. The performance of the vertex finder is compared with the standard ILC reconstruction algorithm.

COVID-19 Detection Using Forced Cough Sounds and Medical Information

The World Health Organization (WHO) has declared the novel coronavirus (COVID-19) outbreak a global pandemic in March 2020. Through a lot of cooperation and the effort of scientists, several vaccines have been created. However, there is no guarantee that the virus will shortly disappear, even if a large part of the population is vaccinated. Therefore, non-invasive methods, with low cost and real-time results, are important to detect infected individuals and enable earlier adequate treatment, in addition to preventing the spread of the virus. An alternative is using forced cough sounds and medical information to distinguish a healthy person from those infected with COVID-19 via artificial intelligence. An additional challenge is the unbalancing of these data, as there are more samples of healthy individuals than contaminated ones. We propose here a Deep Neural Network model to classify people as healthy or sick concerning COVID-19. We used here a model composed by an Convolutional Neural Network and two other Neural Networks with two full-connected layers, each one trained with different data from the same individual. To evaluate the performance of the proposed method, we combined two datasets from the literature: COUGHVID and Coswara. That dataset contains clinical information regarding previous respiratory conditions, symptoms (fever or muscle pain), and a cough record. The results show that our model is simpler (with fewer parameters) than those from the literature and generalizes better the prediction of infected individuals. The proposal presents an average Area Under the ROC Curve (AUC) equal to 0.885 with a confidence interval (0.881 - 0.888), while the literature reports 0.771 with (0.752 - 0.783).

Transfer learning for the efficient detection of COVID-19 from smartphone audio data

Disease detection from smartphone data represents an open research challenge in mobile health (m-health) systems. COVID-19 and its respiratory symptoms are an important case study in this area and their early detection is a potential real instrument to counteract the pandemic situation. The efficacy of this solution mainly depends on the performances of AI algorithms applied to the collected data and their possible implementation directly on the users’ mobile devices. Considering these issues, and the limited amount of available data, in this paper we present the experimental evaluation of 3 different deep learning models, compared also with hand-crafted features, and of two main approaches of transfer learning in the considered scenario: both feature extraction and fine-tuning. Specifically, we considered VGGish, YAMNET, and L3-Net (including 12 different configurations) evaluated through user-independent experiments on 4 different datasets (13,447 samples in total). Results clearly show the advantages of L3-Net in all the experimental settings as it overcomes the other solutions by 12.3% in terms of Precision–Recall AUC as features extractor, and by 10% when the model is fine-tuned. Moreover, we note that to fine-tune only the fully-connected layers of the pre-trained models generally leads to worse performances, with an average drop of 6.6% with respect to feature extraction. Finally, we evaluate the memory footprints of the different models for their possible applications on commercial mobile devices.

Multi-omics integration method based on attention deep learning network for biomedical data classification

Background and objectiveIntegrating multi-omics data for the comprehensive analysis of the biological processes in human diseases has become one of the most challenging tasks of bioinformatics. Deep learning (DL) algorithms have recently become one of the most promising multi-omics data integration analysis methods. However, existing DL-based studies almost integrate the multi-omics data by concatenation in the input data space or the learned feature space, ignoring the correlations between patients and omics. MethodsWe propose a novel multi-omics integration method, called Multi-omics Attention Deep Learning Network (MOADLN), which is used for biomedical data classification. Firstly, for each type of omics data, we use three fully-connected layers and the self-attention mechanism to reduce dimensionality, and construct the correlations between patients, respectively. Then, we apply the feature vector learned from self-attention to generate the initial category labels. Secondly, for the initial label predicted of each omics data, we use an effective Multi-Omics Correlation Discovery Network (MOCDN) to learn the cross-omic correlations in the label space. Finally, we use the softmax classifier for label prediction. ResultsWe demonstrate that our method outperforms several state-of-the-art methods on two datasets with mRNA expression data, DNA methylation data, and miRNA expression data. In addition, we identified essential biomarkers of relevant diseases by MOADLN, and the generality of MOADLN is also demonstrated in the KIRP and KIRC datasets. ConclusionsMOADLN jointly explores correlations between patients in intra-omics and correlations of cross-omics in label space, which is an effective DL-based classification of biomedical data.

An intelligent crash recognition method based on 1DResNet-SVM with distributed vibration sensors

The vibration recognition along the fiber is still a challenge in highway accident monitoring with distributed optical fiber vibration sensing system (DVS). In this paper, a method named 1DResNet-SVM combining 1D residual Neural Network (1DResNet) and support vector machines (SVM) is proposed. One-dimensional raw vibration signals are used to avoid information loss of data. One-dimensional convolution is combined with residual block to solve the degradation of deep network models. Experiments show that the proposed 1DResNet can extract the distinguishable characteristics of DVS vibration signals better than traditional manual feature extraction methods. Then SVM is selected to replace the fully-connected layer in the network which can effectively recognize crash. The proposed 1DResNet-SVM model can achieve an average recognition rate of 96.2%. Three typical events in highway accidents are carried out in field tests, which can reach 100% especially for crash recognition. This paper provides a new method for the identification of crash accidents.

Towards optimal deep fusion of imaging and clinical data via a model-based description of fusion quality.

Due to intrinsic differences in data formatting, data structure, and underlying semantic information, the integration of imaging data with clinical data can be non-trivial. Optimal integration requires robust data fusion, that is, the process of integrating multiple data sources to produce more useful information than captured by individual data sources. Here, we introduce the concept of fusion quality for deep learning problems involving imaging and clinical data. We first provide a general theoretical framework and numerical validation of our technique. To demonstrate real-world applicability, we then apply our technique to optimize the fusion of CT imaging and hepatic blood markers to estimate portal venous hypertension, which is linked to prognosis in patients with cirrhosis of the liver. To develop a measurement method of optimal data fusion quality deep learning problems utilizing both imaging data and clinical data. Our approach is based on modeling the fully connected layer (FCL) of a convolutional neural network (CNN) as a potential function, whose distribution takes the form of the classical Gibbs measure. The features of the FCL are then modeled as random variables governed by state functions, which are interpreted as the different data sources to be fused. The probability density of each source, relative to the probability density of the FCL, represents a quantitative measure of source-bias. To minimize this source-bias and optimize CNN performance, we implement a vector-growing encoding scheme called positional encoding, where low-dimensional clinical data are transcribed into a rich feature space that complements high-dimensional imaging features. We first provide a numerical validation of our approach based on simulated Gaussian processes. We then applied our approach to patient data, where we optimized the fusion of CT images with blood markers to predict portal venous hypertension in patients with cirrhosis of the liver. This patient study was based on a modified ResNet-152 model that incorporates both images and blood markers as input. These two data sources were processed in parallel, fused into a single FCL, and optimized based on our fusion quality framework. Numerical validation of our approach confirmed that the probability density function of a fused feature space converges to a source-specific probability density function when source data are improperly fused. Our numerical results demonstrate that this phenomenon can be quantified as a measure of fusion quality. On patient data, the fused model consisting of both imaging data and positionally encoded blood markers at the theoretically optimal fusion quality metric achieved an AUC of 0.74 and an accuracy of 0.71. This model was statistically better than the imaging-only model (AUC = 0.60; accuracy = 0.62), the blood marker-only model (AUC = 0.58; accuracy = 0.60), and a variety of purposely sub-optimized fusion models (AUC = 0.61-0.70; accuracy = 0.58-0.69). We introduced the concept of data fusion quality for multi-source deep learning problems involving both imaging and clinical data. We provided a theoretical framework, numerical validation, and real-world application in abdominal radiology. Our data suggests that CT imaging and hepatic blood markers provide complementary diagnostic information when appropriately fused.

Learning Spatial-Temporal Dynamics for Short-Term Passenger Flow Prediction in Urban Rail Transit

Accurate short-term passenger flow prediction in urban rail transit (URT) plays an important role in ensuring the stable operation of the URT systems. Because of the complex dynamic spatial-temporal dependencies and potential semantic correlations of the URT network, accurate and effective short-term passenger flow prediction is challenging. To solve these problems, a novel model called the dynamic spatial-temporal graph convolutional network (DSTGCN) was proposed. Firstly, spatial semantic graphs (SSGs) were established to encode the spatial dependencies and semantic correlations of the URT network. Meanwhile, the dynamic graph convolutional network (DGCN) with the spatial attention mechanism was used to learn the dynamic spatial correlations of the nodes in the SSGs. Then, the long short-term memory (LSTM) network was integrated into the DGCN to learn the dynamic changes of passenger flow and capture local temporal dependencies. Moreover, the temporal attention mechanism was introduced after LSTM to capture global dynamic temporal correlations by adjusting the weights of different sequence information. Finally, the full connection layers were used to output the prediction results. Several experiments were conducted on Nanning Metro Line 1 real datasets to evaluate the model. The experimental results showed that the DSTGCN can effectively capture the dynamic spatial-temporal dependencies and semantic associations of the passenger flow. Besides, the prediction performances of the DSTGCN were better than those of existing baseline models, and it can provide technical support for improving the intelligent planning and operation decisions of URT systems.

Spatial-temporal fusion network for maximum ground surface settlement prediction during tunnel excavation

The maximum ground surface settlement prediction is a complex problem as the settlement depends on plenty of intrinsic and extrinsic factors. To obtain the approximate range of the settlement, a hybrid prediction dataset including the geological and construction parameters is built using spatial and temporal series according to the sampling methods. The settlement prediction task is transformed into a multi-modal and multi-variate series prediction task. Hence, a spatial-temporal fusion network (STF-Network) is proposed. The spatial-temporal fusion mechanism is firstly designed to establish the spatial-temporal fusion map, which makes spatial and temporal series interact earlier. Then, the 3D residual unit structure is designed to capture the features of temporal series and spatial-temporal fusion map, and two fully-connected layers are established to capture the spatial structural information. Finally, the final output is merged by the three components. The experimental results for STF-Network demonstrate the superiority over state-of-the-art methods.

An Efficient Incremental Learning Algorithm for Sound Classification

This article proposes an efficient audio incremental learning method to reduce the computational complexity and catastrophic forgetting during the incremental addition of the audio data in Deep Neural Networks (DNN). The computational complexity is reduced by performing training of only fully-connected layers and catastrophic forgetting is reduced by sharing the knowledge from the old learned classes without using previously learned data. Our method has been evaluated extensively on UrbanSound8K, ESC-10, and TUT datasets where state-of-the-art accuracies have been achieved. Moreover, our method has been evaluated on Nvidia 1080-ti GPU, Nvidia TX-2, and Nvidia Xavier development boards to demonstrate the training time and energy consumption savings as compared to recent state-of-the-art methods.

EEG Conformer: Convolutional Transformer for EEG Decoding and Visualization.

Due to the limited perceptual field, convolutional neural networks (CNN) only extract local temporal features and may fail to capture long-term dependencies for EEG decoding. In this paper, we propose a compact Convolutional Transformer, named EEG Conformer, to encapsulate local and global features in a unified EEG classification framework. Specifically, the convolution module learns the low-level local features throughout the one-dimensional temporal and spatial convolution layers. The self-attention module is straightforwardly connected to extract the global correlation within the local temporal features. Subsequently, the simple classifier module based on fully-connected layers is followed to predict the categories for EEG signals. To enhance interpretability, we also devise a visualization strategy to project the class activation mapping onto the brain topography. Finally, we have conducted extensive experiments to evaluate our method on three public datasets in EEG-based motor imagery and emotion recognition paradigms. The experimental results show that our method achieves state-of-the-art performance and has great potential to be a new baseline for general EEG decoding. The code has been released in https://github.com/eeyhsong/EEG-Conformer.

Multi-Stream Acoustic Modelling Using Raw Real and Imaginary Parts of the Fourier Transform

In this paper, we investigate multi-stream acoustic modelling using the raw real and imaginary parts of the Fourier transform of speech signals. Using the raw magnitude spectrum, or features derived from it, as a proxy for the real and imaginary parts leads to irreversible information loss and suboptimal information fusion. We discuss and quantify the importance of such information in terms of speech quality and intelligibility. In the proposed framework, the real and imaginary parts are treated as two streams of information, pre-processed via separate convolutional networks, and then combined at an optimal level of abstraction, followed by further post-processing via recurrent and fully-connected layers. The optimal level of information fusion in various architectures, training dynamics in terms of cross-entropy loss, frame classification accuracy and WER as well as the shape and properties of the filters learned in the first convolutional layer of single- and multi-stream models are analysed. We investigated the effectiveness of the proposed systems in various tasks: TIMIT/NTIMIT (phone recognition), Aurora-4 (noise robustness), WSJ (read speech), AMI (meeting) and TORGO (dysarthric speech). Across all tasks we achieved competitive performance: in Aurora-4, down to 4.6% WER on average, in WSJ down to 4.6% and 6.2% WERs for Eval-92 and Eval-93, for Dev/Eval sets of the AMI-IHM down to 23.3%/23.8% WERs and in the AMI-SDM down to 43.7%/47.6% WERs have been achieved. In TORGO, for dysarthric and typical speech we achieved down to 31.7% and 10.2% WERs, respectively.

AlignVE: Visual Entailment Recognition Based on Alignment Relations

Visual entailment (VE) is to recognize whether the semantics of a hypothesis text can be inferred from the given premise image, which is one special task among recent emerged vision and language understanding tasks. Currently, most of the existing VE approaches are derived from the methods of visual question answering. They recognize visual entailment by quantifying the similarity between the hypothesis and premise in the content semantic features from multi modalities. Such approaches, however, ignore the VE's unique nature of relation inference between the premise and hypothesis. Therefore, in this paper, a new architecture called AlignVE is proposed to solve the visual entailment problem with a relation interaction method. It models the relation between the premise and hypothesis as an alignment matrix. Then it introduces a pooling operation to get feature vectors with a fixed size. Finally, it goes through the fully-connected layer and normalization layer to complete the classification. Experiments show that our alignment-based architecture reaches 72.45\% accuracy on SNLI-VE dataset, outperforming previous content-based models under the same settings.