Layers Of Convolutional Neural Network Research Articles

Photonic convolutional reservoir computing based on VCSEL with multiple optical injections

In this paper, a photonic convolutional time delay reservoir computing (C-TD-RC) system based on vertical cavity surface emitting laser (VCSEL) with multiple optical injections has been proposed. Here, the convolutional neural networks (CNNs) and the time delay reservoir computing (TD-RC) are combined in the system. Using TD-RC instead of the fully connected layer in the traditional CNNs can solve the problem of gradient vanishing and gradient explosion in CNNs. Two classification tasks including handwritten digits recognition and spoken digits classification are solved by the C-TD-RC system. In addition, in order to achieve better classification performance of RC, a new post-processing method combining ridge regression and winner take all strategy has been adopted. Numerical results demonstrate that the minimum Word Error Rate (WER) can reach 0.0135 for handwritten digits recognition and 0.0213 for spoken digits classification tasks by the C-TD-RC system. Furthermore, the effects of the virtual nodes, the injection paths and the number of convolution kernels applied in the first and second convolution layers on the C-TD-RC system are considered. The proposed C-TD-RC scheme is of great significance to the performance enhancement of RC in the future.

A Novel Gait Phase Recognition Method Based on DPF-LSTM-CNN Using Wearable Inertial Sensors.

Gait phase recognition is of great importance in the development of rehabilitation devices. The advantages of Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN) are combined (LSTM-CNN) in this paper, then a gait phase recognition method based on LSTM-CNN neural network model is proposed. In the LSTM-CNN model, the LSTM layer is used to process temporal sequences and the CNN layer is used to extract features A wireless sensor system including six inertial measurement units (IMU) fixed on the six positions of the lower limbs was developed. The difference in the gait recognition performance of the LSTM-CNN model was estimated using different groups of input data collected by seven different IMU grouping methods. Four phases in a complete gait were considered in this paper including the supporting phase with the right hill strike (SU-RHS), left leg swimming phase (SW-L), the supporting phase with the left hill strike (SU-LHS), and right leg swimming phase (SW-R). The results show that the best performance of the model in gait recognition appeared based on the group of data from all the six IMUs, with the recognition precision and macro-F1 unto 95.03% and 95.29%, respectively. At the same time, the best phase recognition accuracy for SU-RHS and SW-R appeared and up to 96.49% and 95.64%, respectively. The results also showed the best phase recognition accuracy (97.22%) for SW-L was acquired based on the group of data from four IMUs located at the left and right thighs and shanks. Comparably, the best phase recognition accuracy (97.86%) for SU-LHS was acquired based on the group of data from four IMUs located at left and right shanks and feet. Ulteriorly, a novel gait recognition method based on Data Pre-Filtering Long Short-Term Memory and Convolutional Neural Network (DPF-LSTM-CNN) model was proposed and its performance for gait phase recognition was evaluated. The experiment results showed that the recognition accuracy reached 97.21%, which was the highest compared to Deep convolutional neural networks (DCNN) and CNN-LSTM.

Radiograph Manufacturer and Model Identification Using Deep-RSI

Malware attacks of healthcare institutions are simultaneously becoming more common and more sophisticated. Artificial intelligence (AI) has resulted in the ability to rapidly alter or generate false images, advancing the ease of forgery of digital images. Digital image manipulation and substitution of radiographs are major threats to healthcare institutions because these altered images may affect patient care. Identifying the source (manufacturer, model) of radiology images is one method of validating the origin of radiology images in a healthcare system. In a previous study, researchers demonstrated that features from magnetic resonance imaging (MRI) could be used to trace and authenticate the source of the MRI images. We previously developed and tested the Deep learning for Radiograph Source Identification (Deep-RSI) approach for source identification of radiographs obtained of the upper extremities (hands, wrists, forearms, elbows, and shoulders). In this research, we present an empirical and quantitative investigation using deep learning to validate the source of digital radiographic images of the lower extremities (knees, legs, ankles, and feet). A convolutional neural network (CNN) is employed to extract features, which are then followed by three fully connected layers (FCNN). To ensure that our proposed method is a content-free approach, we added a new layer before the CNN to extract the initial content-free pixels and train the features using the CNN and FCNN layers. This proposed approach was used to identify the source of each digital image of a lower extremity. Adult patients of both sexes who had radiographs of the lower extremities at Mayo Clinic between 01/01/2010 and 12/31/2021 were evaluated. The data was randomly split by patient into training/validation and test datasets. There were 9 radiographic machine models and 6 manufacturers. Deep-RSI had an accuracy of 99.00% (AUC= 0.99) and 97.00% (AUC=0.94) for detecting the manufacturer and model of the radiographic machine for radiographs of the feet respectively, confirming that forensic evaluation of radiographs can be performed. This is the first medical forensics examination of this type to identify and confirm the source origins for radiographs of the lower extremities. This technique may be helpful to detect radiology malware attacks and scientific fraud.
 

Understanding the class separation process of convolutional neural networks

Extensive research shows that deep neural networks (DNNs) capture better feature representations than the previous handcrafted feature engineering, which leads to a significant performance improvement. However, it raises the question of how does the neural network spontaneously learn the intermediate representation, during the training process, to correctly separate the test data set into different categories? In this paper, based on the existing research work, we continue to take a small step towards understanding the dynamics of convolutional neural networks (CNN). Specifically, we model the test data set as a relationship graph, and each intermediate layer representation learned by CNN will transform the relationship graph correspondingly. Through these continuous transformations, we investigate the class separation process of CNN from two perspectives: 1) show the evolution of the degree of correlation between samples through the histogram of similarity between vertices; 2) by visualizing and quantifying the degree of separation (i.e., modularity) of the important relationship subgraph, we can show the contribution of each convolution layer of CNN to class separation. In the preliminary experiment, it can be found that: 1) each convolution layer of CNN gradually reduced the similarity of most edges between the samples to a very low level, that is, the dense relationship graph gradually evolved into a sparse graph; 2) the remaining important relationship subgraph consist of edges with high similarity shows evident modularity; 3) both visualization and quantification results show that the modularity of intra-class subgraph increases when the layers go deeper. Moreover, the degradation and plateau in the modularity curve reveal the existence of redundant layers. In this paper, we reveal some more in-depth dynamics of CNN, and introduced another modularity, which is widely used as the theoretical guidance tool of layer pruning. It can save more model parameters without losing the accuracy of classification.

Efficient Channel Estimation in Multiple Input Multiple Output Communication Systems using Hybrid Intelligence‐based Deep Learning Model using Horse Dingo Optimizer

AbstractThe estimation of the channel in multiple input multiple output (MIMO) systems over wireless communication has faced many challenges, such as high bit rate transmission with less time complexity, low error rate, and high convergence rate. And also, there is some potential for enhancing the system performance regarding channel efficiency diversity. MIMO system uses a huge count of antennas on the receiver and the transmitter side. This increases the cost requirements and power consumption. For achieving coherent detection in the MIMO system, optimization such as water filling and beam forming are required to find the proper channel. Hence, multiple transceivers used in both the transmitter side and receiver side create a channel estimation issue that is complicated when analyzed to a single input, single output system. Therefore, several solutions have been recommended to decrease the computational cost and power consumption in channel estimation of MIMO systems. This work intends to propose a novel deep structured architecture to handle the channel estimation problem in a MIMO system. The major objective of the developed method is to validate the channel coefficients of MIMO at the transmitter according to the obtained SNR feedback data packet from an acceptor with the help of the hybrid intelligence‐based deep learning model (HI‐DLM). For both scenarios, the HI‐DLM is introduced, in which the convolution neural network (CNN) is integrated with other deep‐structured architecture. Especially, the fully connected layers of CNN are restored by the long short‐term memory. In the HI‐DLM‐aided channel estimation, the hyperparameter optimization is accomplished by the novel dingo‐horse herd optimization for reducing the mean square error and bit error rate to acquire the estimated channel. The corresponding algorithm is validated on the communication of the MIMO system by experimentations. Thus, it is revealed that the offered algorithm provides better convergence and less computational cost.

Spatial attention-based dual stream transformer for concrete defect identification

This paper addresses the problem of automatic crack identification within building structures. Cracks can cause structural damage to a building, necessitating prompt repairs to preserve lives. Detecting structural defects is a difficult process, especially in tall towers. Machine learning and deep learning approaches automate damage identification and crack detection. Machine learning-based methods train models with manually produced features that may contain errors. Deep convolutional neural networks (DCNN), on the other hand, enhance performance by extracting high-dimensional features. However, DCNN algorithms for crack identification lack a global focus on features that can compromise performance. This study inserts a dual-stream transformer module into a single 2D convolution neural network (CNN) layer. The dual-stream block contains a semantic and pixels path. Precise views of local and global features at the pixel level can be obtained through two paths. This increases self-attention information distributed parallel along the semantic and pixel paths to train the model. Further, the DSTNet (Dual Stream Transformer Network) superiority is validated on the three datasets and achieved notable performance.

Application of deep learning model in computer data mining intrusion detection

Abstract In order to improve the autonomous defense ability and correct detection rate of network intrusion detection system, in this essay, an intrusion detection model combining convolutional neural network and Inception network structure is proposed, and the attention mechanism is set in the model, and DropBlock layer is added. In this model, convolutional neural network layer is used to fully extract data features. The attention mechanism is used to calculate the weight of each feature to distinguish the importance of the feature. The DropBlock layer is used to improve the generalization ability of the model, improve the accuracy of intrusion detection and reduce the complexity of the model. Experiments on data sets show that this model has higher accuracy and stronger generalization ability.

Uncovering Patterns in Dairy Cow Behaviour: A Deep Learning Approach with Tri-Axial Accelerometer Data.

The accurate detection of behavioural changes represents a promising method of detecting the early onset of disease in dairy cows. This study assessed the performance of deep learning (DL) in classifying dairy cows' behaviour from accelerometry data acquired by single sensors on the cows' left flanks and compared the results with those obtained through classical machine learning (ML) from the same raw data. Twelve cows with a tri-axial accelerometer were observed for 136 ± 29 min each to detect five main behaviours: standing still, moving, feeding, ruminating and resting. For each 8 s time interval, 15 metrics were calculated, obtaining a dataset of 211,720 observation units and 15 columns. The entire dataset was randomly split into training (80%) and testing (20%) datasets. The DL accuracy, precision and sensitivity/recall were calculated and compared with the performance of classical ML models. The best predictive model was an 8-layer convolutional neural network (CNN) with an overall accuracy and F1 score equal to 0.96. The precision, sensitivity/recall and F1 score of single behaviours had the following ranges: 0.93-0.99. The CNN outperformed all the classical ML algorithms. The CNN used to monitor the cows' conditions showed an overall high performance in successfully predicting multiple behaviours using a single accelerometer.

Federated Machine Learning for Skin Lesion Diagnosis: An Asynchronous and Weighted Approach.

The accurate and timely diagnosis of skin cancer is crucial as it can be a life-threatening disease. However, the implementation of traditional machine learning algorithms in healthcare settings is faced with significant challenges due to data privacy concerns. To tackle this issue, we propose a privacy-aware machine learning approach for skin cancer detection that utilizes asynchronous federated learning and convolutional neural networks (CNNs). Our method optimizes communication rounds by dividing the CNN layers into shallow and deep layers, with the shallow layers being updated more frequently. In order to enhance the accuracy and convergence of the central model, we introduce a temporally weighted aggregation approach that takes advantage of previously trained local models. Our approach is evaluated on a skin cancer dataset, and the results show that it outperforms existing methods in terms of accuracy and communication cost. Specifically, our approach achieves a higher accuracy rate while requiring fewer communication rounds. The results suggest that our proposed method can be a promising solution for improving skin cancer diagnosis while also addressing data privacy concerns in healthcare settings.

Fault Detection for Motor Drive Control System of Industrial Robots Using CNN-LSTM-based Observers

The complex working conditions and nonlinear characteristics of the motor drive control system of industrial robots make it difficult to detect faults. In this paper, a deep learning-based observer, which combines the convolutional neural network (CNN) and the long short-term memory network (LSTM), is employed to approximate the nonlinear driving control system. CNN layers are introduced to extract dynamic features of the data, whereas LSTM layers perform time-sequential prediction of the target system. In terms of application, normal samples are fed into the observer to build an offline prediction model for the target system. The trained CNN-LSTM-based observer is then deployed along with the target system to estimate the system outputs. Online fault detection can be realized by analyzing the residuals. Finally, an application of the proposed fault detection method to a brushless DC motor drive system is given to verify the effectiveness of the proposed scheme. Simulation results indicate the impressive fault detection capability of the presented method for driving control systems of industrial robots.

Generalizable end-to-end deep learning frameworks for real-time attitude estimation using 6DoF inertial measurement units

This paper presents a novel end-to-end deep learning framework for real-time inertial attitude estimation using 6DoF IMU measurements. Inertial Measurement Units are widely used in various applications, including engineering and medical sciences. However, traditional filters used for attitude estimation suffer from poor generalization over different motion patterns and environmental disturbances. To address this problem, we propose two deep learning models that incorporate accelerometer and gyroscope readings as inputs. These models are designed to be generalized to different motion patterns, sampling rates, and environmental disturbances. Our models consist of convolutional neural network layers combined with Bi-Directional Long–Short Term Memory followed by a Fully Forward Neural Network to estimate the quaternion. We evaluate the proposed method on seven publicly available datasets, totaling more than 120 h and 200 kilometers of IMU measurements. Our results show that the proposed method outperforms state-of-the-art methods in terms of accuracy and robustness. Additionally, our framework demonstrates superior generalization over various motion characteristics and sensor sampling rates. Overall, this paper provides a comprehensive and reliable solution for real-time inertial attitude estimation using 6DoF IMUs, which has significant implications for a wide range of applications.

Vision-based fruit recognition via multi-scale attention CNN

Fruit quality assessment, grading and sorting are of vital importance to fruit processing, and all these involve fruit recognition. Vision-based fruit recognition can recognize fruit automatically and further support more applications such as fruit-picking robots, self-checkout service, and dietary guidance. Recent works resort to Convolutional Neural Networks (CNNs) for vision-based fruit recognition because of their powerful expressive capacity. However, most works simply utilize existing CNNs and ignore characteristics of fruit images, resulting in sub-optimal recognition performance. To solve this problem, we adopt a Multi-Scale Attention Network (MSANet), which explores attention from different layers of CNNs and aggregates various visual attentional features from different levels into final comprehensive representations. Extensive evaluations on four fruit benchmark datasets demonstrate that the method achieves state-of-the-art recognition performance. In addition, we report various experimental results from different deep networks as baselines. Codes and models will be released upon publication. Models and code are available at http://123.57.42.89/codes/msanet.html.

A CNN-BiGRU-AM neural network for AI applications in shale oil production prediction

In the coming decades, the demand for shale oil will likely surge because of predicted increases in the global population and productivity. Efficiently predicting shale oil production is therefore critical for understanding both the reliability of applied unconventional resources and crude supply. A novel method was proposed to extract the spatial–temporal properties of production data for shale oil prediction on the basis of a convolutional neural network (CNN) and bidirectional gated recurrent unit (BiGRU) with an attention mechanism (AM). The strengths and advantages of the CNN layer were selected as the input variables affecting shale oil production. The BiGRU network provides new and more detailed temporal information or irregular trends in temporal series components. AM also contributes to understanding the impact of intrinsic information to guarantee learning accuracy. CNN-BiGRU-AM performs the desired behavior of evaluation indicators in contrast to conventional machine learning and deep learning for forecasting shale oil production. The profound impact of this work lies in delivering state-of-the-art research aids that highlight the large but uneven impact of shale oil production prediction.

Sparse adversarial image generation using dictionary learning

Adversarial examples are used to evaluate the robustness of convolutional neural networks (CNNs) to input perturbations. Researchers have proposed different types of adversarial examples that attack CNNs to fool them. These attacks pose a serious threat to applications that use deep neural networks. Existing methods for adversarial image generation struggle in maintaining a balance between attack success rate and imperceptibility (measured in terms of ℓ2-norm) of the generated adversarial examples. Recent sparse methods for this problem focus on limiting the number of pixels in an image but do not cater to the overall imperceptibility of the adversarial images. To address these problems, we introduce adversarial attacks based on K-singular value decomposition sparse dictionary learning. The dictionary is learned using feature maps of the targeted images from the first layer of CNN. The proposed method is evaluated in terms of attack success rate and ℓ2-norm. The extensive experimentation shows our attack achieves a high success rate while maintaining a low imperceptibility score compared to state-of-the-art methods.

Detection of anomaly in surveillance videos using quantum convolutional neural networks

Anomalous behavior identification is the process of detecting behavior that differs from its normal. These incidents will vary from violence to war, road crashes to kidnapping, and so on in a surveillance model. Video anomaly detection from video surveillance is a difficult research activity due to the frequency of anomalous cases. Since certain devices need manual evaluation for the detection of violent or criminal situations at the same time video monitoring of security cameras is also a challenging task and is unreliable. When the data or model dimension is sufficiently large, convolutional neural networks have the limitation of learning inefficiently. Quantum Convolutional Neural Network (QCNN) is the name given to a technology that combines CNN and quantum computing. Quantum computation and CNN are combined to create a more efficient and outperforming solution for solving complicated machine-learning problems. To analyze the anomalies in a sequence of video frames, two models are proposed in this research. In this research 07 layers of Javeria deep convolutional neural network (DCNN) are proposed on the selected hyperparameters named J. DCNN which is also different from the existing models to analyze the abnormal behavior in a video segment. Furthermore, for a comprehensive analysis of the abnormal video frames a model is proposed which is the combination of Javeria quantum and convolutional neural networks (J. QCNN). In this model 04-qubit quantum neural network is used with five layers and an optimal loss rate named J. QCNN. The proposed J. QCNN model is different from the existing deep learning architectures. The proposed models are trained from the scratch for the detection of anomalous from top challenging publicly available video surveillance datasets such as UNI-Crime and UCF Crime. The proposed J. QCNN model classifies the number of violent robberies such as armed thefts containing handguns or knives, and robberies displaying varying levels of viciousness with 0.99 accuracy while J. DCNN model gives 0.97 accuracy. The obtained results are superior in comparison with recent existing cutting-edge published work for real-time anomaly detection in video CCTV.

Deep Cellular Automata-Based Feature Extraction for Classification of the Breast Cancer Image

Feature extraction is an important step in classification. It directly results in an improvement of classification performance. Recent successes of convolutional neural networks (CNN) have revolutionized image classification in computer vision. The outstanding convolution layer of CNN performs feature extraction to obtain promising features from images. However, it faces the overfitting problem and computational complexity due to the complicated structure of the convolution layer and deep computation. Therefore, this research problem is challenging. This paper proposes a novel deep feature extraction method based on a cellular automata (CA) model for image classification. It is established on the basis of a deep learning approach and multilayer CA with two main processes. Firstly, in the feature extraction process, multilayer CA with rules are built as the deep feature extraction model based on CA theory. The model aims at extracting multilayer features, called feature matrices, from images. Then, these feature matrices are used to generate score matrices for the deep feature model trained by the CA rules. Secondly, in the decision process, the score matrices are flattened and fed into the fully connected layer of an artificial neural network (ANN) for classification. For performance evaluation, the proposed method is empirically tested on BreaKHis, a popular public breast cancer image dataset used in several promising and popular studies, in comparison with the state-of-the-art methods. The experimental results show that the proposed method achieves the better results up to 7.95% improvement on average when compared with the state-of-the-art methods.

Crack recognition on concrete structures based on machine crafted and hand crafted features

Concrete structures play a critical role in infrastructure development, and their safety is of utmost importance. Cracks in concrete structures can be a sign of deterioration, which may affect the overall safety of the building. Regular inspections and monitoring of surface cracks are essential to ensure the structural integrity of the building. However, human inspection can be time-consuming and subjective, leading to inconsistent findings. Hence, computer vision based crack recognition is necessary. In this work, we propose three custom designed deep networks and a multi resolution framework, named, Quaternionic wavelet transform (QWT) for crack recognition. The efficiency of deep network (machine crafted features) and QWT (hand crafted features) is studied in detail on three datasets namely SDNET2018, METU (Middle East Technical University) concrete crack image dataset and Historical-crack18-19 dataset. Also, the effect of wavelet scale on crack recognition and the implication of the fully connected layers in deep convolutional neural networks are studied in this research work. Finally, we infer that QWT features achieve a maximum accuracy of 98.44%, 99.80% and 94.67% respectively on the above said datasets, and their performance is comparable to that of machine-crafted (deep) features and with the other state of art methods.

Multimodal Deep Learning for Rice Yield Prediction Using UAV-Based Multispectral Imagery and Weather Data

Precise yield predictions are useful for implementing precision agriculture technologies and making better decisions in crop management. Convolutional neural networks (CNNs) have recently been used to predict crop yields in unmanned aerial vehicle (UAV)-based remote sensing studies, but weather data have not been considered in modeling. The aim of this study was to explore the potential of multimodal deep learning on rice yield prediction accuracy using UAV multispectral images at the heading stage, along with weather data. The effects of the CNN architectures, layer depths, and weather data integration methods on the prediction accuracy were evaluated. Overall, the multimodal deep learning model integrating UAV-based multispectral imagery and weather data had the potential to develop more precise rice yield predictions. The best models were those trained with weekly weather data. A simple CNN feature extractor for UAV-based multispectral image input data might be sufficient to predict crop yields accurately. However, the spatial patterns of the predicted yield maps differed from model to model, although the prediction accuracy was almost the same. The results indicated that not only the prediction accuracies, but also the robustness of within-field yield predictions, should be assessed in further studies.

Hybrid deep neural network with dimension attention for state-of-health estimation of Lithium-ion Batteries

Lithium-ion batteries (LIBs) are widely used and became the main energy storage medium for many devices. Accurate estimation of LIBs state-of-health (SOH) is crucial for safe and reliable operation of devices. This study designs an end-to-end multi-battery shared hybrid neural network (NN) prognostic framework that combines a convolutional neural network (CNN), a multi-layer variant long-short-term memory (VLSTM) NN and a dimensional attention mechanism (CNN-VLSTM-DA) to SOH estimation for LIBs. First, feature extraction and selection on the raw input data are performed by using a CNN. Second, a suitable VLSTM is designed. The network adds a “peephole connection” to the forget gate and output gate, respectively, which enhances the network’s ability to distinguish subtle features between input sequences. Besides, the forget gate and the input gate are coupled, so that, together, they determine the information that needs to be forgotten and the new data that needs to be added. Then, the output data of the CNN layer are fed into a multi-layer VLSTM NN to further capture the temporal correlation of these data. Finally, the attention mechanism is applied to the output of the VLSTM, to assign different weights to the features of each dimension and to give the prediction results. Several experiments are carried out on three datasets from NASA, CALCE and Oxford. These include full charge/discharge data, charge/discharge data in different SOC ranges, and non-fixed discharge current data. The results verify the effectiveness of the proposed method.

Pre-Trained CNN Architecture Analysis for Transformer-Based Indonesian Image Caption Generation Model

Classification and object recognition in image processing has significantly improved computer vision tasks. The method is often used for visual problems, especially in picture classification utilizing the Convolutional Neural Network (CNN). In the popular state-of-the-art (SOTA) task of generating a caption on an image, the implementation is often used for feature extraction of an image as an encoder. Instead of performing direct classification, these extracted features are sent from the encoder to the decoder section to generate the sequence. So, some CNN layers related to the classification task are not required. This study aims to determine which CNN pre-trained architecture or model performs best in extracting image features using a state-of-the-art Transformer model as its decoder. Unlike the original Transformer’s architecture, we implemented a vector-to-sequence way instead of sequence-to-sequence for the model. Indonesian Flickr8k and Flick30k datasets were used in this research. Evaluations were carried out using several pre-trained architectures, including ResNet18, ResNet34, ResNet50, ResNet101, VGG16, Efficientnet_b0, Efficientnet_b1, and Googlenet. The qualitative model inference results and quantitative evaluation scores were analyzed in this study. The test results show that the ResNet50 architecture can produce stable sequence generation with the highest accuracy value. With some experimentation, finetuning the encoder can significantly increase the model evaluation score. As for future work, further exploration with larger datasets like Flickr30k, MS COCO 14, MS COCO 17, and other image captioning datasets in Indonesian also implementing a new Transformers-based method can be used to get a better Indonesian automatic image captioning model.Â