Dual-stream Convolutional Neural Network Research Articles

Infrared image fault diagnosis based on dual-stream attention convolution network

Abstract The development of infrared image detection technology has improved the real-time performance and safety of fault diagnosis. To address the problem of insufficient features caused by the low resolution of infrared images, a fault diagnosis model for induction motor infrared images based on Dual-Stream Attention Convolutional (DSAC) is proposed. Firstly, this model employs a dual-stream convolutional neural network to extract spatial features in the X and Y directions of the infrared images separately. This dual-stream structure allows the network to simultaneously learn and utilize information from both X and Y directions, enabling a more comprehensive capture of temperature distribution and variation trends in the infrared images. Then, a convolutional attention mechanism is introduced to assign weights to the obtained features. The attention maps generated by the convolutional attention layer enhance key features while suppressing unimportant information, thereby enhancing the focusing performance of the DSAC model on critical features. Finally, the weighted dual-stream features are fused to achieve the goal of fault diagnosis for induction motor infrared images. The DSAC model is validated using the induction motor infrared image dataset from
Babol Noshirvani University of Technology, demonstrating excellent diagnostic accuracy and speed. Moreover, it alleviates the problem of sample imbalance to some extent in small-sample datasets, providing a feasible solution for fault diagnosis of induction motor infrared images.

Air Traffic Controller Fatigue Detection by Applying a Dual-Stream Convolutional Neural Network to the Fusion of Radiotelephony and Facial Data

The role of air traffic controllers is to direct and manage highly dynamic flights. Their work requires both efficiency and accuracy. Previous studies have shown that fatigue in air traffic controllers can impair their work ability and even threaten flight safety, which makes it necessary to carry out research into how to optimally detect fatigue in controllers. Compared with single-modality fatigue detection methods, multi-modal detection methods can fully utilize the complementarity between diverse types of information. Considering the negative impacts of contact-based fatigue detection methods on the work performed by air traffic controllers, this paper proposes a novel AF dual-stream convolutional neural network (CNN) architecture that simultaneously extracts controller radio telephony fatigue features and facial fatigue features and performs two-class feature-fusion discrimination. This study designed two independent convolutional processes for facial images and radio telephony data and performed feature-level fusion of the extracted radio telephony and facial image features in the fully connected layer, with the fused features transmitted to the classifier for fatigue state discrimination. The experimental results show that the detection accuracy of radio telephony features under a single modality was 62.88%, the detection accuracy of facial images was 96.0%, and the detection accuracy of the proposed AF dual-stream CNN network architecture reached 98.03% and also converged faster. In summary, a dual-stream network architecture based on facial data and radio telephony data is proposed for fatigue detection that is faster and more accurate than the other methods assessed in this study.

Deep learning model to detect deceptive generative adversarial network generated images using multimedia forensic

Computer-generated imagery has been made more lifelike and misleading by new Generative Adversarial Network (GAN) models, such as StyleGAN, posing severe risks to people's safety, social order, and privacy. Deceptive content creation, including Deepfakes, image tampering, and information hiding, can be facilitated through the misuse of GANs. To tackle these challenges, a detection model is proposed in this research, employing a spatial-frequency joint dual-stream convolutional neural network. Learnable frequency-domain filtering kernels and frequency-domain networks are leveraged to thoroughly learn and extract frequency-domain features, considering the discernible artifacts left by GAN images in the frequency spectrum due to the upsampling process during production. Lastly, the two sets of traits are combined to identify GAN-created faces. The proposed model outperforms state-of-the-art methods, as evidenced by experimental findings on various datasets, both in terms of detection accuracy on high-quality created datasets and generalization across datasets.

WaterFormer: A coupled transformer and CNN network for waterbody detection in optical remotely-sensed imagery

As one of the most significant components of the ecosystem, waterbody needs to be highly monitored at different spatial and temporal scales. Nevertheless, waterbody variations in shape, size, and reflectivity, complicated and varied types of land covers, and environmental scene diversity, present colossal challenges in achieving accurate waterbody detection (WD). In this paper, we propose a novel network coupled with the Transformer and convolutional neural network (CNN), termed WaterFormer, to automatically, efficiently, and accurately delineate waterbodies from optical high-resolution remotely sensed (HR-RS) images. This network mainly includes a dual-stream CNN, a cross-level Vision Transformer, a light-weight attention module, and a sub-pixel up-sampling module. First, the dual-stream network abstracts waterbody features at multi-views and different levels. Then, to exploit the long-range dependencies between low-level spatial information and high-order semantic features, the cross-level Vision Transformer is embedded into the dual-stream, aiming at improving WD accuracy. Afterwards, the light-weight attention module is adopted to provide semantically strong feature abstractions by enhancing discrimination neurons, and the sub-pixel up-sampling module is employed to further generate high-resolution and high-quality class-specific representations. Quantitative and qualitative evaluations demonstrated that the WaterFormer provided a promising means for detecting waterbody areas in satellite images under complex scene conditions. Moreover, comparative analyses with the state-of-the-art (SOTA) alternatives, e.g., MSFENet, MSAFNet, and BiSeNet, also verified the generalization and superiority of the WaterFormer in WD tasks. The assessment results exhibited that the WaterFormer gained an average accuracy of 97.24%, average precision of 94.59%, average recall of 91.95%, average F1-score of 93.24%, and average Kappa index of 0.9133, respectively. Additionally, we presented an open-access HR satellite imagery waterbody dataset, a mesoscale dataset with high-quality and high-precision waterbody annotation to facilitate future research in this field. The dataset has been released at https://github.com/NJdeuK/WD_Dataset.

DS-CNN: Dual-Stream Convolutional Neural Networks-Based Heart Sound Classification for Wearable Devices

Cardiovascular diseases (CVDs) is considered a serious public health problem due to the uncertainty of its onset. Consuming wearable devices have increasing popularities for healthcare monitoring, and many of them are capable of continuous monitoring and early detection of CVDs. This paper proposes a framework for heart sound detection that can be considered for deployment on smart wearable devices to screen CVDs conveniently. A dual-stream convolutional neural network (DS-CNN) is developed to detect abnormal ones from short-term heart sound recordings. Preprocessing module is first employed for noise filtering and amplitude normalization. Then short-time Fourier transform and higher-order spectral are introduced for feature extraction, whose products are subsequently fed into the DS-CNN for screening abnormal heart sound signals. Two open accessible datasets are employed for performance evaluation. The results well demonstrate the classification accuracy of the proposed DS-CNN, and also indicate its advantages for adapting to heart sound recordings collected by different equipments.

Face spoofing detection based on multi-scale color inversion dual-stream convolutional neural network

Face spoofing detection based on multi-scale color inversion dual-stream convolutional neural network

Low-light aware framework for human activity recognition via optimized dual stream parallel network

Human Activity Recognition (HAR) plays a crucial role in communication and the Internet of Things (IoT), by enabling vision sensors to understand and respond to human behavior more intelligently and efficiently. Existing deep learning models are complex to deal with the low illumination, diverse viewpoints, and cluttered backgrounds, which require substantial computing resources and are not appropriate for edge devices. Furthermore, without an effective video analysis technique it processes entire frames, resulting inadequate performance. To address these key challenges, a cloud-assisted IoT computing framework is proposed for HAR in uncertain low-lighting environments, which is mainly composed of two tiers: edge and cloud computing. Initially, a lightweight Convolutional Neural Network (CNN) model is developed which is responsible to enhance the low-light frames, followed by the human detection algorithm to process the selective frames, thus enabling efficient resource utilization. Next, these refined frames are then transmitted to the cloud for accurate HAR, where dual stream CNN and transformer fusion network extract both short-and long-range spatiotemporal discriminative features followed by proposed Optimized Parallel Sequential Temporal Network (OPSTN) with squeeze and excitation attention to efficiently learn HAR in complex scenarios. Finally, extensive experiments are conducted over three challenging HAR datasets to deeply examine the proposed framework from various perspectives such as complex activity recognition, lowlighting, etc., where the results are outperformed compared with the state-of-art methods.

Stain Defect Classification by Gabor Filter and Dual-Stream Convolutional Neural Network

A stain defect is difficult to detect with the human eye because of its characteristic of having a very minimal difference in brightness with the local area of the surface. Recently, with the development of Deep learning, the Convolutional Neural Network (CNN) based stain defect classification method has been proposed. This paper proposes a Dual-stream CNN for stain defect classification using a Gabor filter image. Using Dual-stream structure CNN, Gabor filter images and Gray image (Original) preserve their respective features. The experiment based on the Magnetic Tile (MT) stain data set and the Compact Camera Module (CCM) stain dataset confirms that the proposed method has an improved performance based on the precision, recall, and F1-score in comparison to the Single-stream extraction-based method. Gabor filter images have an advantage in image texture analysis and can be used as an input to the CNNs. The Dual-stream structure better extracts the features needed for classification.

A Two-stream Convolutional Neural Network-based Pornography Recognition Method

The main approach taken to identify pornographic video content is achieved by performing pornography detection on the video content. By extracting features from video key frames and using some common neural network models to recognize the extracted key frame images, a certain accuracy rate can be obtained. However, another key information of video recognition, action information, is ignored, which leads to misclassification of some indistinguishable videos such as sumo wrestling and boxing. A dual-stream convolutional neural network-based pornographic video recognition method is proposed to address this problem. The experimental results show that the dual-stream convolutional neural network effectively improves the recognition rate of indistinguishable pornographic videos.

Solar Power Prediction Using Dual Stream CNN-LSTM Architecture.

The integration of solar energy with a power system brings great economic and environmental benefits. However, the high penetration of solar power is challenging due to the operation and planning of the existing power system owing to the intermittence and randomicity of solar power generation. Achieving accurate predictions for power generation is important to provide high-quality electric energy for end-users. Therefore, in this paper, we introduce a deep learning-based dual-stream convolutional neural network (CNN) and long short-term nemory (LSTM) network followed by a self-attention mechanism network (DSCLANet). Here, CNN is used to learn spatial patterns and LSTM is incorporated for temporal feature extraction. The output spatial and temporal feature vectors are then fused, followed by a self-attention mechanism to select optimal features for further processing. Finally, fully connected layers are incorporated for short-term solar power prediction. The performance of DSCLANet is evaluated on DKASC Alice Spring solar datasets, and it reduces the error rate up to 0.0136 MSE, 0.0304 MAE, and 0.0458 RMSE compared to recent state-of-the-art methods.

Human Motion Pattern Recognition and Feature Extraction: An Approach Using Multi-Information Fusion.

An exoskeleton is a kind of intelligent wearable device with bioelectronics and biomechanics. To realize its effective assistance to the human body, an exoskeleton needs to recognize the real time movement pattern of the human body in order to make corresponding movements at the right time. However, it is of great difficulty for an exoskeleton to fully identify human motion patterns, which are mainly manifested as incomplete acquisition of lower limb motion information, poor feature extraction ability, and complicated steps. Aiming at the above consideration, the motion mechanisms of human lower limbs have been analyzed in this paper, and a set of wearable bioelectronics devices are introduced based on an electromyography (EMG) sensor and inertial measurement unit (IMU), which help to obtain biological and kinematic information of the lower limb. Then, the Dual Stream convolutional neural network (CNN)-ReliefF was presented to extract features from the fusion sensors’ data, which were input into four different classifiers to obtain the recognition accuracy of human motion patterns. Compared with a single sensor (EMG or IMU) and single stream CNN or manual designed feature extraction methods, the feature extraction based on Dual Stream CNN-ReliefF shows better performance in terms of visualization performance and recognition accuracy. This method was used to extract features from EMG and IMU data of six subjects and input these features into four different classifiers. The motion pattern recognition accuracy of each subject under the four classifiers is above 97%, with the highest average recognition accuracy reaching 99.12%. It can be concluded that the wearable bioelectronics device and Dual Stream CNN-ReliefF feature extraction method proposed in this paper enhanced an exoskeleton’s ability to capture human movement patterns, thus providing optimal assistance to the human body at the appropriate time. Therefore, it can provide a novel approach for improving the human-machine interaction of exoskeletons.

ESAR, An Expert Shoplifting Activity Recognition System

Abstract Shoplifting is a troubling and pervasive aspect of consumers, causing great losses to retailers. It is the theft of goods from the stores/shops, usually by hiding the store item either in the pocket or in carrier bag and leaving without any payment. Revenue loss is the most direct financial effect of shoplifting. Therefore, this article introduces an Expert Shoplifting Activity Recognition (ESAR) system to reduce shoplifting incidents in stores/shops. The system being proposed seamlessly examines each frame in video footage and alerts security personnel when shoplifting occurs. It uses dual-stream convolutional neural network to extract appearance and salient motion features in the video sequences. Here, optical flow and gradient components are used to extract salient motion features related to shoplifting movement in the video sequence. Long Short Term Memory (LSTM) based deep learner is modeled to learn the extracted features in the time domain for distinguishing person actions (i.e., normal and shoplifting). Analyzing the model behavior for diverse modeling environments is an added contribution of this paper. A synthesized shoplifting dataset is used here for experimentations. The experimental outcomes show that the proposed approach attains better consequences up to 90.26% detection accuracy compared to the other prevalent approaches.

Cross-Database Micro-Expression Recognition Based on a Dual-Stream Convolutional Neural Network

Cross-database micro-expression recognition (CDMER) under semi supervised conditions is a difficult task, where the target (testing) and source (training) samples come from different micro-expression (ME) databases, resulting in the inconsistency of the feature distributions between each other, and hence affecting the performance of many existing MER methods. To address this problem, we propose a dual-stream convolutional neural network (DSCNN) for dealing with CDMER tasks. In the DSCNN, two stream branches are designed to study temporal and facial region cues in ME samples with the goal of recognizing MEs. In addition, in the training process, the domain discrepancy loss is used to enforce the target and source samples to have similar feature distributions in some layers of the DSCNN. Extensive CDMER experiments are conducted to evaluate the DSCNN. The results show that our proposed DSCNN model achieves a higher recognition accuracy when compared with some representative CDMER methods.

Dual-Stream Convolutional Neural Network With Residual Information Enhancement for Pansharpening

Deep-learning-based pansharpening methods have achieved remarkable results due to their powerful feature representation ability. However, the existing deep-learning-based pansharpening methods not only lack information exchange and sharing between features of different resolutions but also cannot effectively use the residual information at different levels. These disadvantages may lead to the loss of spatial information and spectral information in the pansharpened image. To address the above problems, we propose a novel dual-stream convolutional neural network with residual information enhancement (DSCNN-RIE) for pansharpening. The proposed network is mainly composed of a set of dual-stream information complementation blocks (DSICBs), which can extract various spatial details at two different resolutions using convolutional filters of various sizes simultaneously, and can transfer complementary information effectively between two different resolutions. Furthermore, to improve the learning ability of the network and enhance the feature extraction, an RIE strategy is presented to stack different levels of residuals into the outputs of cascaded DSICBs. The final pansharpened image is obtained by integrating the extracted features using the shallow feature information of the source images. Experimental results on three datasets demonstrate that DSCNN-RIE outperforms ten other state-of-the-art pansharpening methods in both subjective and objective image-quality evaluations.

Gesture recognition using dual-stream CNN based on fusion of sEMG energy kernel phase portrait and IMU amplitude image

Since the surface electromyography signal (sEMG) is a weak and unstable signal, human action recognition based on sEMG time domain or frequency domain feature extraction has poor stability and low recognition rate. Based on modeling analysis of sEMG, the dual-stream convolutional neural network (CNN) is built to perform feature fusion of sEMG energy kernel phase portrait and inertial measurement unit(IMU) data for gesture recognition in this paper. Firstly, methods of matrix counting method and time window amplitude are utilized to represent the sEMG and IMU data as images. Secondly, the moving average method is presented to filter the images. Thirdly, the dual-stream CNN is used to perform feature extraction, fusion and classification of sEMG and IMU images respectively. Finally, the experiments yield an average recognition accuracy of 95.78% for 6 gestures performed by 5 subjects, which demonstrates that the proposed method has better performance for human gestures recognition.

Winter road surface condition classification using convolutional neural network (CNN): visible light and thermal image fusion

During winter, road conditions play a crucial role in traffic flow efficiency and road safety. Icy, snowy, slushy, and wet road conditions reduce tire friction and affect vehicle stability which could lead to dangerous crashes. To keep traffic operations safe, cities spend a significant budget on winter maintenance operations such as snow plowing and spreading of salt and sand. This paper proposes a methodology for automated winter road surface condition classification using convolutional neural network (CNN) and the combination of thermal and visible light cameras. As part of this research, 4244 pairs of visible light and thermal images are captured from pavement surfaces and classified into snowy, icy, wet, and slushy surface conditions. Two single-stream CNN models (visible light and thermal streams) and one dual-stream CNN model are developed. The average F1-Score of dual-stream model is 0.866, 0.935, 0.985, and 0.888 on snowy, icy, wet, and slushy, respectively. The weighted average F1-Score is 0.94.

An Action Recognition Algorithm for Sprinters Using Machine Learning

The advancements in modern science and technology have greatly promoted the progress of sports science. Advanced technological methods have been widely used in sports training, which have not only improved the scientific level of training but also promoted the continuous growth of sports technology and competition results. With the development of sports science and the gradual deepening of sport practices, the use of scientific training methods and monitoring approaches has improved the effect of sports training and athletes’ performance. This paper takes sprint as the research problem and constructs the image of sprinter’s action recognition based on machine learning. In view of the shortcomings of traditional dual-stream convolutional neural network for processing long-term video information, the time-segmented dual-stream network, based on sparse sampling, is used to better express the characteristics of long-term motion. First, the continuous video frame data is divided into multiple segments, and a short sequence of data containing user actions is formed by randomly sampling each segment of the video frame sequence. Next, it is applied to the dual-stream network for feature extraction. The optical flow image extraction involved in the dual-stream network is implemented by the system using the Lucas–Kanade algorithm. The system in this paper has been tested in actual scenarios, and the results show that the system design meets the expected requirements of the sprinters.

Classification of Motor Imagery EEG Based on Time-Domain and Frequency-Domain Dual-Stream Convolutional Neural Network

Background and objectiveAn important task of the brain-computer interface (BCI) of motor imagery is to extract effective time-domain features, frequency-domain features or time-frequency domain features from the raw electroencephalogram (EEG) signals for classification of motor imagery. However, choosing an appropriate method to combine time domain and frequency domain features to improve the performance of motor imagery recognition is still a research hotspot. MethodsIn order to fully extract and utilize the time-domain and frequency-domain features of EEG in classification tasks, this paper proposed a novel dual-stream convolutional neural network (DCNN), which can use time domain signal and frequency domain signal as the inputs, and the extracted time-domain features and frequency-domain features are fused by linear weighting for classification training. Furthermore, the weight can be learned by the DCNN automatically. ResultsThe experiments based on BCI competition II dataset III and BCI competition IV dataset 2a showed that the model proposed by this study has better performance than other conventional methods. The model used time-frequency signal as the inputs had better performance than the model only used time-domain signals or frequency-domain signals. The accuracy of classification was improved for each subject compared with the models only used one signals as the inputs. ConclusionsFurther analysis shown that the fusion weight of different subject is specifically, adjusting the weight coefficient automatically is helpful to improve the classification accuracy.

CNN-based segmentation of speech balloons and narrative text boxes from comic book page images

Most of the recent research works on comic document images have focused on the reading and distribution of comics digitally due to the evolution of technologies. In this work, the extraction of narrative text boxes and speech balloons, which contain the conversations among comic characters along with their feelings, is presented. Due to the huge variety of drawing styles, the shape of these speech balloons is complex, and extraction is difficult. We present a shape-aware dual-stream convolutional neural network for the segmentation of narrative text boxes and speech balloons of various shapes. In our dual-stream architecture, an added shape module processes edge information of the speech balloons and narrative texts with the main module. Later, the concatenation of these two modules produces more accurate segmentation of speech balloons and narrative text boxes. The proposed method achieves significant performance improvements in terms of both region accuracy (mIOU) and boundary accuracy (F-measure and Hausdorff distance) compared to other state-of-the-art methods on various publicly available comic datasets (namely eBDtheque, DCM and Manga 109 dataset subset) in different languages. In addition, we have developed a new dataset (BCBId) for comics in Bangla, the eighth most spoken language in the world, and propose a method for the development of ground-truth images in a semiautomatic way.

FPGA Implementation of a Binarized Dual Stream Convolutional Neural Network for Service Robots

In this study, with the aim of installing an object recognition algorithm on the hardware device of a service robot, we propose a Binarized Dual Stream VGG-16 (BDS-VGG16) network model to realize high-speed computations and low power consumption. The BDS-VGG16 model has improved in terms of the object recognition accuracy by using not only RGB images but also depth images. It achieved a 99.3% accuracy in tests using an RGB-D Object Dataset. We have also confirmed that the proposed model can be installed in a field-programmable gate array (FPGA). We have further installed BDS-VGG16 Tiny, a small BDS-VGG16 model in XCZU9EG, a system on a chip with a CPU and a middle-scale FPGA on a single chip that can be installed in robots. We have also integrated the BDS-VGG16 Tiny with a robot operating system. As a result, the BDS-VGG16 Tiny installed in the XCZU9EG FPGA realizes approximately 1.9-times more computations than the one installed in the graphics processing unit (GPU) with a power efficiency approximately 8-times higher than that installed in the GPU.