Two-stream Convolutional Neural Network Research Articles

Construction and Application of Quality Assessment Model of No-reference Images Two-Stream Convolutional Neural Network

Construction and Application of Quality Assessment Model of No-reference Images Two-Stream Convolutional Neural Network

DBCAN: DFormer-Based Cross-Attention Network for RGB Depth Semantic Segmentation

Existing RGB-depth semantic segmentation methods primarily rely on symmetric two-stream Convolutional Neural Networks (CNNs) to extract RGB and spatial features separately. However, these architectures have limitations in incorporating spatial features and efficiently fusing RGB and depth information. In this study, we propose a novel architecture called the DFormer-Based Cross-Attention Network (DBCAN), which utilizes DFormer as an encoder for feature extraction and integrates several modifications to address these challenges. While DFormer is leveraged for its strong feature extraction capabilities, our modifications in the decoder focus on improving cross-modal fusion and spatial feature incorporation. We introduce three modules in the decoding process: the Object-Region Generated Module (ORGM), the Feature-Region Relation Module (FRRM), and the Spatial-Semantic Fusion Module (SSFM), which enhance feature interaction and segmentation accuracy. Experimental results on the NYUDepthv2 and SUN-RGBD datasets demonstrate that DBCAN achieves state-of-the-art performance, highlighting the effectiveness of our architectural enhancements in overcoming the limitations of existing models.

Research on Small Sample Rolling Bearing Fault Diagnosis Method Based on Mixed Signal Processing Technology

The diagnosis of bearing faults is a crucial aspect of ensuring the optimal functioning of mechanical equipment. However, in practice, the use of small samples and variable operating conditions may result in suboptimal generalization performance, reduced accuracy, and overfitting for these methods. To address this challenge, this study proposes a bearing fault diagnosis method based on a symmetric two-stream convolutional neural network (CNN). The method employs hybrid signal processing techniques to address the issue of limited data. The method employs a symmetric parallel convolutional neural network (CNN) for the analysis of bearing data. Initially, the data are transformed into time–frequency maps through the utilization of the short-time Fourier transform (STFT) and the simultaneous compressed wavelet transform (SCWT). Subsequently, two sets of one-dimensional vectors are generated by reconstructing the high-resolution features of the faulty samples using a symmetric parallel convolutional neural network (CNN). Feature splicing and fusion are then performed to generate bearing fault diagnosis information and assist fault classification. The experimental results demonstrate that the proposed mixed-signal processing method is effective on small-sample datasets, and verify the feasibility and generality of the symmetric parallel CNN-support vector machine (SVM) model for bearing fault diagnosis under small-sample conditions.

Two-Stream Convolutional Neural Networks for Breathing Pattern Classification: Real-Time Monitoring of Respiratory Disease Patients.

A two-stream convolutional neural network (TCNN) for breathing pattern classification has been devised for the continuous monitoring of patients with infectious respiratory diseases. The TCNN consists of a convolutional neural network (CNN)-based autoencoder and classifier. The encoder of the autoencoder generates deep compressed feature maps, which contain the most important information constituting data. These maps are concatenated with feature maps generated by the classifier to classify breathing patterns. The TCNN, single-stream CNN (SCNN), and state-of-the-art classification models were applied to classify four breathing patterns: normal, slow, rapid, and breath holding. The input data consisted of chest tissue hemodynamic responses measured using a wearable near-infrared spectroscopy device on 14 healthy adult participants. Among the classification models evaluated, random forest had the lowest classification accuracy at 88.49%, while the TCNN achieved the highest classification accuracy at 94.63%. In addition, the proposed TCNN performed 2.6% better in terms of classification accuracy than an SCNN (without an autoencoder). Moreover, the TCNN mitigates the issue of declining learning performance with increasing network depth, as observed in the SCNN model. These results prove the robustness of the TCNN in classifying breathing patterns despite using a significantly smaller number of parameters and computations compared to state-of-the-art classification models.

Constructing two-stream input matrices in a convolutional neural network for photovoltaic power prediction

To enhance the prediction accuracy of photovoltaic (PV) power and provide a more exact decision basis for power system management, an end-to-end reinforced two-stream convolutional neural network (RTS-CNN) is proposed. A temporal data converter is firstly devised to adaptively transform the input series into two different input matrices for improving the representational ability of the input data. One matrix is obtained through the tensor dimension adjustment, while the other uses Fast Fourier Transform (FFT) to acquire periodic information to reconstruct the shape of raw data. Next, to fully extract features from these two input matrices, the RTS-CNN model selects the powerful ConvNeXt block as the foundational building block of the encoder. Subsequently, the PV power prediction is realized using a decoder augmented with the Convolutional Block Attention Module (CBAM). Finally, for better performance and learning speed, the model is trained using the Second-order Clipped Stochastic Optimization (Sophia). Three sets of comparative experiments are conducted on the dataset obtained from the Desert Knowledge Australia Solar Centre (DKASC), and the model performance is further validated using the Diebold-Mariano (DM) test and external validation. The experimental results show that the proposed RTS-CNN model has outstanding performance in predicting PV power under different temporal resolutions and seasonal conditions. In the spring season, with time intervals of 15 min, 30 min, and 1 h, the RTS-CNN model achieves root mean square errors (RMSE) of 0.5113, 0.5984, and 0.7801, respectively.

Spatial-temporal multiscale feature optimization based two-stream convolutional neural network for action recognition

Spatial-temporal multiscale feature optimization based two-stream convolutional neural network for action recognition

AF-FTTSnet: An end-to-end two-stream convolutional neural network for online quality monitoring of robotic welding

Online welding quality monitoring (WQM) is crucial for intelligent welding, and deep learning approaches considering spatiotemporal features for WQM tasks show great potential. However, one of the important challenges for existing approaches is to balance the spatiotemporal representation learning capability and computational efficiency, which makes it challenging to adapt welding processes with complex and drastic molten pool dynamic behavior. This paper proposes a novel approach for WQM using molten pool visual sensing and deep learning considering spatiotemporal features, the proposed deep learning network called attention fusion based frame-temporality two-stream network (AF-FTTSnet). Firstly, a passive vision sensor is used to acquire continuous dynamic molten pool images. Meanwhile, temporal difference images are computed to provide novel features and temporal representations. Then, a two-stream feature extraction module is designed to concurrently extract rich spatiotemporal features from molten pool images and temporal difference images. Finally, an attention fusion module with the ability to automatically identify and weight the most relevant features is designed to achieve optimal fusion of the two-stream features. The shop welding experimental results indicate that the proposed AF-FTTSnet model can effectively and robustly recognize five typical welding states during helium arc welding, with an accuracy of 99.26%. This model has been demonstrated to exhibit significant performance improvements compared to mainstream temporal sequence models. Available: https://github.com/Just199806/TSCNN/tree/master.

Spatiotemporal Deep Learning-Based Cine Loop Quality Filter for Handheld Point-of-Care Echocardiography.

The reliability of automated image interpretation of point-of-care (POC) echocardiography scans depends on the quality of the acquired ultrasound data. This work reports on the development and validation of spatiotemporal deep learning models to assess the suitability of input ultrasound cine loops collected using a handheld echocardiography device for processing by an automated quantification algorithm (e.g. ejection fraction estimation). POC echocardiograms (n=885 DICOM cine loops from 175 patients) from two sites were collected using a handheld ultrasound device and annotated for image quality at the frame-level. Attributes of high-quality frames for left ventricular (LV) quantification included a temporally-stable LV, reasonable coverage of LV borders, and good contrast between the borders and chamber. Attributes of low-quality frames included temporal instability of the LV and/or imaging artifacts (e.g., lack of contrast, haze, reverberation, acoustic shadowing). Three different neural network architectures were investigated - (a) frame-level convolutional neural network (CNN) which operates on individual echo frames (VectorCNN), (b) single-stream sequence-level CNN which operates on a sequence of echo frames (VectorCNN+LSTM) and (c) two-stream sequence-level CNNs which operate on a sequence of echo and optical flow frames (VectorCNN+LSTM+Average, VectorCNN+LSTM+MinMax, and VectorCNN+LSTM+ConvPool). Evaluation on a sequestered test dataset containing 76 DICOM cine loops with 16,914 frames showed that VectorCNN+LSTM can effectively utilize both spatial and temporal information to regress the quality of an input frame (accuracy: 0.925, sensitivity = 0.860, specificity = 0.952), compared to the frame-level VectorCNN that only utilizes spatial information in that frame (accuracy: 0.903, sensitivity = 0.791, specificity = 0.949). Furthermore, an independent sample t-test indicated that the cine loops classified to be of adequate quality by the VectorCNN+LSTM model had a statistically significant lower bias in the automatically estimated EF (mean bias = - 3.73 ± 7.46 %, versus a clinically obtained reference EF) compared to the loops classified as inadequate (mean bias = -15.92 ± 12.17 %) (p = 0.007). Thus, cine loop stratification using the proposed spatiotemporal CNN model improves the reliability of automated point-of-care echocardiography image interpretation.

Human Action Recognition using Methods Deep Learning

The objective of human action recognition is to recognize and comprehend human behaviour in videos with the help of expertly matched tags. Thus, there are a plethora of applications for human action recognition, such as video surveillance and patient monitoring. In this paper, three methods are developed and implemented to address these challenging tasks. Convolutional neural networks (CNNs) are the basis for the CNN+LSTM, 3D CNN, and TwoStream CNN algorithms, which are used to recognize human actions in videos.

Target Detection Based on Two-Stream Convolution Neural Network With Self-Powered Sensors Information

With the rapid development of artificial intelligence, neural network is widely used in various fields. Target detection algorithm is mainly based on neural network, but the accuracy of target detection algorithm is greatly related to the complexity of scene and texture. A target detection algorithm based on RGB-D image from the perspective of the lightweight of target detection network model and the integration of depth map to overcome the weak environmental illumination with self-powered sensors information is proposed. This paper analyzes the network model structure of YOLOv4 and MobileNet, compares the variation of parameter number between depthwise separable convolution and convolutional neural network, and combines the advantages of YOLOv4 network and MobileNetv3 network. The main network of three effective feature layers in YOLOv4 is replaced by MobileNetv3 network for initial feature layer extraction to strengthen the feature extraction network. At the same time, the standard convolution models in the network are replaced by depthwise separable convolution. The proposed method is compared with YOLOv4 and YOLOv4-MobileNetv3 in this paper, and the experimental results show that the proposed network retains its original advantages in accuracy, but the size of the network model is about 23% of that of YOLOv4 network model, and the processing speed is about 42% higher than that of YOLOv4 network model, and the detection accuracy can still reach 83% in the environment with poor lighting conditions.

Detection of background forgery using a two-stream convolutional neural network architecture

Detection of background forgery using a two-stream convolutional neural network architecture

Retracted: A Novel English Translation Model in Complex Environments Using Two-Stream Convolutional Neural Networks.

[This retracts the article DOI: 10.1155/2022/8426460.].

Facial Expression Identification using Two-Stream Convolutional Neural Networks (TSCNNs) and Inception 3D Convolutional Neural Network (CNN)

Facial Expression Identification using Two-Stream Convolutional Neural Networks (TSCNNs) and Inception 3D Convolutional Neural Network (CNN)

Automated major depressive disorder diagnosis using a dual-input deep learning model and image generation from EEG signals

Major depressive disorder (MDD) is conventionally diagnosed through a questionnaire. Since approaches to diagnose MDD may lead to inaccurate diagnoses, many studies have presented electroencephalogram (EEG)-based machine learning techniques. The present paper introduces a deep learning approach based on image construction from EEGs. Two images are constructed from EEGs based on spectral and functional connectivity features. Afterward, the constructed images are applied to a two-stream convolutional neural network, and the outputs are concatenated. Finally, the concatenating result is applied to a sequential model of long–short-term memory, fully connected, and softmax layers to classify each sample into the MDD and healthy control (HC) classes. To validate the proposed approach, a public EEG dataset was used consisting of EEG data recorded from 34 MDD patients and 30 HC-matched participants. This framework obtained an AC of 98.03%, SE of 98.85%, SP of 97.19%, F1 of 98.07%, and FDR of 2.69% for the random splitting assessment method and achieved an average AC of 99.11%, SE of 98.97%, SP of 99.25%, F1 of 99.13%, and FDR of 0.71% using a 10-fold cross-validation process. Considering the accurate performance of the proposed method, it can be developed as a computer-aided diagnosis tool to diagnose MDD automatically.

Direction-guided two-stream convolutional neural networks for skeleton-based action recognition

In skeleton-based action recognition, treating skeleton data as pseudoimages using convolutional neural networks (CNNs) has proven to be effective. However, among existing CNN-based approaches, most focus on modeling information at the joint-level ignoring the size and direction information of the skeleton edges, which play an important role in action recognition, and these approaches may not be optimal. In addition, combining the directionality of human motion to portray action motion variation information is rarely considered in existing approaches, although it is more natural and reasonable for action sequence modeling. In this work, we propose a novel direction-guided two-stream convolutional neural network for skeleton-based action recognition. In the first stream, our model focuses on our defined edge-level information (including edge and edge_motion information) with directionality in the skeleton data to explore the spatiotemporal features of the action. In the second stream, since the motion is directional, we define different skeleton edge directions and extract different motion information (including translation and rotation information) in different directions to better exploit the motion features of the action. In addition, we propose a description of human motion inscribed by a combination of translation and rotation, and explore how they are integrated. We conducted extensive experiments on two challenging datasets, the NTU-RGB+D 60 and NTU-RGB+D 120 datasets, to verify the superiority of our proposed method over state-of-the-art methods. The experimental results demonstrate that the proposed direction-guided edge-level information and motion information complement each other for better action recognition.

Convolutional Transformer Fusion Blocks for Multi-Modal Gesture Recognition

Gesture recognition defines an important information channel in human-computer interaction. Intuitively, combining inputs from multiple modalities improves the recognition rate. In this work, we explore multi-modal video-based gesture recognition tasks by fusing spatio-temporal representation of relevant distinguishing features from different modalities. We present a self-attention based transformer fusion architecture to distill the knowledge from different modalities in two-stream convolutional neural networks (CNNs). For this, we introduce convolutions into the self-attention function and design the Convolutional Transformer Fusion Blocks (CTFB) for multi-modal data fusion. These fusion blocks can be easily added at different abstraction levels of the feature hierarchy in existing two-stream CNNs. In addition, the information exchange between two-stream CNNs along the feature hierarchy has so far been barely explored. We propose and evaluate different architectures for multi-level fusion pathways using CTFB to gain insights into the information flow between both streams. Our method achieves state-of-the-art or competitive performance on three benchmark gesture recognition datasets: a) IsoGD, b) NVGesture, and c) IPN hand. Extensive evaluation demonstrates the effectiveness of the proposed CTFB both in terms of recognition rate as well as resource efficiency.

Contrastive Accelerometer–Gyroscope Embedding Model for Human Activity Recognition

Recently, the widespread use of smart devices has invoked more interest in sensor-based applications. Human activity recognition (HAR) with body-worn sensors is an underlying task that aims to recognize a person’s physical activity from on-body sensor readings. In this article, we address the HAR problem that utilizes accelerometers and gyroscopes. While deep-learning-based feature extraction methods are advancing, recent HAR systems with multimodal sensors mostly use data-level fusion, aggregating different sensor signals into one multichannel input. However, they neglect the fact that different sensors capture different physical properties and produce distinct patterns. In this article, we propose a two-stream convolutional neural network (CNN) model that processes the accelerometer and gyroscope signals separately. The modality-specific features are fused in feature-level and jointly used for the recognition task. Furthermore, we introduce a self-supervised learning (SSL) task that pairs the accelerometer and the gyroscope embeddings acquired from the same activity instance. This auxiliary objective allows the feature extractors of our model to communicate during training and exploit complementary information, achieving better representations for HAR. We name our end-to-end multimodal HAR system the Contrastive Accelerometer–Gyroscope Embedding (CAGE) model. CAGE outperforms preceding HAR models in four publicly available benchmarks. The code is available at github.com/quotation2520/CAGE4HAR.

Learning Spatiotemporal Graph Representations for Visual Perception Using EEG Signals.

Perceiving and recognizing objects enable interaction with the external environment. Recently, decoding brain signals based on brain-computer interface (BCI) that recognize the user's intentions by just looking at objects has attracted attention as a next-generation intuitive interface. However, classifying signals from different objects is very challenging, and in practice, decoding performance for visual perception is not yet high enough to be used in real environments. In this study, we aimed to classify single-trial electroencephalography signals evoked by visual stimuli into their corresponding semantic category. We proposed a two-stream convolutional neural network to increase classification performance. The model consists of a spatial stream and a temporal stream that use graph convolutional neural network and channel-wise convolutional neural network respectively. Two public datasets were used to evaluate the proposed model; (i) SU DB (a set of 72 photographs of objects belonging to 6 semantic categories) and MPI DB (8 exemplars belonging to two categories). Our results outperform state-of-the-art methods, with accuracies of 54.28 ± 7.89% for SU DB (6-class) and 84.40 ± 8.03% for MPI DB (2-class). These results could facilitate the application of intuitive BCI systems based on visual perception.

Binary Dense SIFT Flow Based Position-Information Added Two-Stream CNN for Pedestrian Action Recognition

Pedestrian behavior recognition in the driving environment is an important technology to prevent pedestrian accidents by predicting the next movement. It is necessary to recognize current pedestrian behavior to predict future pedestrian behavior. However, many studies have recognized human visible characteristics such as face, body parts or clothes, but few have recognized pedestrian behavior. It is challenging to recognize pedestrian behavior in the driving environment due to the changes in the camera field of view due to the illumination conditions in outdoor environments and vehicle movement. In this paper, to predict pedestrian behavior, we introduce a position-information added two-stream convolutional neural network (CNN) with multi task learning that is robust to the limited conditions of the outdoor driving environment. The conventional two-stream CNN is the most widely used model for human-action recognition. However, the conventional two-stream CNN based on optical flow has limitations regarding pedestrian behavior recognition in a moving vehicle because of the assumptions of brightness constancy and piecewise smoothness. To solve this problem for a moving vehicle, the binary descriptor dense scale-invariant feature transform (SIFT) flow, a feature-based matching algorithm, is robust in moving-pedestrian behavior recognition, such as walking and standing, in a moving vehicle. However, recognizing cross attributes, such as crossing or not crossing the street, is challenging using the binary descriptor dense SIFT flow because people who cross the road or not act the same walking action, but their location on the image is different. Therefore, pedestrian position information should be added to the conventional binary descriptor dense SIFT flow two-stream CNN. Thus, learning biased toward action attributes is evenly learned across action and cross attributes. In addition, YOLO detection and the Siamese tracker are used instead of the ground-truth boundary box to prove the robustness in the action- and cross-attribute recognition from a moving vehicle. The JAAD and PIE datasets were used for training, and only the JAAD dataset was used as a testing dataset for comparison with other state-of-the-art research on multitask and single-task learning.

Bolt-Loosening Detection Using 1D and 2D Input Data Based on Two-Stream Convolutional Neural Networks.

At present, the detection accuracy of bolt-loosening diagnoses is still not high. In order to improve the detection accuracy, this paper proposes a fault diagnosis model based on the TSCNN model, which can simultaneously extract fault features from vibration signals and time-frequency images and can precisely detect the bolt-loosening states. In this paper, the LeNet-5 network is improved by adjusting the size and number of the convolution kernels, introducing the dropout operation, and building a two-dimensional convolutional neural network (2DCNN) model. Combining the advantages of a one-dimensional convolutional neural network (1DCNN) with wide first-layer kernels to suppress high-frequency noise, a two-stream convolutional neural network (TSCNN) is proposed based on 1D and 2D input data. The proposed model uses raw vibration signals and time-frequency images as input and automatically extracts sensitive features and representative information. Finally, the effectiveness and superiority of the proposed approach are verified by practical experiments that are carried out on a machine tool guideway. The experimental results show that the proposed approach can effectively achieve end-to-end bolt-loosening fault diagnoses, with an average recognition accuracy of 99.58%. In addition, the method can easily achieve over 93% accuracy when the SNR is over 0 dB without any denoising preprocessing. The results show that the proposed approach not only achieves high classification accuracy but also has good noise immunity.