Two-stream Convolutional Neural Network Research Articles

AF-FTTSnet: An end-to-end two-stream convolutional neural network for online quality monitoring of robotic welding

Online welding quality monitoring (WQM) is crucial for intelligent welding, and deep learning approaches considering spatiotemporal features for WQM tasks show great potential. However, one of the important challenges for existing approaches is to balance the spatiotemporal representation learning capability and computational efficiency, which makes it challenging to adapt welding processes with complex and drastic molten pool dynamic behavior. This paper proposes a novel approach for WQM using molten pool visual sensing and deep learning considering spatiotemporal features, the proposed deep learning network called attention fusion based frame-temporality two-stream network (AF-FTTSnet). Firstly, a passive vision sensor is used to acquire continuous dynamic molten pool images. Meanwhile, temporal difference images are computed to provide novel features and temporal representations. Then, a two-stream feature extraction module is designed to concurrently extract rich spatiotemporal features from molten pool images and temporal difference images. Finally, an attention fusion module with the ability to automatically identify and weight the most relevant features is designed to achieve optimal fusion of the two-stream features. The shop welding experimental results indicate that the proposed AF-FTTSnet model can effectively and robustly recognize five typical welding states during helium arc welding, with an accuracy of 99.26%. This model has been demonstrated to exhibit significant performance improvements compared to mainstream temporal sequence models. Available: https://github.com/Just199806/TSCNN/tree/master.

Spatiotemporal Deep Learning-Based Cine Loop Quality Filter for Handheld Point-of-Care Echocardiography.

The reliability of automated image interpretation of point-of-care (POC) echocardiography scans depends on the quality of the acquired ultrasound data. This work reports on the development and validation of spatiotemporal deep learning models to assess the suitability of input ultrasound cine loops collected using a handheld echocardiography device for processing by an automated quantification algorithm (e.g. ejection fraction estimation). POC echocardiograms (n=885 DICOM cine loops from 175 patients) from two sites were collected using a handheld ultrasound device and annotated for image quality at the frame-level. Attributes of high-quality frames for left ventricular (LV) quantification included a temporally-stable LV, reasonable coverage of LV borders, and good contrast between the borders and chamber. Attributes of low-quality frames included temporal instability of the LV and/or imaging artifacts (e.g., lack of contrast, haze, reverberation, acoustic shadowing). Three different neural network architectures were investigated - (a) frame-level convolutional neural network (CNN) which operates on individual echo frames (VectorCNN), (b) single-stream sequence-level CNN which operates on a sequence of echo frames (VectorCNN+LSTM) and (c) two-stream sequence-level CNNs which operate on a sequence of echo and optical flow frames (VectorCNN+LSTM+Average, VectorCNN+LSTM+MinMax, and VectorCNN+LSTM+ConvPool). Evaluation on a sequestered test dataset containing 76 DICOM cine loops with 16,914 frames showed that VectorCNN+LSTM can effectively utilize both spatial and temporal information to regress the quality of an input frame (accuracy: 0.925, sensitivity = 0.860, specificity = 0.952), compared to the frame-level VectorCNN that only utilizes spatial information in that frame (accuracy: 0.903, sensitivity = 0.791, specificity = 0.949). Furthermore, an independent sample t-test indicated that the cine loops classified to be of adequate quality by the VectorCNN+LSTM model had a statistically significant lower bias in the automatically estimated EF (mean bias = - 3.73 ± 7.46 %, versus a clinically obtained reference EF) compared to the loops classified as inadequate (mean bias = -15.92 ± 12.17 %) (p = 0.007). Thus, cine loop stratification using the proposed spatiotemporal CNN model improves the reliability of automated point-of-care echocardiography image interpretation.

Target Detection Based on Two-Stream Convolution Neural Network With Self-Powered Sensors Information

With the rapid development of artificial intelligence, neural network is widely used in various fields. Target detection algorithm is mainly based on neural network, but the accuracy of target detection algorithm is greatly related to the complexity of scene and texture. A target detection algorithm based on RGB-D image from the perspective of the lightweight of target detection network model and the integration of depth map to overcome the weak environmental illumination with self-powered sensors information is proposed. This paper analyzes the network model structure of YOLOv4 and MobileNet, compares the variation of parameter number between depthwise separable convolution and convolutional neural network, and combines the advantages of YOLOv4 network and MobileNetv3 network. The main network of three effective feature layers in YOLOv4 is replaced by MobileNetv3 network for initial feature layer extraction to strengthen the feature extraction network. At the same time, the standard convolution models in the network are replaced by depthwise separable convolution. The proposed method is compared with YOLOv4 and YOLOv4-MobileNetv3 in this paper, and the experimental results show that the proposed network retains its original advantages in accuracy, but the size of the network model is about 23% of that of YOLOv4 network model, and the processing speed is about 42% higher than that of YOLOv4 network model, and the detection accuracy can still reach 83% in the environment with poor lighting conditions.

Detection of background forgery using a two-stream convolutional neural network architecture

Detection of background forgery using a two-stream convolutional neural network architecture

Retracted: A Novel English Translation Model in Complex Environments Using Two-Stream Convolutional Neural Networks.

[This retracts the article DOI: 10.1155/2022/8426460.].

Automated major depressive disorder diagnosis using a dual-input deep learning model and image generation from EEG signals

Major depressive disorder (MDD) is conventionally diagnosed through a questionnaire. Since approaches to diagnose MDD may lead to inaccurate diagnoses, many studies have presented electroencephalogram (EEG)-based machine learning techniques. The present paper introduces a deep learning approach based on image construction from EEGs. Two images are constructed from EEGs based on spectral and functional connectivity features. Afterward, the constructed images are applied to a two-stream convolutional neural network, and the outputs are concatenated. Finally, the concatenating result is applied to a sequential model of long–short-term memory, fully connected, and softmax layers to classify each sample into the MDD and healthy control (HC) classes. To validate the proposed approach, a public EEG dataset was used consisting of EEG data recorded from 34 MDD patients and 30 HC-matched participants. This framework obtained an AC of 98.03%, SE of 98.85%, SP of 97.19%, F1 of 98.07%, and FDR of 2.69% for the random splitting assessment method and achieved an average AC of 99.11%, SE of 98.97%, SP of 99.25%, F1 of 99.13%, and FDR of 0.71% using a 10-fold cross-validation process. Considering the accurate performance of the proposed method, it can be developed as a computer-aided diagnosis tool to diagnose MDD automatically.

Direction-guided two-stream convolutional neural networks for skeleton-based action recognition

In skeleton-based action recognition, treating skeleton data as pseudoimages using convolutional neural networks (CNNs) has proven to be effective. However, among existing CNN-based approaches, most focus on modeling information at the joint-level ignoring the size and direction information of the skeleton edges, which play an important role in action recognition, and these approaches may not be optimal. In addition, combining the directionality of human motion to portray action motion variation information is rarely considered in existing approaches, although it is more natural and reasonable for action sequence modeling. In this work, we propose a novel direction-guided two-stream convolutional neural network for skeleton-based action recognition. In the first stream, our model focuses on our defined edge-level information (including edge and edge_motion information) with directionality in the skeleton data to explore the spatiotemporal features of the action. In the second stream, since the motion is directional, we define different skeleton edge directions and extract different motion information (including translation and rotation information) in different directions to better exploit the motion features of the action. In addition, we propose a description of human motion inscribed by a combination of translation and rotation, and explore how they are integrated. We conducted extensive experiments on two challenging datasets, the NTU-RGB+D 60 and NTU-RGB+D 120 datasets, to verify the superiority of our proposed method over state-of-the-art methods. The experimental results demonstrate that the proposed direction-guided edge-level information and motion information complement each other for better action recognition.

Convolutional Transformer Fusion Blocks for Multi-Modal Gesture Recognition

Gesture recognition defines an important information channel in human-computer interaction. Intuitively, combining inputs from multiple modalities improves the recognition rate. In this work, we explore multi-modal video-based gesture recognition tasks by fusing spatio-temporal representation of relevant distinguishing features from different modalities. We present a self-attention based transformer fusion architecture to distill the knowledge from different modalities in two-stream convolutional neural networks (CNNs). For this, we introduce convolutions into the self-attention function and design the Convolutional Transformer Fusion Blocks (CTFB) for multi-modal data fusion. These fusion blocks can be easily added at different abstraction levels of the feature hierarchy in existing two-stream CNNs. In addition, the information exchange between two-stream CNNs along the feature hierarchy has so far been barely explored. We propose and evaluate different architectures for multi-level fusion pathways using CTFB to gain insights into the information flow between both streams. Our method achieves state-of-the-art or competitive performance on three benchmark gesture recognition datasets: a) IsoGD, b) NVGesture, and c) IPN hand. Extensive evaluation demonstrates the effectiveness of the proposed CTFB both in terms of recognition rate as well as resource efficiency.

Contrastive Accelerometer–Gyroscope Embedding Model for Human Activity Recognition

Recently, the widespread use of smart devices has invoked more interest in sensor-based applications. Human activity recognition (HAR) with body-worn sensors is an underlying task that aims to recognize a person’s physical activity from on-body sensor readings. In this article, we address the HAR problem that utilizes accelerometers and gyroscopes. While deep-learning-based feature extraction methods are advancing, recent HAR systems with multimodal sensors mostly use data-level fusion, aggregating different sensor signals into one multichannel input. However, they neglect the fact that different sensors capture different physical properties and produce distinct patterns. In this article, we propose a two-stream convolutional neural network (CNN) model that processes the accelerometer and gyroscope signals separately. The modality-specific features are fused in feature-level and jointly used for the recognition task. Furthermore, we introduce a self-supervised learning (SSL) task that pairs the accelerometer and the gyroscope embeddings acquired from the same activity instance. This auxiliary objective allows the feature extractors of our model to communicate during training and exploit complementary information, achieving better representations for HAR. We name our end-to-end multimodal HAR system the Contrastive Accelerometer–Gyroscope Embedding (CAGE) model. CAGE outperforms preceding HAR models in four publicly available benchmarks. The code is available at github.com/quotation2520/CAGE4HAR.

Learning Spatiotemporal Graph Representations for Visual Perception Using EEG Signals.

Perceiving and recognizing objects enable interaction with the external environment. Recently, decoding brain signals based on brain-computer interface (BCI) that recognize the user's intentions by just looking at objects has attracted attention as a next-generation intuitive interface. However, classifying signals from different objects is very challenging, and in practice, decoding performance for visual perception is not yet high enough to be used in real environments. In this study, we aimed to classify single-trial electroencephalography signals evoked by visual stimuli into their corresponding semantic category. We proposed a two-stream convolutional neural network to increase classification performance. The model consists of a spatial stream and a temporal stream that use graph convolutional neural network and channel-wise convolutional neural network respectively. Two public datasets were used to evaluate the proposed model; (i) SU DB (a set of 72 photographs of objects belonging to 6 semantic categories) and MPI DB (8 exemplars belonging to two categories). Our results outperform state-of-the-art methods, with accuracies of 54.28 ± 7.89% for SU DB (6-class) and 84.40 ± 8.03% for MPI DB (2-class). These results could facilitate the application of intuitive BCI systems based on visual perception.

Binary Dense SIFT Flow Based Position-Information Added Two-Stream CNN for Pedestrian Action Recognition

Pedestrian behavior recognition in the driving environment is an important technology to prevent pedestrian accidents by predicting the next movement. It is necessary to recognize current pedestrian behavior to predict future pedestrian behavior. However, many studies have recognized human visible characteristics such as face, body parts or clothes, but few have recognized pedestrian behavior. It is challenging to recognize pedestrian behavior in the driving environment due to the changes in the camera field of view due to the illumination conditions in outdoor environments and vehicle movement. In this paper, to predict pedestrian behavior, we introduce a position-information added two-stream convolutional neural network (CNN) with multi task learning that is robust to the limited conditions of the outdoor driving environment. The conventional two-stream CNN is the most widely used model for human-action recognition. However, the conventional two-stream CNN based on optical flow has limitations regarding pedestrian behavior recognition in a moving vehicle because of the assumptions of brightness constancy and piecewise smoothness. To solve this problem for a moving vehicle, the binary descriptor dense scale-invariant feature transform (SIFT) flow, a feature-based matching algorithm, is robust in moving-pedestrian behavior recognition, such as walking and standing, in a moving vehicle. However, recognizing cross attributes, such as crossing or not crossing the street, is challenging using the binary descriptor dense SIFT flow because people who cross the road or not act the same walking action, but their location on the image is different. Therefore, pedestrian position information should be added to the conventional binary descriptor dense SIFT flow two-stream CNN. Thus, learning biased toward action attributes is evenly learned across action and cross attributes. In addition, YOLO detection and the Siamese tracker are used instead of the ground-truth boundary box to prove the robustness in the action- and cross-attribute recognition from a moving vehicle. The JAAD and PIE datasets were used for training, and only the JAAD dataset was used as a testing dataset for comparison with other state-of-the-art research on multitask and single-task learning.

Bolt-Loosening Detection Using 1D and 2D Input Data Based on Two-Stream Convolutional Neural Networks.

At present, the detection accuracy of bolt-loosening diagnoses is still not high. In order to improve the detection accuracy, this paper proposes a fault diagnosis model based on the TSCNN model, which can simultaneously extract fault features from vibration signals and time-frequency images and can precisely detect the bolt-loosening states. In this paper, the LeNet-5 network is improved by adjusting the size and number of the convolution kernels, introducing the dropout operation, and building a two-dimensional convolutional neural network (2DCNN) model. Combining the advantages of a one-dimensional convolutional neural network (1DCNN) with wide first-layer kernels to suppress high-frequency noise, a two-stream convolutional neural network (TSCNN) is proposed based on 1D and 2D input data. The proposed model uses raw vibration signals and time-frequency images as input and automatically extracts sensitive features and representative information. Finally, the effectiveness and superiority of the proposed approach are verified by practical experiments that are carried out on a machine tool guideway. The experimental results show that the proposed approach can effectively achieve end-to-end bolt-loosening fault diagnoses, with an average recognition accuracy of 99.58%. In addition, the method can easily achieve over 93% accuracy when the SNR is over 0 dB without any denoising preprocessing. The results show that the proposed approach not only achieves high classification accuracy but also has good noise immunity.

A Novel English Translation Model in Complex Environments Using Two-Stream Convolutional Neural Networks.

Although translation is an essential component of learning English, it does not receive the attention it merits in the modern English classroom. Teachers and students primarily emphasize listening, reading, and writing while neglecting the development of translation skills. The English test in China now reflects the fact that there are now very specific requirements for students' translation skills. As a result, we should emphasize developing students' translation skills when teaching them English. The following experimental data can be obtained following the study and experiment on the English translation simulation model based on the two-stream convolutional neural network: English vocabulary and grammar have passing and excellent rates of 90 and 57 percent, respectively, while reading has passing and excellent rates of 69 and 8 percent, respectively. The ability of students to translate into English has significantly improved after using the English translation simulation model based on the two-stream convolutional neural network.

Temporal grafter network: Rethinking LSTM for effective video recognition

Long short-term memory (LSTM) networks are widely used to handle temporal or sequential data, and have great potential for video recognition. Existing LSTM-based video recognition methods either insert LSTM modules at the end of 2D convolutional neural networks (CNNs), called global LSTM methods, or build networks solely by stacking multiple LSTM modules. Unfortunately, these LSTM-based video recognition methods are not competitive, compared to state-of-the-art 3D CNNs or two-stream CNNs. In order to fully explore the potential of LSTM, this paper rethinks its role in video recognition network architectures and proposes a novel Temporal Grafter Network (TGN). Specifically, we develop an efficient and effective variant of convolutional LSTM module, which is grafted between different stages of very deep 2D CNNs for temporal modeling and delivery. Our TGN can capture local motion patterns of varying scales inherent in feature maps from high to low resolutions, while attending to the spatial context information and modeling global temporal dependency across the whole video. The proposed TGN can capture and transmit temporal information throughout very deep 2D CNNs, overcoming the downsides of existing LSTM-based methods and being able to make full use of the potential of LSTM for effective video recognition and early action recognition. We perform extensive ablation study to verify the effectiveness of our proposed methods, and experiments on three widely used video benchmarks show that our methods can achieve performance matching or better than the state-of-the-arts.

Spatiotemporal image fusion using multiscale attention-aware two-stream convolutional neural networks

Spatiotemporal fusion (STF) is considered as a promising way to produce remote sensing images at fine scales in both space and time by blending two types of satellite images. The learning-based STF approaches with deep convolutional neural networks can provide the unified framework to address both gradual and abrupt changes. This paper intends to develop an enhanced learning-based STF using multiscale attention-aware two-stream convolutional neural networks (MACNN). With a coarse image at the prediction date and two pairs of coarse and fine images at other dates as inputs, it employs a multiscale module to characterize different sizes of objects and a spatial and channel attention module to emphasize important information in feature learning. Two experiments on real Landsat and MODIS images are conducted to demonstrate the effectiveness of the proposed MACNN and it outperforms four existing STF methods in both visual and quantitative.

Single Image Dehazing Based on Two-Stream Convolutional Neural Network

Objective The haze weather environment leads to the deterioration of the visual effect of the image, and it is difficult to carry out the work of the advanced vision task. Therefore, dehazing the haze image is an important step before the execution of the advanced vision task. Traditional dehazing algorithms achieve image dehazing by improving image brightness and contrast or constructing artificial priors such as color attenuation priors and dark channel priors, but the effect is unstable when dealing with complex scenes. In the method based on convolutional neural network, the image dehazing network of the encoding and decoding structure does not consider the difference before and after the dehazing image, and the image spatial information is lost in the encoding stage. In order to overcome these problems, this paper proposes a novel end-to-end two-stream convolutional neural network for single image dehazing. Method The network model is composed of a spatial information feature stream and a high-level semantic feature stream. The spatial information feature stream retains the detailed information of the dehazing image, and the high-level semantic feature stream extracts the multi-scale structural features of the dehazing image. A spatial information auxiliary module is designed between the feature streams. This module uses the attention mechanism to construct a unified expression of different types of information, and realizes the gradual restoration of the clear image with the semantic information auxiliary spatial information in the dehazing network. A parallel residual twicing module is proposed, which performs dehazing on the difference information of features at different stages to improve the model’s ability to discriminate haze images. Result The peak signal-to-noise ratio and structural similarity are used to quantitatively evaluate the similarity between the dehazing results of each algorithm and the original image. The structure similarity and peak signal-to-noise ratio of the method in this paper reached 0.852 and 17.557dB on the Hazerd dataset, which were higher than all comparison algorithms. On the SOTS dataset, the indicators are 0.955 and 27.348dB, which are sub-optimal results. In experiments with real haze images, this method can also achieve excellent visual restoration effects.

Multi-head attention-based two-stream EfficientNet for action recognition

Recent years have witnessed the popularity of using two-stream convolutional neural networks for action recognition. However, existing two-stream convolutional neural network-based action recognition approaches are incapable of distinguishing some roughly similar actions in videos such as sneezing and yawning. To solve this problem, we propose a Multi-head Attention-based Two-stream EfficientNet (MAT-EffNet) for action recognition, which can take advantage of the efficient feature extraction of EfficientNet. The proposed network consists of two streams (i.e., a spatial stream and a temporal stream), which first extract the spatial and temporal features from consecutive frames by using EfficientNet. Then, a multi-head attention mechanism is utilized on the two streams to capture the key action information from the extracted features. The final prediction is obtained via a late average fusion, which averages the softmax score of spatial and temporal streams. The proposed MAT-EffNet can focus on the key action information at different frames and compute the attention multiple times, in parallel, to distinguish similar actions. We test the proposed network on the UCF101, HMDB51 and Kinetics-400 datasets. Experimental results show that the MAT-EffNet outperforms other state-of-the-art approaches for action recognition.

Adoption of Artificial Intelligence Along with Gesture Interactive Robot in Musical Perception Education Based on Deep Learning Method

The present work aims to improve students’ interest in music teaching and promote modern teaching. A distributed application system of artificial intelligence gesture interactive robot is designed through deep learning technology and applied to music perception education. First, the user’s gesture instruction data is collected through the double channel convolution neural network (DCCNN). It uses the double-size convolution kernel to extract feature information in the image and collect the video frame’s gesture instruction. Secondly, a two-stream convolutional neural network (two-stream CNN) recognizes the collected gesture instruction data. The spatial and temporal information is extracted from RGB color mode (RGB) images and optical flow images and input into the two-stream CNN to fuse the prediction results of each network as the final detection result. Then, the distributed system used by the interactive robot is introduced. This structure can improve the stability of the interactive systems and reduce the requirements for local hardware performance. Finally, experiments are conducted to test the gesture command acquisition and recognition network, and the performance of the gesture interactive robot in practice. The results indicate that combining convolution kernels of [Formula: see text] and [Formula: see text] can increase the recognition accuracy of DCCNN to 98% and effectively collect gesture instruction data. The gesture recognition accuracy of two-stream CNN after training reaches 90%, higher than the mainstream dynamic gesture recognition algorithm trained with the same data set. Finally, the recognition test of gesture instructions is carried out on the gesture interactive robot reported here. The results show that the recognition accuracy of the gesture interactive robots is more than 90%, meeting the routine interaction needs. Therefore, the interactive gesture robot has good reliability and stability and is applicable to music perception teaching. The research reported here has guiding significance for establishing music teaching with multiple perception modes.

Spatial-temporal interaction learning based two-stream network for action recognition

Two-stream convolutional neural networks have been widely applied to action recognition. However, two-stream networks are usually adopted to capture spatial information and temporal information separately, which normally ignore the strong complementarity and correlation between spatial and temporal information in videos. To solve this problem, we propose a Spatial-Temporal Interaction Learning Two-stream network (STILT) for action recognition. Our proposed two-stream (i.e., a spatial stream and a temporal stream) network has a spatial–temporal interaction learning module, which uses an alternating co-attention mechanism between two streams to learn the correlation between spatial features and temporal features. The spatial–temporal interaction learning module allows the two streams to guide each other and then generates optimized spatial attention features and temporal attention features. Thus, the proposed network can establish the interactive connection between two streams, which efficiently exploits the attended spatial and temporal features to improve recognition accuracy. Experiments on three widely used datasets (i.e., UCF101, HMDB51 and Kinetics) show that the proposed network outperforms the state-of-the-art models in action recognition.

Video behavior recognition algorithm based on two-stream heterogeneous convolutional neural network

Abstract In video behavior recognition, making full use of the spatio-temporal information contained in video frame is the critical point to further improve the recognition accuracy. In our study, we propose a video behavior recognition algorithm based on two-stream heterogeneous network, so that we can further extract the spatio-temporal feature information in the video frame, and finally take full advantage of the spatio-temporal features in the video sequence. In view of the characteristics of RGB and optical flow images, this paper uses DenseNet121 and Inception-V4 to structure a two-stream network, to fully extract the spatio-temporal feature information in the video. In this paper, UCF101 dataset is used to evaluate the algorithm, and the recognition accuracy is 91.7%, which verified the reliability of the proposed algorithm.