Two-stream Model Research Articles

SFTNet: A microexpression-based method for depression detection

Background and objectivesDepression is a typical mental illness, and early screening can effectively prevent exacerbation of the condition. Many studies have found that the expressions of depressed patients are different from those of other subjects, and microexpressions have been used in the clinical detection of mental illness. However, there are few methods for the automatic detection of depression based on microexpressions. MethodsA new dataset of 156 participants (76 in the case group and 80 in the control group) was created. All data were collected in the context of a new emotional stimulation experiment and doctor-patient conversation. We first analyzed the Average Number of Occurrences (ANO) and Average Duration (AD) of facial expressions in the case group and the control group. Then, we proposed a two-stream model SFTNet for identifying depression based on microexpressions, which consists of a single-temporal network (STNet) and a full-temporal network (FTNet). STNet is used to extract features from facial images at a single time node, FTNet is used to extract features from all-time nodes, and the decision network combines the two features to identify depression through decision fusion. The code for SFTNet is available at https://github.com/muzixingyun/SFTNet. ResultsWe found that the AD of all subjects was less than 20 frames (2/3 seconds) and that the facial expressions of the control group were richer. SFTNet achieved excellent results on the emotional stimulus experimental dataset, with Accuracy, Precision and Recall of 0.873, 0.888 and 0.846, respectively. We also conducted experiments on the doctor-patient conversation dataset, and the Accuracy, Precision and Recall were 0.829, 0.817 and 0.837, respectively. SFTNet can also be applied to microexpression detection task with more accuracy than SOTA models. ConclusionsIn the emotional stimulation experiment, the subjects in the case group are more likely to show negative emotions. Compared to SOTA models, our depression detection method is more accurate and can assist doctors in the diagnosis of depression.

Identification of epileptic networks with graph convolutional network incorporating oscillatory activities and evoked synaptic responses

Stereoelectroencephalography (SEEG) offers unique neural data from in-depth brain structures with fine temporal resolutions to better investigate the origin of epileptic brain activities. Although oscillatory patterns from different frequency bands and functional connectivity computed from the SEEG datasets are employed to study the epileptic zones, direct electrical stimulation-evoked electrophysiological recordings of synaptic responses, namely cortical-cortical evoked potentials (CCEPs), from the same SEEG electrodes are not explored for the localization of epileptic zones. Here we proposed a two-stream model with unsupervised learning and graph convolutional network tailored to the SEEG and CCEP datasets in individual patients to perform localization of epileptic zones. We compared our localization results with the clinically marked electrode sites determined for surgical resections. Our model had good classification capability when compared to other state-of-the-art methods. Furthermore, based on our prediction results we performed group-level brain-area mapping analysis for temporal, frontal and parietal epilepsy patients and found that epileptic and non-epileptic brain networks were distinct in patients with different types of focal epilepsy. Our unsupervised data-driven model provides personalized localization analysis for the epileptic zones. The epileptic and non-epileptic brain areas disclosed by the prediction model provide novel insights into the network-level pathological characteristics of epilepsy.

Falling Behavior Detection System for Elevator Passengers Based on Deep Learning and Edge Computing

Due to factors such as dense populations and narrow viewing angles, previous deep-learning models for detecting passenger behavior in elevators often lack effectiveness. Traditional cloud-based data transmission methods have issues with high latency, high resource usage, and privacy threats, particularly during periods of high usage. To address these issues, we proposed a falling behavior detection system for elevator passengers based on deep learning and edge computing. A two-stream neural network model improved by 3D ResNet is presented, which utilizes edge computing for elevator passenger fall detection. Our homemade dataset of elevator passenger falling behavior is utilized to train and evaluate the system. The results demonstrate that the system is effective in detecting passengers’ falling behavior in elevators, with an average accuracy of 89.2%. The feasibility of the system in an elevator is also verified, and it has performed well. The application of this system in this field holds significant research value.

Predicting Quality of Services Based on a Two-Stream Deep Learning Model With User and Service Graphs

Predicting Quality of Services Based on a Two-Stream Deep Learning Model With User and Service Graphs

Adaptive undersampling and short clip-based two-stream CNN-LSTM model for surgical phase recognition on cholecystectomy videos

Surgical phase recognition is challenging due to overfitting problems caused by imbalanced data among surgical phases. We proposed an adaptive sampling rate-based undersampling method that could generate the number of each surgical phase data similarly to alleviate biased learning. To improve the performance of our method, we also introduced a two-stream CNN-LSTM model that could extract temporal information on behavioral changes between each image frame. First, we extracted a total of 40,236 short clips using an adaptive subsampling rate from the entire video. Each short clip was entered into a pre-trained GoogLeNet. The output with visual information was then immediately fed into a sequence-to-sequence LSTM model to extract temporal information of neighbor frames within a short clip. At the same time, another sequence-to-vector LSTM was used, to extract temporal information from all successive image frames to predict the final surgical phase. The proposed method was evaluated with a public dataset Cholec80. The proposed approach outperformed state-of-the-art methods, showing a high F1-score of 87.12% and an AUC of 98.00%. In addition, the F1-score deviation between all phases decreased by about 10% compared to that before applying undersampling. Experimental results confirmed that employing our proposed method could learn enrich temporal information from short clips. It outperformed the conventional one-stream CNN-LSTM architecture.

DualMLP: a two-stream fusion model for 3D point cloud classification

DualMLP: a two-stream fusion model for 3D point cloud classification

Two-Stream Spatial–Temporal Transformer Networks For Driver Drowsiness Detection

<p>For driver drowsiness detection in the real world, the existing methods have good performance in general. However, when the face is blocked, the light is dim, and the driver’s head posture changes, the performance will deteriorate significantly. In this paper, a two-stream Spatial-Temporal transformer network intended to perform diver drowsiness detection task is proposed to solve the above problems. The spatial-temporal graph is extracted from the video and then the results are obtained from 2s-STTN. The model is a two-stream transformer network model. In our model, the Spatial Self-Attention module is used to learn the embedding of different facial landmarks, and the Temporal Self-Attention module is used to learn the correlation between the frames of facial feature points. Different activated facial landmarks are separated and recognized by class activation mapping technology. Each flow recognizes different activated facial features, extracts spatial or temporal features, and integrates the information about facial features, so as to improve the performance of the system. 2s-STTN can not only mine the long-term dependence of driver behavior from video, but also mine the driver drowsiness information provided by the unobstructed facial signs when the face is blocked. By conducting experiments and comparing our model with other models, it is demonstrated that the proposed model has good performance in driver drowsiness detection.</p> <p> </p>

Shuffle Graph Convolutional Network for Skeleton-Based Action Recognition

A shuffle graph convolutional network (Shuffle-GCN) is proposed to recognize human action by analyzing skeleton data. It uses channel split and channel shuffle operations to process multi-feature channels of skeleton data, which reduces the computational cost of graph convolution operation. Compared with the classical two-stream adaptive graph convolutional network model, the proposed method achieves a higher precision with 1/3 of the floating-point operations (FLOPs). Even more, a channel-level topology modeling method is designed to extract more motion information of human skeleton by learning the graph topology from different channels dynamically. The performance of Shuffle-GCN is tested under 56,880 action clips from the NTU RGB+D dataset with the accuracy 96.0% and the computational complexity 12.8 GFLOPs. The proposed method offers feasible solutions for developing practical applications of action recognition.

Learning facial expression and body gesture visual information for video emotion recognition

Recent research has shown that facial expressions and body gestures are two significant implications in identifying human emotions. However, these studies mainly focus on contextual information of adjacent frames, and rarely explore the spatio-temporal relationships between distant or global frames. In this paper, we revisit the facial expression and body gesture emotion recognition problems, and propose to improve the performance of video emotion recognition by extracting the spatio-temporal features via further encoding temporal information. Specifically, for facial expression, we propose a super image-based spatio-temporal convolutional model (SISTCM) and a two-stream LSTM model to capture the local spatio-temporal features and learn global temporal cues of emotion changes. For body gestures, a novel representation method and an attention-based channel-wise convolutional model (ACCM) are introduced to learn key joints features and independent characteristics of each joint. Extensive experiments on five common datasets are carried out to prove the superiority of the proposed method, and the results proved learning two visual information leads to significant improvement over the existing state-of-the-art methods.

A multi-resolution fusion approach for human activity recognition from video data in tiny edge devices

Human Activity Recognition (HAR) is the process of automatic recognition of Activities of Daily Living (ADL) from human motion data captured in various data modalities by wearable and ambient sensors. Advances in Deep Learning, especially Convolutional Neural Networks (CNNs) and sequential models have revolutionalized HAR from video data sources. Although these models are incredibly accurate and utilize both spatial and temporal information, they require huge computation and memory resources — making them unsuitable for edge or wearable applications. Tiny Machine Learning (TinyML) aims to optimize these models in terms of compute and memory requirements – aiming to make them suitable for always-on resource constrained devices – leading to a reduction in communication latency and network traffic for HAR frameworks. In this paper, we propose a two-stream multi-resolution fusion architecture for HAR from video data modality. The context stream takes a resized image as input and the fovea stream takes the cropped center portion of the resized image as input, reducing the overall dimensionality. We tested two quantization methods: Post-Training Quantization (PTQ) and Quantization Aware Training (QAT) to optimize these models for deployment in edge devices and tested the performance in two challenging video datasets: KTH and UCF11. We performed ablation studies to validate the two-stream model performance. We deployed the proposed architecture in commercial resource constrained devices and monitored their performance in terms of inference latency and power consumption. The results indicate that the proposed architecture clearly outperforms other relevant single-stream models tested in this work in terms of accuracy, precision, recall, and F1-Score while also reducing the overall model size.

FinalMLP: An Enhanced Two-Stream MLP Model for CTR Prediction

Click-through rate (CTR) prediction is one of the fundamental tasks in online advertising and recommendation. Multi-layer perceptron (MLP) serves as a core component in many deep CTR prediction models, but it has been widely shown that applying a vanilla MLP network alone is ineffective in learning complex feature interactions. As such, many two-stream models (e.g., Wide&Deep, DeepFM, and DCN) have recently been proposed, aiming to integrate two parallel sub-networks to learn feature interactions from two different views for enhanced CTR prediction. In addition to one MLP stream that learns feature interactions implicitly, most of the existing research focuses on designing another stream to complement the MLP stream with explicitly enhanced feature interactions. Instead, this paper presents a simple two-stream feature interaction model, namely FinalMLP, which employs only MLPs in both streams yet achieves surprisingly strong performance. In contrast to sophisticated network design in each stream, our work enhances CTR modeling through a feature selection module, which produces differentiated feature inputs to two streams, and a group-wise bilinear fusion module, which effectively captures stream-level interactions across two streams. We show that FinalMLP achieves competitive or even better performance against many existing two-stream CTR models on four open benchmark datasets and also brings significant CTR improvements during an online A/B test in our industrial news recommender system. We envision that the simple yet effective FinalMLP model could serve as a new strong baseline for future development of two-stream CTR models. Our source code will be available at MindSpore/models and FuxiCTR/model_zoo.

Two-Stream Network One-Class Classification Model for Defect Inspections.

Defect inspection is important to ensure consistent quality and efficiency in industrial manufacturing. Recently, machine vision systems integrating artificial intelligence (AI)-based inspection algorithms have exhibited promising performance in various applications, but practically, they often suffer from data imbalance. This paper proposes a defect inspection method using a one-class classification (OCC) model to deal with imbalanced datasets. A two-stream network architecture consisting of global and local feature extractor networks is presented, which can alleviate the representation collapse problem of OCC. By combining an object-oriented invariant feature vector with a training-data-oriented local feature vector, the proposed two-stream network model prevents the decision boundary from collapsing to the training dataset and obtains an appropriate decision boundary. The performance of the proposed model is demonstrated in the practical application of automotive-airbag bracket-welding defect inspection. The effects of the classification layer and two-stream network architecture on the overall inspection accuracy were clarified by using image samples collected in a controlled laboratory environment and from a production site. The results are compared with those of a previous classification model, demonstrating that the proposed model can improve the accuracy, precision, and F1 score by up to 8.19%, 10.74%, and 4.02%, respectively.

Recognition and location of marine animal sounds using two-stream ConvNet with attention

There are abundant resources and many endangered marine animals in the ocean. Using sound to effectively identify and locate them, and estimate their distribution area, has a very important role in the study of the complex diversity of marine animals (Hanny et al., 2013). We design a Two-Stream ConvNet with Attention (TSCA) model, which is a two-stream model combined with attention, in which one branch processes the temporal signal and the other branch processes the frequency domain signal; It makes good use of the characteristics of high time resolution of time domain signal and high recognition rate of frequency domain signal features of sound, and it realizes rapid localization and recognition of sound of marine species. The basic network architecture of the model is YOLO (You Only Look Once) (Joseph et al., 2016). A new loss function focal loss is constructed to strengthen the impact on the tail class of the sample, overcome the problem of data imbalance and avoid over fitting. At the same time, the attention module is constructed to focus on more detailed sound features, so as to improve the noise resistance of the model and achieve high-precision marine species identification and location. In The Watkins Marine Mammal Sound Database, the recognition rate of the algorithm reached 92.04% and the positioning accuracy reached 78.4%.The experimental results show that the algorithm has good robustness, high recognition accuracy and positioning accuracy.

Deep learning-based middle cerebral artery blood flow abnormality detection using flow velocity waveform derived from transcranial Doppler ultrasound

Since the brain is unlike any other organ in that it cannot store energy and has a high metabolic demand, constant blood flow is essential for healthy brain function. The maximum flow velocity waveform that is produced by transcranial doppler echo ultrasonography has different qualities for a healthy subject and a critically ill patient with conditions such as intraparenchymal or subarachnoid hemorrhage, hydrocephalus, or traumatic brain injury. Depending on the degree of the injury, the symptoms of traumatic brain damage can present themselves right away or not until days or weeks later. To aid in the early and accurate detection of patients with severe brain conditions, a classification system is proposedthat can distinguish between healthy control and patient utilizing the maximum flow velocity waveform derived from Transcranial doppler ultrasound. In this research, we manually labelled the data to remove mediocre and corrupted signals and pre-processed low-quality signals into high-quality ones using a Cycle Generative Adversarial Network (CycleGAN). This study proposes a two-stream deep learning model, DopplerNet2+, based on a Self-organized Operational Neural Network (Self-ONN), which achieves an overall accuracy, precision, recall, sensitivity, f1 score, and specificity of 99.45%, 99.45%, 99.45%, and 99.37% for the classification issue. DopplerNet2+ has a better area under the curve (AUC) of 1.00 and a better Kolmogorov-Smirnov (KS) statistic of 0.996 at the 0.812 thresholds than 11 other Self-ONN models trained with different inputs. The results show that the proposed models can successfully carry out the targeted classification task.

Speech Emotion Recognition Based on Two-Stream Deep Learning Model Using Korean Audio Information

Identifying a person’s emotions is an important element in communication. In particular, voice is a means of communication for easily and naturally expressing emotions. Speech emotion recognition technology is a crucial component of human–computer interaction (HCI), in which accurately identifying emotions is key. Therefore, this study presents a two-stream-based emotion recognition model based on bidirectional long short-term memory (Bi-LSTM) and convolutional neural networks (CNNs) using a Korean speech emotion database, and the performance is comparatively analyzed. The data used in the experiment were obtained from the Korean speech emotion recognition database built by Chosun University. Two deep learning models, Bi-LSTM and YAMNet, which is a CNN-based transfer learning model, were connected in a two-stream architecture to design an emotion recognition model. Various speech feature extraction methods and deep learning models were compared in terms of performance. Consequently, the speech emotion recognition performance of Bi-LSTM and YAMNet was 90.38% and 94.91%, respectively. However, the performance of the two-stream model was 96%, which was a minimum of 1.09% and up to 5.62% improved compared with a single model.

A Weakly Supervised Learning Framework for Salient Object Detection via Hybrid Labels

Fully-supervised salient object detection (SOD) methods have made great progress, but such methods often rely on a large number of pixel-level annotations, which are time-consuming and labour-intensive. In this paper, we focus on a new weakly-supervised SOD task under hybrid labels, where the supervision labels include a large number of coarse labels generated by the traditional unsupervised method and a small number of real labels. To address the issues of label noise and quantity imbalance in this task, we design a new pipeline framework with three sophisticated training strategies. In terms of model framework, we decouple the task into label refinement sub-task and salient object detection sub-task, which cooperate with each other and train alternately. Specifically, the R-Net is designed as a two-stream encoder-decoder model equipped with Blender with Guidance and Aggregation Mechanisms (BGA), aiming to rectify the coarse labels for more reliable pseudo-labels, while the S-Net is a replaceable SOD network supervised by the pseudo labels generated by the current R-Net. Note that, we only need to use the trained S-Net for testing. Moreover, in order to guarantee the effectiveness and efficiency of network training, we design three training strategies, including alternate iteration mechanism, group-wise incremental mechanism, and credibility verification mechanism. Experiments on five SOD benchmarks show that our method achieves competitive performance against weakly-supervised/unsupervised methods both qualitatively and quantitatively. The code and results can be found from the link of <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://rmcong.github.io/proj_Hybrid-Label-SOD.html</uri> .

Dual Stream Conditional Generative Adversarial Network Fusion for Video Abnormal Behavior Detection

Deep learning has been successfully applied to video anomaly detection. However, the way that deep network learn spatio-temporal features autonomously will ignore the specificity of different pattern features. Therefore, this paper focuses on how to efficiently learn deep appearance feature, introduces the idea of learning appearance information by predicting future frame, and proposes dual stream conditional generative adversarial network fusion for video abnormal behavior detection. The video frame and its corresponding optical flow image are transferred to the conditional generative adversarial network to learn the motion feature representation. In addition, inputting the video frame and its corresponding future frame to the network to generate the appearance representation complementary to motion feature. The model is only trained with normal events, therefore it is not able to generate abnormal events accurately. During the test, for the foreground moving targets, the images generated by the model are compared with the ground truth to obtain a two-stream anomaly probability distribution model based on the mean square error used to achieve the purpose of region anomaly detection. Experiments on the public datasets show that the proposed method can effectively detect and locate abnormal behaviors in the video.

View-Invariant Center-of-Pressure Metrics Estimation With Monocular RGB Camera

Center of pressure (CoP) metrics, including CoP path length and sway area, have been used as gold standard measurements of postural and balance control in biomechanical studies. A recent study of computer-vision-based CoP metrics estimation from 3D body landmark sequences offers a more portable and comprehensive solution than conventional force plate methods to obtain these important metrics for real-time evaluation of balance control. However, obtaining accurate 3D body landmarks requires a calibrated motion capture system or on-body markers, which involves lengthy data collection and processing time and limits their implementation in home and clinical environments. Existing methods that instead use 2D body landmarks fail to adapt to different camera positions. To overcome these challenges, we propose a view-invariant deep learning framework for video-level CoP metrics estimation, including CoP path length and sway area, using pose dimension lifting and graph convolutional network (GCN). This work is the first step toward obtaining gold-standard CoP metrics with an accessible, monocular RGB camera. We propose to use a dimension lifting convolutional neural network (CNN) to obtain view-invariant 3D body landmark features from 2D body landmarks. We also propose a two-stream regression model using GCN and discrete cosine transform (DCT) for a robust CoP metrics estimation. To facilitate the line of research, we release a novel multi-view body landmark dataset containing 2D body landmarks of a wide variety of action patterns from four different camera views with synchronized CoP labels and corresponding 3D body landmarks, which enables cross-view evaluation with different camera angles. We subsequently validate the proposed method through a cross-dataset training by training the dimension lifting model on an existing balance dataset and evaluating the CoP metrics estimation on the multi-view body landmark dataset. The experiments validate that our framework achieves state-of-the-art accuracy for both CoP path length and CoP sway area using a monocular RGB camera input for unseen views.

Spatio-Temporal Human Action Recognition Model using Deep Learning Techniques

Two-stream human recognition achieved great success in the development of video action recognition using deep learning. Recently many studies have shown that two-stream action recognition is a powerful feature extractor. The main contribution in this work is to develop a two-stream model based on spatial and temporal networks using convolutional neural networks with a convolution long-short term memory. The two-stream model with ImageNet pre-trained weights is used to retrieve spatial and temporal features. Output feature maps of the two-stream model are fused using sum fusion and fed as input to convolutional long-short-term memory. SoftMax function is used to get the final classification score. To avoid overfitting, we have adopted the data augmentation techniques. Finally, we demonstrated that the proposed model performs well in comparison to state-of-the-art two-stream models with an accuracy of 96.1% on UCF 101 dataset and 70.9% accuracy on the HMDB dataset.

A Lightweight Two-Stream Model for Driver Emotion Recognition

In recent years, more and more attention has been paid to the traffic safety of drivers. Studies show that the motor vehicle speed will affect the incidence of traffic accidents and traffic accident casualty rate. In this paper, machine learning vision technology is used for real-time recognition of driver emotions, and a lightweight SIC-NET+RC-NET dual-stream recognition model is designed to cope with different scenarios of vehicle speed during driving. SIC-NET is a lightweight CNN network model, which greatly reduces the number of model parameters and computational power requirements, sacrifices part of accuracy, and greatly improves the real-time performance of model recognition. To improve the residual convolutional network model of inception blocks, RC-NET nested multiple micro-convolutional neural networks are used to identity mapping between different inception blocks to enhance the network training effect, expand network width and convolutional depth, and alleviate network degradation. This article stipulates that when the vehicle speed is greater than 60km/h, SIC-NET is used. When the vehicle speed is less than or equal to 60km/h, SIC-NET+RC-NRT is used. The FER2013 dataset was used for testing and validation in this paper. When SIC-NET is used, the model has better real-time performance and can provide a more timely warning when the vehicle speed is higher. When SIC-NET+RC-NET is used, the accuracy of the model in the FER2013 dataset is 69.65%, and the model can better cope with the scenario of a low vehicle speed. The experiment shows that the lightweight dual-flow model has good robustness and identification effect, which solves the problem of using the same model for different vehicle speeds and can effectively ensure the safety of the driver’s life and property.