Multi-stream Network Research Articles

Appearance-posture fusion network for distracted driving behavior recognition

In recent years, detection techniques using computer vision and deep learning have shown promise in assessing driver distraction. This paper proposes a fusion network that combines a Spatial-Temporal Graph Convolutional Network (ST-GCN) and a hybrid convolutional network to integrate multimodal input data for recognizing distracted driver behavior. Specifically, to address the limitations of the ST-GCN method in modeling long-distance joint interaction features and inadequate temporal feature extraction, we design the Spatially Symmetric Configuration Partitioning Graph Convolutional Network (SSCP-GCN) to model relative motion information of symmetric relationships between limbs. Specifically, we utilize densely connected blocks for processing multi-scale temporal information between consecutive frames, thereby enhancing the reuse of bottom features. Furthermore, the expression of important temporal information is augmented by the introduction of the channel attention mechanism. To tackle the problem that the Mixed Convolution (MC) combining 3D convolution with 2D convolution cannot extract higher-order timing information and has limitations in modeling global dependency relationships, we compensate for its inability to capture higher-order temporal semantic information using the Time Shift Module (TSM) without consuming additional computational resources. Additionally, the 3D Multi-Head Self-Attention mechanism (3D MHSA) is employed to integrate global spatial–temporal information of high-level features, avoiding the issue of model complexity proliferation caused by the deep stacking design of Convolutional Neural Networks (CNN). Lastly, we introduce a multistream network framework that integrates driver posture and appearance features to harness complementary advantages, enabling us to combine multimodal input features to achieve better model performance. Experimental results indicate that the accuracy of the network designed in this paper reaches 95.6% and 94.3% on ASU dataset and NTU-RGB+D dataset, respectively. The small size of the model offers the possibility for practical application of the algorithm.

Multi-stream network with key frame sampling for human action recognition

Multi-stream network with key frame sampling for human action recognition

A multi-stream network for retrosynthesis prediction

A multi-stream network for retrosynthesis prediction

Graph-Based Progressive Fusion Network for Multi-Modality Vehicle Re-Identification

Vehicle re-identification (Re-ID) is a critical task in intelligent transportation, aiming to match vehicle images of the same identity captured by non-overlapping cameras. However, it is difficult to achieve satisfactory results based on RGB images alone in darkness. Therefore, it is of great importance to consider multi-modality vehicle re-identification. Currently, the proposed works deal with different modality features through direct summation and fusion based on heat map, which however ignores the relationship between them. Meanwhile, there is a huge gap between the different modalities, which needs to be reduced. In this paper, to solve the above two problems, we propose a Graph-based Progressive Fusion Network (GPFNet) using a graph convolutional network to adaptively fuse multi-modality features in an end-to-end learning framework. GPFNet consists of a CNN feature extraction module (FEM), a GCN feature fusion module (FFM), and a loss function module (LFM). Firstly, in FEM, we employ a multi-stream network architecture to extract single-modality features and common-modality features and employ a random modality substitution module to extract mixed-modality features. Secondly, in FFM, we design an efficient graph structure to associate the features of different modalities and adopt a progressive two-stage strategy to fuse them. Finally, in LFM, we use GCN-aware multi-modality loss to constrain the features. For reducing modality differences and contributing better initial mixed-modality features to FFM, we propose random modality substitution as a data enhancement method for multi-modality datasets. Extensive experiments on multi-modality vehicle Re-ID datasets RGBN300 and RGBNT100 show that our model achieves state-of-the-art performance.

FPRnet: A lightweight multi-domain multi-stream network for complex horizontal oil-water two-phase flow pattern recognition

The complicated spatio-temporal structure of horizontal oil-water flows renders the identification of flow patterns exceedingly challenging, owing to the presence of interfacial waves and droplet entrainment. In this study, a multi-domain fusion feature (MFF) as flow pattern descriptors (FPD) and a lightweight multi-stream networks (LMN) as flow pattern classifier (FPC) are proposed to identify the flow patterns of horizontal oil-water two-phase flow. MFF can map time series into different domains and compactly encode them into higher dimensional expressions, thus displaying implicit flow pattern signals incorporating nonlinear, time-frequency, and probabilistic transition domain features. Finally, the comparison of LMN, variational and other networks is evaluated on the horizontal oil-water flows pattern identification task. Extensive comparison experiments show that the time-frequency feature exhibits optimal characterization for the D O/W flow pattern, the URP feature is most accurately characterized for the D W/O and D O/W flow patterns, and the MTF feature exhibits superior characterization for the remaining three flow patterns. The 98.40 % recognition accuracy demonstrates that MFF has a good representation of flow patterns and the proposed LMN outperforms existing classifiers in flow pattern recognition.

Progressive Moire Removal and Texture Complementation for Image Demoireing

Taking photos of digital screens often produces color-distorting moire patterns caused by inconsistency between the color filter array of cameras and the sub-pixel layout of screens, which severely degrades the quality of photos. Most existing demoireing methods employ multi-stream network architecture to simultaneously process the same moire image with different resolutions, but they neglect the complementarity among different resolutions. In this paper, we propose a novel moire removal model to address this issue. Unlike the existing multi-stream based approaches, in this model, we present a progressive texture complementation block to exploit the complementary information from different resolutions in order to progressively remove moire textures and restore image content. Additionally, we propose a residual moire removal block, in which the depthwise separable convolution is utilized to remove moire from image while reducing computation overhead. This block also includes a local color correction structure, which is used to correct color shifts presented in the moire images. Experimental results on two public datasets show that our method outperforms state-of-the-art methods. Besides, the quantity of parameters and FLOPs of our model are tens of times fewer than the off-the-shelf models. Furthermore, our network framework can adapt well to another low-level vision task, rain removal, in which our model also achieves state-of-the-art performance.

A novel federated deep learning scheme for glioma and its subtype classification.

Deep learning (DL) has shown promising results in molecular-based classification of glioma subtypes from MR images. DL requires a large number of training data for achieving good generalization performance. Since brain tumor datasets are usually small in size, combination of such datasets from different hospitals are needed. Data privacy issue from hospitals often poses a constraint on such a practice. Federated learning (FL) has gained much attention lately as it trains a central DL model without requiring data sharing from different hospitals. We propose a novel 3D FL scheme for glioma and its molecular subtype classification. In the scheme, a slice-based DL classifier, EtFedDyn, is exploited which is an extension of FedDyn, with the key differences on using focal loss cost function to tackle severe class imbalances in the datasets, and on multi-stream network to exploit MRIs in different modalities. By combining EtFedDyn with domain mapping as the pre-processing and 3D scan-based post-processing, the proposed scheme makes 3D brain scan-based classification on datasets from different dataset owners. To examine whether the FL scheme could replace the central learning (CL) one, we then compare the classification performance between the proposed FL and the corresponding CL schemes. Furthermore, detailed empirical-based analysis were also conducted to exam the effect of using domain mapping, 3D scan-based post-processing, different cost functions and different FL schemes. Experiments were done on two case studies: classification of glioma subtypes (IDH mutation and wild-type on TCGA and US datasets in case A) and glioma grades (high/low grade glioma HGG and LGG on MICCAI dataset in case B). The proposed FL scheme has obtained good performance on the test sets (85.46%, 75.56%) for IDH subtypes and (89.28%, 90.72%) for glioma LGG/HGG all averaged on five runs. Comparing with the corresponding CL scheme, the drop in test accuracy from the proposed FL scheme is small (-1.17%, -0.83%), indicating its good potential to replace the CL scheme. Furthermore, the empirically tests have shown that an increased classification test accuracy by applying: domain mapping (0.4%, 1.85%) in case A; focal loss function (1.66%, 3.25%) in case A and (1.19%, 1.85%) in case B; 3D post-processing (2.11%, 2.23%) in case A and (1.81%, 2.39%) in case B and EtFedDyn over FedAvg classifier (1.05%, 1.55%) in case A and (1.23%, 1.81%) in case B with fast convergence, which all contributed to the improvement of overall performance in the proposed FL scheme. The proposed FL scheme is shown to be effective in predicting glioma and its subtypes by using MR images from test sets, with great potential of replacing the conventional CL approaches for training deep networks. This could help hospitals to maintain their data privacy, while using a federated trained classifier with nearly similar performance as that from a centrally trained one. Further detailed experiments have shown that different parts in the proposed 3D FL scheme, such as domain mapping (make datasets more uniform) and post-processing (scan-based classification), are essential.

Skeleton-Based Multifeatures and Multistream Network for Real-Time Action Recognition

Action recognition is a hot topic in the field of computer vision. It has been widely used in human–computer/robot interaction, abnormal behavior monitoring, and medical assistive. Because of the excellent robustness of skeleton data, it has attracted many scholars to research skeleton-based action recognition. Most of the current skeleton-based action recognition methods suffer from the incomplete and poor generalization of the input features, inadequate feature extraction by the network model, and an imbalance between recognition accuracy and model size. We analyze the critical skeleton features for action recognition to solve these problems and propose a multifeatures and multistream network (MM-Net) for real-time action recognition. First, three pairs of features are proposed, which are the joint distance (JD) and JD velocity (JDV), joint angle (JA) and JA velocity (JAV), and fast-action joint position (FJP) and slow-action joint position (SJP). Second, an MM-Net is proposed by using a 1-D convolutional neural network (1DCNN) to reduce the number of parameters of the model and fully extract the three pairs of features. As a result, MM-Net achieves the highest accuracies on both JHMDB (86.5%) and SHREC (96.4% on coarse and 93.3% on fine datasets). In addition, MM-Net is applied to a human–robot interaction (HRI) platform, which proves the practicality of MM-Net.

Pose-Driven Realistic 2-D Motion Synthesis.

A realistic 2-D motion can be treated as a deforming process of an individual appearance texture driven by a sequence of human poses. In this article, we thereby propose to transform the 2-D motion synthesis into a pose conditioned realistic motion image generation task considering the promising performance of pose estimation technology and generative adversarial nets (GANs). However, the problem is that GAN is only suitable to do the region-aligned image translation task while motion synthesis involves a large number of spatial deformations. To avoid this drawback, we design a two-step and multistream network architecture. First, we train a special GAN to generate the body segment images with given poses in step-I. Then in step-II, we input the body segment images as well as the poses into the multistream network so that it only needs to generate the textures in each aligned body region. Besides, we provide a real face as another input of the network to improve the face details of the generated motion image. The synthesized results with realism and sharp details on four training sets demonstrate the effectiveness of the proposed model.

Automatic translation of sign language with multi-stream 3D CNN and generation of artificial depth maps

Sign languages play an essential role in the cognitive and social development of the deaf, consisting of a natural form of communication and being a symbol of identity and culture. However, hearing loss has a severe social impact due to an existing communication barrier, preventing access to essential services such as education and health. A bi-directional sign language translation may be the solution to bridging the communication gap between the deaf and the listener, completing a two-way communication cycle. Virtual personal assistants can benefit from this technology by extending how users interact with the intelligent system. With this idea, in this work we develop a multi-stream deep learning model to recognize signs of Brazilian (BSL), Indian (ISL), and Korean (KSL) Sign Languages. We combine different types of information for the classification task, using single-stream and multi-stream 3D Convolutional Neural Networks. In addition, considering the largest source of sign data globally – the internet – we propose a depth sensor-free classification method, with depth maps artificially generated through Generative Adversarial Networks. In order to consider the main parameters that encode sign languages, the final architecture is composed of a multi-stream network that receives the segmented hands, the faces, the distances and speeds of the points of articulation, and the RGB frames associated with artificial depth maps. Finally, we provide a visual explanation to understand which regions were important for model decision-making. The best models were obtained using the multi-stream network, presenting an accuracy of 0.91 ± 0.07, and f1-score of 0.90 ± 0.08 on publicly available BSL data set. The results suggest that the multi-stream network with artificially generated depth maps is suitable for the task of sign recognition in different languages.

DISNet: A sequential learning framework to handle occlusion in human action recognition with video acquisition sensors

Human action recognition (HAR) requires a clear line of sight for the video acquisition system to properly collect the data. However, the data acquisition is hampered due to presence of the obstructions in front of the action happening in real time environment. In recently reported HAR techniques, the widely used multi-stream sequential networks are having two major limitations. First, the performance of the existing algorithms degrades comprehensively in the presence of partial loss of action data due to obstruction. Second, the performance of the existing networks is restricted by not exploiting the dependency relationship between different streams of a multi-stream network. To handle these problems, a novel double input sequential network (DISNet) is proposed which takes care the HAR algorithm in the presence of partial loss of action data. The DISNet, which learns inter-stream information, is jointly trained on the normal data and the artificially created obstructed data of a single video to provide immunity to the HAR network against obstructions. The proposed DISNet is evaluated on publicly available datasets like KTH, UCF-sports, JHMDB and UCF101, which provides performance accuracy of 83.57%, 82.27%, 50.24% and 54.96% respectively.

A Multi-Stream Sequence Learning Framework for Human Interaction Recognition

Human interaction recognition (HIR) is challenging due to multiple humans’ involvement and their mutual interaction in a single frame, generated from their movements. Mainstream literature is based on three-dimensional (3-D) convolutional neural networks (CNNs), processing only visual frames, where human joints data play a vital role in accurate interaction recognition. Therefore, this article proposes a multistream network for HIR that intelligently learns from skeletons’ key points and spatiotemporal visual representations. The first stream localises the joints of the human body using a pose estimation model and transmits them to a 1-D CNN and bidirectional long short-term memory to efficiently extract the features of the dynamic movements of each human skeleton. The second stream feeds the series of visual frames to a 3-D convolutional neural network to extract the discriminative spatiotemporal features. Finally, the outputs of both streams are integrated via fully connected layers that precisely classify the ongoing interactions between humans. To validate the performance of the proposed network, we conducted a comprehensive set of experiments on two benchmark datasets, UT-interaction and TV human interaction, and found 1.15% and 10.0% improvement in the accuracy.

End-to-end driving model based on deep learning and attention mechanism

End-to-end deep learning has gained considerable interests in autonomous driving vehicles. End-to-end autonomous driving uses the deep convolutional neural network to establish input-to-output mapping. However, existing end-to-end driving models only predict steering angle with front-facing camera data and poorly extract spatial-temporal information. Based on deep learning and attention mechanism, we propose an end-to-end driving model which combines the multi-stream attention module with the multi-stream network. As a multimodal multitask model, the proposed end-to-end driving model not only fully extracts spatial-temporal information from multimodality, but also adopts the multitask learning method with hard parameter sharing to predict the steering angle and speed. Furthermore, the proposed multi-stream attention module predicts the attention weights of streams based on the multimodal feature fusion, which encourages the proposed end-to-end driving model to pay attention to streams that positively impact the prediction result. We demonstrate the efficiency of the proposed driving model on the public Udacity dataset compared to existing models. Experimental results show that the proposed driving model has better performances than other existing methods.

Pose-Guided Graph Convolutional Networks for Skeleton-Based Action Recognition

Graph convolutional networks (GCN), which can model the human body skeletons as spatial and temporal graphs, have shown remarkable potential in skeleton-based action recognition. However, in the existing GCN-based methods, graph-structured representation of the human skeleton makes it difficult to be fused with other modalities, especially in the early stages. This may limit their scalability and performance in action recognition tasks. In addition, the pose information, which naturally contains informative and discriminative clues for action recognition, is rarely explored together with skeleton data in existing methods. In this work, we proposed pose-guided GCN (PG-GCN), a multi-modal framework for high-performance human action recognition. In particular, a multi-stream network is constructed to simultaneously explore the robust features from both the pose and skeleton data, while a dynamic attention module is designed for early-stage feature fusion. The core idea of this module is to utilize a trainable graph to aggregate features from the skeleton stream with that of the pose stream, which leads to a network with more robust feature representation ability. Extensive experiments show that the proposed PG-GCN can achieve state-of-the-art performance on the NTU RGB+D 60 and NTU RGB+D 120 datasets.

Multi-Stream Deep Neural Network for Diabetic Retinopathy Severity Classification Under a Boosting Framework

Diabetic Retinopathy (DR) is an eye disorder in patients with diabetes. Detection of DR presence and its complications using fundus images at an early stage helps prevent its progression to the advanced levels. In the recent years, several well-designed Convolutional Neural Networks (CNN) have been proposed to detect the presence of DR with the help of publicly available datasets. However, these existing CNN-based classifiers focus on utilizing different architectural settings to improve the performance of detection task only i.e. presence or absence of DR. The further classification of the severity and type of the disease, however, remains a non-trivial task. To this end, we propose a multi-stream ensemble deep network to classify diabetic retinopathy severity. The proposed approach takes advantages of the deep networks and principal component analysis (PCA) to learn inter-class and intra-class variations from the raw image features. Ensemble machine learning classifiers are then applied to achieve high classification accuracy and robust performance on the obtained deep features. Specifically, a multi-stream network is made using pre-trained deep learning architectures i.e. ResNet-50 and DenseNet-121 to serve as the main feature extractors. Further application of PCA reduces the dimensionality of features and effectively separates the variation space of inter-class and intra-class images. Finally, an ensemble machine learning classifier using AdaBoost and random forest algorithms is built to further improve classification accuracy. The proposed approach has been compared with multiple conventional CNN-based approaches on Messidor-2 (two categories) and EyePACS (two, five categories) datasets. The experiment results show that our proposed approach achieves superior performance (upto 95.58% accuracy) and can be considered a promising method for automatic diabetic retinopathy detection.

Hidden Markov Model-Based Video Recognition for Sports

In this paper, we conduct an in-depth study and analysis of sports video recognition by improved hidden Markov model. The feature module is a complex gesture recognition module based on hidden Markov model gesture features, which applies the hidden Markov model features to gesture recognition and performs the recognition of complex gestures made by combining simple gestures based on simple gesture recognition. The combination of the two modules forms the overall technology of this paper, which can be applied to many scenarios, including some special scenarios with high-security levels that require real-time feedback and some public indoor scenarios, which can achieve different prevention and services for different age groups. With the increase of the depth of the feature extraction network, the experimental effect is enhanced; however, the two-dimensional convolutional neural network loses temporal information when extracting features, so the three-dimensional convolutional network is used in this paper to extract features from the video in time and space. Multiple binary classifications of the extracted features are performed to achieve the goal of multilabel classification. A multistream residual neural network is used to extract features from video data of three modalities, and the extracted feature vectors are fed into the attention mechanism network, then, the more critical information for video recognition is selected from a large amount of spatiotemporal information, further learning the temporal dependencies existing between consecutive video frames, and finally fusing the multistream network outputs to obtain the final prediction category. By training and optimizing the model in an end-to-end manner, recognition accuracies of 92.7% and 64.4% are achieved on the dataset, respectively.

Multi‐stream densely connected network for semantic segmentation

Semantic segmentation is a challenging task in computer vision which is widely used in autonomous driving and scene understanding. State-of-the-art semantic segmentation networks, like DeepLab and PSPNet, make full use of multiple feature information to improve spatial resolution. However, the feature resolution in the scale-axis is not dense enough for practical applications. To tackle this problem, a multi-stream network is designed with atrous convolutional layers at multiple rates to capture objects and context at multiple scales. Furthermore, intra-connections and inter-connections are designed to fuse multi-scale features densely which produce a feature pyramid with much larger scale diversity and larger receptive field by involving small quantity of computation. The proposed module can be easily used in other methods and it helps to increase the performance. Compared with existing methods, the proposed network, called Multi-stream Densely Connected Network, reaches competitive results on ADE20K dataset, PASCAL VOC 2012 dataset, and Cityscapes dataset.

A Lightweight Hierarchical Model with Frame-Level Joints Adaptive Graph Convolution for Skeleton-Based Action Recognition

In skeleton-based human action recognition methods, human behaviours can be analysed through temporal and spatial changes in the human skeleton. Skeletons are not limited by clothing changes, lighting conditions, or complex backgrounds. This recognition method is robust and has aroused great interest; however, many existing studies used deep-layer networks with large numbers of required parameters to improve the model performance and thus lost the advantage of less computation of skeleton data. It is difficult to deploy previously established models to real-life applications based on low-cost embedded devices. To obtain a model with fewer parameters and a higher accuracy, this study designed a lightweight frame-level joints adaptive graph convolutional network (FLAGCN) model to solve skeleton-based action recognition tasks. Compared with the classical 2s-AGCN model, the new model obtained a higher precision with 1/8 of the parameters and 1/9 of the floating-point operations (FLOPs). Our proposed network characterises three main improvements. First, a previous feature-fusion method replaces the multistream network and reduces the number of required parameters. Second, at the spatial level, two kinds of graph convolution methods capture different aspects of human action information. A frame-level graph convolution constructs a human topological structure for each data frame, whereas an adjacency graph convolution captures the characteristics of the adjacent joints. Third, the model proposed in this study hierarchically extracts different levels of action sequence features, making the model clear and easy to understand; further, it reduces the depth of the model and the number of parameters. A large number of experiments on the NTU RGB + D 60 and 120 data sets show that this method has the advantages of few required parameters, low computational costs, and fast speeds. It also has a simple structure and training process that make it easy to deploy in real-time recognition systems based on low-cost embedded devices.

Monocular Depth Estimation using Integrated Model with Multi-task Learning and Multi-stream

Scene understanding is a central problem in a field of computer vision. Depth estimation, in particular, is one of the important applications in scene understanding, robotics, and 3-D reconstruction. Estimating a dense depth map from a single image is receiving increased attention because a monocular camera is popular, small and suitable for a wide range of environments. In addition, both multi-task learning and multi-stream, which use unlabeled information, improve the monocular depth estimation efficiently. However, there are only a few networks optimized for both of them. Therefore, in this paper, we propose a monocular depth estimation task with a multi-task and multistream network architecture. Furthermore, the integrated network which we develop makes use of depth gradient information and can be applied to both supervised and unsupervised learning. In our experiments, we confirmed that our supervised learning architecture improves the accuracy of depth estimation by 0.13 m on average. Additionally, the experimental result on unsupervised learning found that it improved structure-from-motion performance.

Omnidirectional Image Quality Assessment by Distortion Discrimination Assisted Multi-Stream Network

Omnidirectional image (OI) quality assessment is crucial to facilitate the development of virtual reality (VR) related technology. In this work, a distortion discrimination assisted multi-stream network is proposed for OI quality assessment. The multi-stream architecture is constructed by generating the viewport images received by the retina at one point to simulate the characteristics of humans perceiving VR contents. Additionally, the strategy of generating several viewport image sets from one OI is proposed for data augmentation. Furthermore, the facts that the human brain has the ability for both quality assessment and distortion type distinguishment, and the process of human brain handling two tasks exists information interaction inspire us to employ an auxiliary distortion discrimination task to facilitate the quality assessment task learning. Extensive experiments conducted on two public OI databases demonstrate the superiority of the proposed method to both traditional 2D quality metrics and existing metrics specific for OIs. Moreover, utilizing the assistant task is proven to be more effective than the single task learning for OI quality evaluation. Better generalization performance is also verified to be another valuable trait of the proposed method.