Dual-stream Network Research Articles

Noise Resistible Network for Unsupervised Domain Adaptation on Person Re-Identification

Unsupervised domain adaptation on person re-identification (re-ID), which adapts the model trained on source dataset to the target dataset, has drawn increasing attention over the past few years. It is more practical than the traditional supervised methods when applied in the real-world scenarios since they require a huge number of manual annotations in a specific domain, which is unrealistic and even under personal privacy concerns. Currently, pseudo label-based method is one of the most promising solutions in this area. However, in such methods, pseudo label noise is ignored and remains a huge challenge hindering further performance improvements. To solve this problem, this paper proposes a novel unsupervised domain adaptation re-ID framework named Noise Resistible Network (NRNet), which mainly consists of two dual-stream networks. For one thing, during pseudo label generation, NRNet utilizes one dual-stream network, denoted as clustering network, to generate discriminative features in the unseen domain for further clustering, reducing the pseudo label noise. For another, to avoid the problem of close loop noise amplification in conventional methods, the other dual-stream network named temporally average network is constructed outside the clustering loop to learn how to identify the images of the same person. In addition, two dual-stream networks are designed with a guiding mechanism, which allows the shallow network to learn more representative feature embedding from the deep network. Extensive experimental results on two widely-used benchmark datasets, i.e., Market-1501 and DukeMTMC-reID demonstrate that our proposed NRNet outperforms the state-of-the-art methods.

Testing the Feasibility of a Multi-Model Fusion Method for Monitoring the Action of Rehabilitating Stroke Patients in Care Management

Post-stroke care encounters challenges, including high cost, lack of professionals, and insufficient rehabilitation state evaluation. Computer technology can alleviate these issues, as it allows health care professionals (HCP) to quantify the workload and thus enhance rehabilitation care quality. In this paper, a novel multi-model fusion method, terms as pose dual-stream network (PDSN), is devised, aiming to test the feasibility of monitoring the training actions of rehabilitating stroke patients in care management. In particular, this deep-learning-based algorithm combines human pose estimation and dual-stream networks in an innovative way. We utilize an improved OpenPose to estimate human pose from videos obtained by the low-cost monocular camera. In dual-stream networks, the spatial and motion streams are flexibly integrated. The spatial stream network combines the Gated Recurrent Unit (GRU) and attention mechanism to extract spatiotemporal data, while the motion stream network is composed of improved multi-layer 1D Convolutional Neural Networks (CNN), which enhanced by causal and dilated convolution skillfully. Additionally, an adaptive weight fusion strategy is used to fuse the two networks for the final action classification. Results show high accuracy on two public datasets and a dataset created by us, which validate the superiority and feasibility of our method.

Boundary-Aware Dual-Stream Network for VHR Remote Sensing Images Semantic Segmentation

Semantic segmentation for very-high-resolution remote sensing images has been a research hotspot in the field of remote sensing image analysis. However, most existing methods still suffer from a challenge that object boundaries cannot be finely recovered. To tackle the problem, we develop a dual-stream network based on the U-Net architecture, Instead of the traditional skip connections, a boundary attention module is proposed to introduce the boundary information from the EDN module to the SSN module. Experiments on ISPRS Potsdam and Vaihingen datasets show the effectiveness of the proposed network, especially in man-made objects with distinct boundaries.

Visual Odometry Based on Improved Dual-Stream Network Structure

Visual Odometry Based on Improved Dual-Stream Network Structure

Expression-Aware Face Reconstruction via a Dual-Stream Network

Recently, 3D face reconstruction from a single image has achieved promising progress by adopting the 3D Morphable Model (3DMM). However, face images taken in-the-wild usually involve expressions with a large range of variety. This poses difficulty to use 3DMM to represent such various facial expressions owing to the limited expressive ability of its linear model, thereby resulting in distortion and ambiguity in local facial regions. To tackle this problem, we present a novel dual-stream network composed of a geometry stream and a texture stream to deal with expression variations. Specifically, in the geometry stream, we propose novel Attribute Spatial Maps (ASMs) to decompose a face into the identity and expression attributes and then separately record the essential spatial information of the two facial attributes in the 2D image space. This avoids the interaction between the two attributes, thus preserving the identity information and further improving the ability of coping with expression variations. In the texture stream, we propose to generate facial appearance with realistic texture and canonical layout by our Semantic Region Stylization Mechanism (SRSM), that transfers the style from an input face to a 3DMM albedo map in a region-adaptive manner. Moreover, we also propose a Shared Semantic Region Prediction Module (SSRPM) to explore the common correspondence of semantic regions between the above two face texture representations. Both quantitative and qualitative evaluations on public datasets demonstrate the effectiveness of our approach in face reconstruction under expression variations.

CNN-GCN-based dual-stream network for scene classification of remote sensing images

CNN-GCN-based dual-stream network for scene classification of remote sensing images

Study on the Identification Method of Human Upper Limb Flag Movements Based on Inception-ResNet Double Stream Network

To investigate the recognition effect of flag motions based on 9 axis sensors, starting with deep learning methods, this paper proposes a framework for feature extraction and recognition of flag movements recognition by employing an improved Inception-ResNet dual-stream network. In traditional signal recognition studies, Support Vector Machine (SVM), Random Forest (RF) and One Dimensional Convolutional Neural Network (CNN) are usually used to extract signals' features. In the meanwhile, the time series data sets such as flag movements are usually the standard data set after processing. Therefore, there are usually some limitations in traditional systems. First, in actual environment, there exists a lack of effective segmentation detection method for the samples of long time series, resulting in the deviation of the data set in the recognition process. Second, the One-Dimensional CNN framework used and the machine learning frameworks used in previous studies are difficult to process large quantities of data with too high computational memory. Based on these problems, this study proposes a signal change point detection model based on the diversity factory function in the signal segmentation and detection stage, miniaturizes the convolution kernel in the original CNN by using the Inception-ResNet(I-R) dual-flow network separable convolution method, and proposes a CrossEntropy-Logistic(C-L) joint classification loss function. Through conducting comparative experiments, it is found that the average calculation parameter of the CNN framework based on the Inception-ResNet model is 2.7×10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">7</sup> , which is approximately 37% lower than the number of 3.7×10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">7</sup> in the original CNN model. Finally, the recognition rate between C-L joint loss function and other models such as Multi-Layer Perceptron and Ensemble Learning in recent years are compared. Compared with Ensemble Learning-CrossEntropy (ELC) model, the C-L joint loss function can improve the recognition rate by nearly 5% according to the results of flag movements identification measured by several classification models.

A Local-Global Dual-Stream Network for Building Extraction From Very-High-Resolution Remote Sensing Images.

Buildings constitute one of the most important landscapes in remote sensing (RS) images and have been broadly analyzed in a wide range of applications from urban planning to other socioeconomic studies. As very-high-resolution (VHR) RS imagery becomes more accessible, the current building extraction methods are confronted with the challenges of the diverse appearances, various scales, and complicated structures of buildings in complex scenes. With the development of context-aware deep learning methods, it has been proven by numerous works that capturing contextual information can offer spatial relation cues for robust recognition and detection of the objects. In this article, we propose a novel local-global dual-stream network (DS-Net) that adaptively captures local and long-range information for the accurate mapping of building rooftops in VHR RS images. The local branch and the global branch of DS-Net work in a complementary manner to each other with different fields of view on the input image. Through a well-defined dual-stream architecture, DS-Net learns hierarchical representations for both the local and global branches, and a deep feature sharing strategy is further developed to enforce more collaborative integration of the two branches. Extensive experiments were carried out to verify the effectiveness of our model on three widely used VHR RS data sets: the Massachusetts buildings data set, the Inria Aerial Image Labeling data set, and the DeepGlobe Building Detection Challenge data set. Empirically, the proposed DS-Net achieves competitive or superior performance compared with the current state-of-the-art methods in terms of quantitative measures and visual evaluations.

Explicit-implicit dual stream network for image quality assessment

Communications industry has remarkably changed with the development of fifth-generation cellular networks. Image, as an indispensable component of communication, has attracted wide attention. Thus, finding a suitable approach to assess image quality is important. Therefore, we propose a deep learning model for image quality assessment (IQA) based on explicit-implicit dual stream network. We use frequency domain features of kurtosis based on wavelet transform to represent explicit features and spatial features extracted by convolutional neural network (CNN) to represent implicit features. Thus, we constructed an explicit-implicit (EI) parallel deep learning model, namely, EI-IQA model. The EI-IQA model is based on the VGGNet that extracts the spatial domain features. On this basis, the number of network layers of VGGNet is reduced by adding the parallel wavelet kurtosis value frequency domain features. Thus, the training parameters and the sample requirements decline. We verified, by cross-validation of different databases, that the wavelet kurtosis feature fusion method based on deep learning has a more complete feature extraction effect and a better generalisation ability. Thus, the method can simulate the human visual perception system better, and subjective feelings become closer to the human eye. The source code about the proposed EI-IQA model is available on github https://github.com/jacob6/EI-IQA.

ELNet:Automatic classification and segmentation for esophageal lesions using convolutional neural network.

Automatic and accurate esophageal lesion classification and segmentation is of great significance to clinically estimate the lesion statuses of the esophageal diseases and make suitable diagnostic schemes. Due to individual variations and visual similarities of lesions in shapes, colors, and textures, current clinical methods remain subject to potential high-risk and time-consumption issues. In this paper, we propose an Esophageal Lesion Network (ELNet) for automatic esophageal lesion classification and segmentation using deep convolutional neural networks (DCNNs). The underlying method automatically integrates dual-view contextual lesion information to extract global features and local features for esophageal lesion classification and lesion-specific segmentation network is proposed for automatic esophageal lesion annotation at pixel level. For the established clinical large-scale database of 1051 white-light endoscopic images, ten-fold cross-validation is used in method validation. Experiment results show that the proposed framework achieves classification with sensitivity of 0.9034, specificity of 0.9718, and accuracy of 0.9628, and the segmentation with sensitivity of 0.8018, specificity of 0.9655, and accuracy of 0.9462. All of these indicate that our method enables an efficient, accurate, and reliable esophageal lesion diagnosis in clinics.

Learning adaptive geometry for unsupervised domain adaptation

Unsupervised domain adaptation is an effective approach to solve the problem of dataset bias. However, most existing unsupervised domain adaptation methods assume that the geometry structures of data distributions are similar in the source and target domains. This assumption is invalid in many practical applications, because the training and test datasets usually differ in the variability modes and/or variation degrees. This paper handles the problem of inconsistent geometries by aligning both data representations and geometries. To overcome the lack of target labels in aligning geometries, this paper proposes learning the adaptive geometry that is derived from the domain-shared label space. Source and target geometries are aligned by constraining them with the unified criteria of the adaptive geometry. Combining the adaptive geometry learning and adversarial learning techniques, we develop a geometry-aware dual-stream network to learn the geometry-aligned representations. Experimental results show that our method achieves good performance on cross-dataset recognition tasks.

Skeleton-Based Pedestrian Abnormal Behavior Detection with Spatio-Temporal Model in Public Places

In computer vision, there is growing interest in the recognition of pedestrian abnormal behaviors. The abnormal behavior of a person could be the sign of some dangerous activities. However, it’s still challenging to extract the discriminative spatial and temporal features effectively faced with video data. In this paper, we propose skeleton-based pedestrian abnormal behavior detection models. The base model is consisting of ResNet as a spatial feature extractor, LSTM as a global temporal feature extractor, and the ResNet network that use the dual-stream network to extract local temporal features. The proposed model is an improvement of all ResNet into Conv1x1_ResNet, and added a layer of Conv1x1_ResNet after dual-stream Conv1x1_ResNet to extract more accurate global space features. The proposed model achieved the highest accuracy of 89.29%, and the averaged get batch time is 0.3399 ms. The base model achieved 88.12% accuracy, and the averaged get batch time is 0.3174 ms less than the time taken by other models. Both models reach speed of 80 frames/sec. Compared with the models made in previous work, the base model has the shortest training time, and the proposed model provides the highest accuracy in the field of pedestrian detection.

Learning latent geometric consistency for 6D object pose estimation in heavily cluttered scenes

6D object pose (3D rotation and translation) estimation from RGB-D image is an important and challenging task in computer vision and has been widely applied in a variety of applications such as robotic manipulation, autonomous driving, augmented reality etc. Prior works extract global feature or reason about local appearance from an individual frame, which neglect the spatial geometric relevance between two frames, limiting their performance for occluded or truncated objects in heavily cluttered scenes. In this paper, we present a dual-stream network for estimating 6D pose of a set of known objects from RGB-D images. Our novelty stands in contrast to prior work that learns latent geometric consistency in pairwise dense feature representations from multiple observations of the same objects in a self-supervised manner. We show in experiments that our method outperforms state-of-the-art approaches on 6D object pose estimation in two challenging datasets, YCB-Video and LineMOD.

Deep Dual-Stream Network with Scale Context Selection Attention Module for Semantic Segmentation

The fusion of multi-scale features has been an effective method to get state-of-the-art performance in semantic segmentation. In this work, we concentrate on two tricky problems—the intra-class inconsistency and the blur on the localization of object boundaries and tackle them by combining two separate multi-scale context features respectively. Specifically, we propose a dual-stream structure with the scale context selection attention module to enhance the capabilities for multi-scale processing, where one stream collects global-scale context and the other captures local-scale information. Meanwhile, the embedded scale context selection attention module in each stream can adaptively focus on different scale context information to get optimal scale features. Based on our dual-stream structure with attention modules, our network can efficiently make use of multi-scale context to generate more comprehensive and powerful features. Our experiments show that our dual-stream network with scale context selection attention module achieves promising performance on the PASCAL VOC 2012 and PASCAL-Person-Part datasets.

DELTA: A deep dual-stream network for multi-label image classification

Multi-label image classification problem is one of the most important and fundamental problems in computer vision. In an image with multiple labels, the objects usually locate at various positions with different scales and poses. Moreover, some labels are associated with the entire image instead of a small region. Therefore, both the global and local information are important for classification. To effectively extract and make full use of these information, in this paper, we present a novel deep Dual-stream nEtwork for the muLTi-lAbel image classification task, DELTA for short. As its name indicates, it is composed of two streams, i.e., the Multi-Instance network and the Global Priors network. The former is used to extract the multi-scale class-related local instances features by modeling the classification problem in a multi-instance learning framework. The latter is devised to capture the global priors from the input image as the global information. These two streams are fused by the final fusion layer. In this way, DELTA can extract and make full use of both the global and local information for classification. Extensive experiments on three benchmark datasets, i.e., PASCAL VOC 2007, PASCAL VOC 2012 and Microsoft COCO, demonstrate that DELTA significantly outperforms several state-of-the-art methods. Moreover, DELTA can automatically locate the key image patterns that trigger the labels.