Multi-stream Fusion Research Articles

QualityNet: A multi-stream fusion framework with spatial and channel attention for blind image quality assessment

This study introduces a novel Blind Image Quality Assessment (BIQA) approach leveraging a multi-stream spatial and channel attention model. Our method addresses challenges posed by diverse image content and distortions by integrating feature maps from two distinct backbones. Through spatial and channel attention mechanisms, our algorithm prioritizes regions of interest, enhancing its ability to capture crucial image details. Extensive evaluations on four benchmark datasets demonstrate superior performance compared to existing methods, closely aligning with human perceptual assessment. Our approach exhibits exceptional generalization capabilities on both authentic and synthetic distortion databases. Moreover, it demonstrates a distinctive focus on perceptual foreground information, enhancing its practical applicability. Thorough quantitative analyses underscore the algorithm’s superior performance, establishing its dominance over existing methods.

Multi-stream domain adversarial prototype network for integrated smart roller TBM main bearing fault diagnosis across various low rotating speeds

The well-being of tunnel boring machines (TBM) heavily relies on the health status of main bearings, significantly impacting both safety and operational efficiency. Investigating the operation monitoring and diagnosis algorithms of TBM main bearings in practical scenarios with limited fault data and real-time variations in low-speed heavy-load operating conditions poses a research challenge. To solve these issues, a novel domain-adversarial prototype network diagnostic algorithm with a multi-stream fusion feature encoder (MDAPN) based on smart roller monitoring data is proposed. In contrast to existing external vibration monitoring schemes for bearings, monitoring data from smart rollers can mitigate complex external interference and greatly shorten the transmission path of fault sources. The algorithm encompasses a multi-stream fusion feature encoder, domain discriminator, and prototype network classifier. Specifically, the multi-stream fusion feature encoder adopts the two-stream convolutional network to deeply extract the axial-radial fused features of rollers, incorporating shallow statistical feature information. Domain-invariant features are generated based on domain adversarial learning strategies, and the prototype network classifier reduces dependence on target domain samples. An integrated smart roller main bearing fault simulation testbed was built, conducting 6 sets of cross-domain experiments. The proposed method reached an average accuracy of 98.41 % under 5-shot, surpassing the baseline method by 6.57 % at least. This validates that the MDAPN based on smart roller state exhibits excellent fault recognition performance for the main bearing under varying operational conditions with scarce available samples.

Behavioral Recognition of Skeletal Data Based on Targeted Dual Fusion Strategy

The deployment of multi-stream fusion strategy on behavioral recognition from skeletal data can extract complementary features from different information streams and improve the recognition accuracy, but suffers from high model complexity and a large number of parameters. Besides, existing multi-stream methods using a fixed adjacency matrix homogenizes the model’s discrimination process across diverse actions, causing reduction of the actual lift for the multi-stream model. Finally, attention mechanisms are commonly applied to the multi-dimensional features, including spatial, temporal and channel dimensions. But their attention scores are typically fused in a concatenated manner, leading to the ignorance of the interrelation between joints in complex actions. To alleviate these issues, the Front-Rear dual Fusion Graph Convolutional Network (FRF-GCN) is proposed to provide a lightweight model based on skeletal data. Targeted adjacency matrices are also designed for different front fusion streams, allowing the model to focus on actions of varying magnitudes. Simultaneously, the mechanism of Spatial-Temporal-Channel Parallel Attention (STC-P), which processes attention in parallel and places greater emphasis on useful information, is proposed to further improve model’s performance. FRF-GCN demonstrates significant competitiveness compared to the current state-of-the-art methods on the NTU RGB+D, NTU RGB+D 120 and Kinetics-Skeleton 400 datasets. Our code is available at: https://github.com/sunbeam-kkt/FRF-GCN-master.

A novel multi-stream fusion network for underwater image enhancement

A novel multi-stream fusion network for underwater image enhancement

Multi-stream fusion network for continuous gesture recognition based on sEMG

Multi-stream fusion network for continuous gesture recognition based on sEMG

Multi-stream fusion network for continuous gesture recognition based on sEMG

Multi-stream fusion network for continuous gesture recognition based on sEMG

Multi-Stream Fusion Network for Skeleton-Based Construction Worker Action Recognition.

The global concern regarding the monitoring of construction workers' activities necessitates an efficient means of continuous monitoring for timely action recognition at construction sites. This paper introduces a novel approach-the multi-scale graph strategy-to enhance feature extraction in complex networks. At the core of this strategy lies the multi-feature fusion network (MF-Net), which employs multiple scale graphs in distinct network streams to capture both local and global features of crucial joints. This approach extends beyond local relationships to encompass broader connections, including those between the head and foot, as well as interactions like those involving the head and neck. By integrating diverse scale graphs into distinct network streams, we effectively incorporate physically unrelated information, aiding in the extraction of vital local joint contour features. Furthermore, we introduce velocity and acceleration as temporal features, fusing them with spatial features to enhance informational efficacy and the model's performance. Finally, efficiency-enhancing measures, such as a bottleneck structure and a branch-wise attention block, are implemented to optimize computational resources while enhancing feature discriminability. The significance of this paper lies in improving the management model of the construction industry, ultimately aiming to enhance the health and work efficiency of workers.

Occlusion-Aware Graph Neural Networks for Skeleton Action Recognition

Due to its important applications in industry, the research of action recognition has attracted many attention. In particular, skeleton-based human action recognition methods have evolved to become more and more competitive. Nevertheless, occlusion is still a very challenging task in human action recognition as yet. Off-the-shelf works are usually based on complete skeleton data, few people consider action recognition in occlusion. For the sake of improving the recognition accuracy on occluded skeleton data, we put forward an occlusion-aware multi-stream fusion graph convolutional network (dubbed MSFGCN). Multiple streams are comprised in MSFGCN, and different occlusion cases can be disposed by different streams. Besides, in order to construct more discriminative features, multi-modal features were extracted simultaneously, such as joint coordinates, relative coordinates, small-scale temporal differences and large-scale temporal differences. In particular, it is the first time to take advantage of motion features at large and small scales at the same time, which helps to distinguish actions with different motion amplitudes. What's more, considering the different importance of different parts of the human body for different action recognition, the content adaptation operation is used to further optimize the recognition performance. Effective training strategies are also presented to further improve the performance of the model. Experimental results show that the proposed MSFGCN has a great advantage over other methods on occluded skeleton datasets. To showcase the effectiveness of different modules, extensive ablation experiments are performed on various skeleton action recognition datasets.

Multi-stream Global–Local Motion Fusion Network for skeleton-based action recognition

Skeleton-based action recognition is widely used in varied areas such as human–machine interaction and virtual reality. Benefit from the powerful expression ability to depict structural data, graph convolutional networks (GCNs) have been developed to address this task by modeling the human body skeletons as spatial–temporal graphs. However, most existing GCN-based methods usually ignore the diversity of the motion information between channels of the input feature. And how to enhance the ability to capture the long-term global correlations in spatial and temporal dimensions is also a fundamental challenge. In this work, we propose a novel multi-stream framework Global–Local Motion Fusion Network (GLMFN), which integrates the global and local motion information of spatial–temporal dimensions. Specifically, we design a grouping graph convolution module to enforce the ability to aggregate local spatial motion information. Besides, to learn richer semantic features, we propose two modules based on the self-attention operator: a spatial self-attention module and a temporal self-attention module. The former is responsible for extracting spatial long-term motion relationships, while the latter aims to capture temporal long-term motion relationships. Moreover, we present a multi-stream fusion strategy with a series of treatments for body joints to achieve a better recognition effect. To validate the efficacy and efficiency of the proposed model, we perform exhaustive experiments on the NTU-RGBD dataset and NTU-RGBD-120 dataset, and our method achieves the state-of-the-art performance on both datasets.

Prostate cancer segmentation from MRI by a multistream fusion encoder.

Targeted prostate biopsy guided by multiparametric magnetic resonance imaging (mpMRI) detects more clinically significant lesions than conventional systemic biopsy. Lesion segmentation is required for planning MRI-targeted biopsies. The requirement for integrating image features available in T2-weighted and diffusion-weighted images poses a challenge in prostate lesion segmentation frommpMRI. A flexible and efficient multistream fusion encoder is proposed in this work to facilitate the multiscale fusion of features from multiple imaging streams. A patch-based loss function is introduced to improve the accuracy in segmenting smalllesions. The proposed multistream encoder fuses features extracted in the three imaging streams at each layer of the network, thereby allowing improved feature maps to propagate downstream and benefit segmentation performance. The fusion is achieved through a spatial attention map generated by optimally weighting the contribution of the convolution outputs from each stream. This design provides flexibility for the network to highlight image modalities according to their relative influence on the segmentation performance. The encoder also performs multiscale integration by highlighting the input feature maps (low-level features) with the spatial attention maps generated from convolution outputs (high-level features). The Dice similarity coefficient (DSC), serving as a cost function, is less sensitive to incorrect segmentation for small lesions. We address this issue by introducing a patch-based loss function that provides an average of the DSCs obtained from local image patches. This local average DSC is equally sensitive to large and small lesions, as the patch-based DSCs associated with small and large lesions have equal weights in this average DSC. The framework was evaluated in 931 sets of images acquired in several clinical studies at two centers in Hong Kong and the United Kingdom. In particular, the training, validation, and test sets contain 615, 144, and 172 sets of images, respectively. The proposed framework outperformed single-stream networks and three recently proposed multistream networks, attaining F1 scores of 82.2 and 87.6% in the lesion and patient levels, respectively. The average inference time for an axial image was 11.8 ms. The accuracy and efficiency afforded by the proposed framework would accelerate the MRI interpretation workflow of MRI-targeted biopsy and focaltherapies.

Multi-Scale and Multi-Stream Fusion Network for Pansharpening

Pansharpening refers to the use of a panchromatic image to improve the spatial resolution of a multi-spectral image while preserving spectral signatures. However, existing pansharpening methods are still unsatisfactory at balancing the trade-off between spatial enhancement and spectral fidelity. In this paper, a multi-scale and multi-stream fusion network (named MMFN) that leverages the multi-scale information of the source images is proposed. The proposed architecture is simple, yet effective, and can fully extract various spatial/spectral features at different levels. A multi-stage reconstruction loss was adopted to recover the pansharpened images in each multi-stream fusion block, which facilitates and stabilizes the training process. The qualitative and quantitative assessment on three real remote sensing datasets (i.e., QuickBird, Pléiades, and WorldView-2) demonstrates that the proposed approach outperforms state-of-the-art methods.

Automated detection for Retinopathy of Prematurity with knowledge distilling from multi-stream fusion network

Retinopathy of Prematurity (ROP) is a potentially blinding eye disease that primarily occurs in premature infants with low birth weight. It is the main cause of childhood blindness worldwide. Various methods are available for the staging of ROP detection. However, relatively few research has focused on the early-stage detection of ROP and Treatment-Requiring ROP (TR-ROP). Besides, most of the networks proposed in recent research contain tremendous neural network parameters. This study aimed to propose a lightweight TR-ROP detection neural network with knowledge distilling from a multi-stream fusion neural network based on early-stage fundus images. A multi-stream fusion neural network was first trained with high accuracy, then knowledge distillation was used to transfer knowledge from it to a lighter model to be suitable for deployment into an embedded ROP detection device. Experiments were conducted by using a five-fold cross-validation with a dataset consisting entirely of early-stage fundus images. The proposed network could achieve promising results, with accuracy, sensitivity, and specificity of 0.9734, 0.9456, and 0.9823, in ROP detection, and 0.9222, 0.9516, and 0.8571, in TR-ROP detection, which is superior to the existing state-of-the-art methods.

CAE-Net: Cross-Modal Attention Enhancement Network for RGB-T Salient Object Detection

RGB salient object detection (SOD) performs poorly in low-contrast and complex background scenes. Fortunately, the thermal infrared image can capture the heat distribution of scenes as complementary information to the RGB image, so the RGB-T SOD has recently attracted more and more attention. Many researchers have committed to accelerating the development of RGB-T SOD, but some problems still remain to be solved. For example, the defective sample and interfering information contained in the RGB or thermal image hinder the model from learning proper saliency features, meanwhile the low-level features with noisy information result in incomplete salient objects or false positive detection. To solve these problems, we design a cross-modal attention enhancement network (CAE-Net). First, we concretely design a cross-modal fusion (CMF) module to fuse cross-modal features, where the cross-attention unit (CAU) is employed to enhance the two modal features, and channel attention is used to dynamically weigh and fuse the two modal features. Then, we design the joint-modality decoder (JMD) to fuse cross-level features, where the low-level features are purified by higher level features, and multi-scale features are sufficiently integrated. Besides, we add two single-modality decoder (SMD) branches to preserve more modality-specific information. Finally, we employ a multi-stream fusion (MSF) module to fuse three decoders’ features. Comprehensive experiments are conducted on three RGB-T datasets, and the results show that our CAE-Net is comparable to the other methods.

Improving Skeleton-Based Action Recognition Using Part-Aware Graphs in a Multi-Stream Fusion Context

Improving Skeleton-Based Action Recognition Using Part-Aware Graphs in a Multi-Stream Fusion Context

Dynamic Hand Gesture Classification Based on Multichannel Radar Using Multistream Fusion 1-D Convolutional Neural Network

Radar-based dynamic hand gesture classification has been an active research field in recent years. Deep learning methods using radar sensors are widely used to classify dynamic hand gestures. The existing deep learning methods need the overhead of performing the slow- and fast-time Fourier transforms to obtain the various spectrum images as the input of the deep convolutional neural network (DCNN). In this article, a dynamic hand gesture classification method based on the multichannel radar using a multistream fusion 1-D convolutional neural network (MSF-1-D-CNN) is proposed. The proposed MSF-1-D-CNN has four branches in parallel, and each branch has the inception modules to extract features from the raw echo data of each receiving antenna. Then, the extracted features from each branch are concatenated, and the long short-term memory (LSTM) layer is utilized to extract the temporal characteristic of the concatenated features. Finally, the dense layer with the softmax function is utilized to obtain the classification result of hand gestures. The experimental results show that, compared with existing methods, the proposed method using multichannel radar data can provide improved classification accuracy when the hand has a large incident angle and distance from the radar. Moreover, the proposed method can also reduce the network parameters and computational complexity, which has the potential to be implemented on commercial embedded systems.

A Graph Skeleton Transformer Network for Action Recognition

Skeleton-based action recognition is a research hotspot in the field of computer vision. Currently, the mainstream method is based on Graph Convolutional Networks (GCNs). Although there are many advantages of GCNs, GCNs mainly rely on graph topologies to draw dependencies between the joints, which are limited in capturing long-distance dependencies. Meanwhile, Transformer-based methods have been applied to skeleton-based action recognition because they effectively capture long-distance dependencies. However, existing Transformer-based methods lose the inherent connection information of human skeleton joints because they do not yet focus on initial graph structure information. This paper aims to improve the accuracy of skeleton-based action recognition. Therefore, a Graph Skeleton Transformer network (GSTN) for action recognition is proposed, which is based on Transformer architecture to extract global features, while using undirected graph information represented by the symmetric matrix to extract local features. Two encodings are utilized in feature processing to improve joints’ semantic and centrality features. In the process of multi-stream fusion strategies, a grid-search-based method is used to assign weights to each input stream to optimize the fusion results. We tested our method using three action recognition datasets: NTU RGB+D 60, NTU RGB+D 120, and NW-UCLA. The experimental results show that our model’s accuracy is comparable to state-of-the-art approaches.

Forward-reverse adaptive graph convolutional networks for skeleton-based action recognition

In the latest research based on skeleton data, the graph convolutional networks (GCN) based methods have achieved excellent performance on action recognition tasks. Existing GCN-based methods commonly adopt the strategy of fusing the data flow of joints and bones to obtain better results. However, these approaches ignore the reversibility of the skeleton data in the temporal dimension. Compared with the forward sequences input, which can achieve better results in certain actions through the end-to-end networks, the reverse skeleton data has excellent discrimination and richer information for some specific actions. In this work, we propose the novel forward-reverse adaptive graph convolutional networks (FR-AGCN) for skeleton-based action recognition. The sequences of joints and bones, as well as their reverse information, are modeled in the multi-stream networks at the same time. By extracting the features of forward and reverse deep information and performing multi-stream fusion, this strategy can significantly improve the recognition accuracy. Extensive experiments on two large-scale datasets NTU 60 & 120 show that the performance of our strategies has exciting advantages. On the latest dataset UAV-Human, the proposed FR-AGCN outperforms other state-of-the-art (SOTA) methods. Concretely, compared with 4s Shift-GCN, one of the most advanced models, FR-AGCN obtains significant improvements of +6.0% on CSv1 benchmark and +2.46% on CSv2 benchmark.

MSNet: A Multi-Stream Fusion Network for Remote Sensing Spatiotemporal Fusion Based on Transformer and Convolution

Remote sensing products with high temporal and spatial resolution can be hardly obtained under the constrains of existing technology and cost. Therefore, the spatiotemporal fusion of remote sensing images has attracted considerable attention. Spatiotemporal fusion algorithms based on deep learning have gradually developed, but they also face some problems. For example, the amount of data affects the model’s ability to learn, and the robustness of the model is not high. The features extracted through the convolution operation alone are insufficient, and the complex fusion method also introduces noise. To solve these problems, we propose a multi-stream fusion network for remote sensing spatiotemporal fusion based on Transformer and convolution, called MSNet. We introduce the structure of the Transformer, which aims to learn the global temporal correlation of the image. At the same time, we also use a convolutional neural network to establish the relationship between input and output and to extract features. Finally, we adopt the fusion method of average weighting to avoid using complicated methods to introduce noise. To test the robustness of MSNet, we conducted experiments on three datasets and compared them with four representative spatiotemporal fusion algorithms to prove the superiority of MSNet (Spectral Angle Mapper (SAM) < 0.193 on the CIA dataset, erreur relative global adimensionnelle de synthese (ERGAS) < 1.687 on the LGC dataset, and root mean square error (RMSE) < 0.001 on the AHB dataset).

Evaluation of Multi-Stream Fusion for Multi-View Image Set Comparison

We consider the problem of image set comparison, i.e., to determine whether two image sets show the same unique object (approximately) from the same viewpoints. Our proposition is to solve it by a multi-stream fusion of several image recognition paths. Immediate applications of this method can be found in fraud detection, deduplication procedure, or visual searching. The contribution of this paper is a novel distance measure for similarity of image sets and the experimental evaluation of several streams for the considered problem of same-car image set recognition. To determine a similarity score of image sets (this score expresses the certainty level that both sets represent the same object visible from the same set of views), we adapted a measure commonly applied in blind signal separation (BSS) evaluation. This measure is independent of the number of images in a set and the order of views in it. Separate streams for object classification (where a class represents either a car type or a car model-and-view) and object-to-object similarity evaluation (based on object features obtained alternatively by the convolutional neural network (CNN) or image keypoint descriptors) were designed. A late fusion by a fully-connected neural network (NN) completes the solution. The implementation is of modular structure—for semantic segmentation we use a Mask-RCNN (Mask regions with CNN features) with ResNet 101 as a backbone network; image feature extraction is either based on the DeepRanking neural network or classic keypoint descriptors (e.g., scale-invariant feature transform (SIFT)) and object classification is performed by two Inception V3 deep networks trained for car type-and-view and car model-and-view classification (4 views, 9 car types, and 197 car models are considered). Experiments conducted on the Stanford Cars dataset led to selection of the best system configuration that overperforms a base approach, allowing for a 67.7% GAR (genuine acceptance rate) at 3% FAR (false acceptance rate).

A Pansharpening Generative Adversarial Network with Multilevel Structure Enhancement and a Multistream Fusion Architecture

Deep learning has been widely used in various computer vision tasks. As a result, researchers have begun to explore the application of deep learning for pansharpening and have achieved remarkable results. However, most current pansharpening methods focus only on the mapping relationship between images and the lack overall structure enhancement, and do not fully and completely research optimization goals and fusion rules. Therefore, for these problems, we propose a pansharpening generative adversarial network with multilevel structure enhancement and a multistream fusion architecture. This method first uses multilevel gradient operators to obtain the structural information of the high-resolution panchromatic image. Then, it combines the spectral features with multilevel gradient information and inputs them into two subnetworks of the generator for fusion training. We design a comprehensive optimization goal for the generator, which not only minimizes the gap between the fused image and the real image but also considers the adversarial loss between the generator and the discriminator and the multilevel structure loss between the fused image and the panchromatic image. It is worth mentioning that we comprehensively consider the spectral information and the multilevel structure as the input of the discriminator, which makes it easier for the discriminator to distinguish real and fake images. Experiments show that our proposed method is superior to state-of-the-art methods in both the subjective visual and objective assessments of fused images, especially in road and building areas.