Fusion Block Research Articles

CTA-Net: A gaze estimation network based on dual feature aggregation and attention cross fusion

Recent work has demonstrated the Transformer model is effective for computer vision tasks. However, the global self-attention mechanism utilized in Transformer models does not adequately consider the local structure and details of images, which may result in the loss of information and local details, causing decreased estimation accuracy in gaze estimation tasks when compared to convolution or sequential stacking methods. To address this issue, we propose a parallel CNNs-Transformer aggregation network (CTA-Net) for gaze estimation, which fully leverages the advantages of the Transformer model in modeling global context while the convolutional neural networks (CNNs) model in retaining local details. Specifically, Transformer and ResNet are deployed to extract facial and eye information, respectively. Additionally, an attention cross fusion (ACFusion) Block is embedded with CNN branch, which decomposes features in space and channels to supplement lost features, suppress noise, and help extract eye features more effectively. Finally, a dual-feature aggregation (DFA) module is proposed to effectively fuse the output features of both branches with the help feature a selection mechanism and a residual structure. Experimental results on the MPIIGaze and Gaze360 datasets demonstrate that our CTA-Net achieves state-of-the-art results.

Underwater calibration image enhancement based on image block decomposition and fusion

Underwater calibration image enhancement based on image block decomposition and fusion

RPM: RF-Based Pose Machines

Radio-frequency (RF) based human sensing technologies, due to their great practical value in various applications and privacy-preserving nature, have gained tremendous attention in recent years. However, without fully exploiting the characteristics of radio signals, the performance of existing methods are still limited. First, RF features of the moving human body have different representations in dimensions such as channel and scale, which is challenging when performing feature fusion. Besides, the human body is specularly reflective with respect to the radar, which means the human body cannot be fully captured by a single RF snapshot. Therefore, the radar signal reflected by the human body is sparse and incomplete, which is difficult to extract high-quality features for 3D human pose estimation. In this paper, we present the RF-based Pose Machines (RPM), a novel framework which can generate 3D skeletons from RF signals. Considering the characteristics of RF signals, RPM includes several modules to overcome the challenges. Firstly, a Feature Fusion Network (FFN) is designed to effectively fuse radio signals from horizontal and vertical planes based on the channels' correlation and maintain high-quality feature via a multi-scale fusion block. A Spatio-Temporal Attention network is then designed to reconstruct 3D skeletons from the sparse and incomplete RF signals. Specifically, a spatial attention module is designed to model non-local relationships among joints and reconstruct body parts that a single RF snapshot cannot capture. Afterwards, a temporal attention module is proposed to refine 3D pose based on temporal coherency learned from frame queries. To evaluate the performance of our RPM framework, we construct a large-scale dataset of synchronized 3d skeletons and RF signals, RFSkeleton3D. Our experimental results show that RPM locates 3D key points of the human body with an average error of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$5.71 cm$</tex-math></inline-formula> and maintains its performance in new environments with occlusion or bad illumination. The dataset and codes will be made in public.

SSTF-Unet: Spatial-Spectral Transformer-Based U-Net for High-Resolution Hyperspectral Image Acquisition.

To obtain a high-resolution hyperspectral image (HR-HSI), fusing a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI) is a prominent approach. Numerous approaches based on convolutional neural networks (CNNs) have been presented for hyperspectral image (HSI) and multispectral image (MSI) fusion. Nevertheless, these CNN-based methods may ignore the global relevant features from the input image due to the geometric limitations of convolutional kernels. To obtain more accurate fusion results, we provide a spatial-spectral transformer-based U-net (SSTF-Unet). Our SSTF-Unet can capture the association between distant features and explore the intrinsic information of images. More specifically, we use the spatial transformer block (SATB) and spectral transformer block (SETB) to calculate the spatial and spectral self-attention, respectively. Then, SATB and SETB are connected in parallel to form the spatial-spectral fusion block (SSFB). Inspired by the U-net architecture, we build up our SSTF-Unet through stacking several SSFBs for multiscale spatial-spectral feature fusion. Experimental results on public HSI datasets demonstrate that the designed SSTF-Unet achieves better performance than other existing HSI and MSI fusion approaches.

Single-image super-resolution reconstruction based on phase-aware visual multi-layer perceptron (MLP).

Many advanced super-resolution reconstruction methods have been proposed recently, but they often require high computational and memory resources, making them incompatible with low-power devices in reality. To address this problem, we propose a simple yet efficient super-resolution reconstruction method using waveform representation and multi-layer perceptron (MLP) for image processing. Firstly, we partition the original image and its down-sampled version into multiple patches and introduce WaveBlock to process these patches. WaveBlock represents patches as waveform functions with amplitude and phase and extracts representative feature representations by dynamically adjusting phase terms between tokens and fixed weights. Next, we fuse the extracted features through a feature fusion block and finally reconstruct the image using sub-pixel convolution. Extensive experimental results demonstrate that SRWave-MLP performs excellently in both quantitative evaluation metrics and visual quality while having significantly fewer parameters than state-of-the-art efficient super-resolution methods.

Efficient multi-branch dynamic fusion network for super-resolution of industrial component image

This work aims to promote the application of a high-performance super-resolution (SR) method in industry. Considering the lack of industrial datasets to evaluate performance, an industrial image SR dataset called WCI110 is first established, comprising 110 typical welding component images with 2040 × 1524 pixels. Subsequently, a parallel fusion structure of CNN and Transformer (FPFCT) is designed to achieve a high-quality reconstruction of component images. This model mainly contains the flexible parallel fusion (FPF) block and dynamic edge attention (DEA) network. The irreparable information loss caused by the inheritable sampling imperfection of CNN and Transformer can be effectively avoided by the parallel fusion structure in FPF block, while the contour features of multi-scale targets can be dynamically enhanced by DEA modules. The results show that the performance of FPFCT is higher than most advanced SR methods based on CNN, Transformer, or hybrid of CNN and Transformer. The images reconstructed by FPFCT are shown to help find and locate defects more effectively than other methods because its reconstructed target is closer to ground truth in terms of contour sharpness and size. Moreover, FPFCT achieves a remarkable advance in model parameters and time consumption, with a drop of more than 37.5 % in model parameters compared to the state-of-the-art SwinIR, and a reduction of more than 48.9 % in single-image delay time during reconstruction. The high efficiency of FPFCT makes it a promising image preprocessing tool for industrial images.

MSCT-UNET: multi-scale contrastive transformer within U-shaped network for medical image segmentation

Objective. Automatic mutli-organ segmentation from anotomical images is essential in disease diagnosis and treatment planning. The U-shaped neural network with encoder–decoder has achieved great success in various segmentation tasks. However, a pure convolutional neural network (CNN) is not suitable for modeling long-range relations due to limited receptive fields, and a pure transformer is not good at capturing pixel-level features. Approach. We propose a new hybrid network named MSCT-UNET which fuses CNN features with transformer features at multi-scale and introduces multi-task contrastive learning to improve the segmentation performance. Specifically, the multi-scale low-level features extracted from CNN are further encoded through several transformers to build hierarchical global contexts. Then the cross fusion block fuses the low-level and high-level features in different directions. The deep-fused features are flowed back to the CNN and transformer branch for the next scale fusion. We introduce multi-task contrastive learning including a self-supervised global contrast learning and a supervised local contrast learning into MSCT-UNET. We also make the decoder stronger by using a transformer to better restore the segmentation map. Results. Evaluation results on ACDC, Synapase and BraTS datasets demonstrate the improved performance over other methods compared. Ablation study results prove the effectiveness of our major innovations. Significance. The hybrid encoder of MSCT-UNET can capture multi-scale long-range dependencies and fine-grained detail features at the same time. The cross fusion block can fuse these features deeply. The multi-task contrastive learning of MSCT-UNET can strengthen the representation ability of the encoder and jointly optimize the networks. The source code is publicly available at: https://github.com/msctunet/MSCT_UNET.git.

Multi-Scale-Denoising Residual Convolutional Network for Retinal Disease Classification Using OCT.

Macular pathologies can cause significant vision loss. Optical coherence tomography (OCT) images of the retina can assist ophthalmologists in diagnosing macular diseases. Traditional deep learning networks for retinal disease classification cannot extract discriminative features under strong noise conditions in OCT images. To address this issue, we propose a multi-scale-denoising residual convolutional network (MS-DRCN) for classifying retinal diseases. Specifically, the MS-DRCN includes a soft-denoising block (SDB), a multi-scale context block (MCB), and a feature fusion block (FFB). The SDB can determine the threshold for soft thresholding automatically, which removes speckle noise features efficiently. The MCB is designed to capture multi-scale context information and strengthen extracted features. The FFB is dedicated to integrating high-resolution and low-resolution features to precisely identify variable lesion areas. Our approach achieved classification accuracies of 96.4% and 96.5% on the OCT2017 and OCT-C4 public datasets, respectively, outperforming other classification methods. To evaluate the robustness of our method, we introduced Gaussian noise and speckle noise with varying PSNRs into the test set of the OCT2017 dataset. The results of our anti-noise experiments demonstrate that our approach exhibits superior robustness compared with other methods, yielding accuracy improvements ranging from 0.6% to 2.9% when compared with ResNet under various PSNR noise conditions.

Edge devices-oriented surface defect segmentation by GhostNet Fusion Block and Global Auxiliary Layer

Edge devices-oriented surface defect segmentation by GhostNet Fusion Block and Global Auxiliary Layer

FSE-Net: Rethinking the up-sampling operation in encoder-decoder structure for retinal vessel segmentation

Automatic retinal vessel segmentation plays a crucial role in the diagnosis and assessment of various ophthalmologic diseases. Currently, the primary retinal vessel segmentation algorithms are based on the encoder-decoder structure. However, these U-Net analogs suffer from the loss of both spatial and semantic information, caused by continuous up-sampling operations in the decoder structure. In this paper, we rethink the above problem and build a novel deep neural network for retinal vessel segmentation, called FSE-Net. Specifically, to address the issue of feature information loss and enhance the performance of retinal vessel segmentation, we eliminate the decoder structure. In particular, we introduced a multi-head feature fusion block (MFF) as a substitute for the continuous up-sampling operation. Additionally, the encoder stage of FSE-Net incorporates a residual feature separable block (RFSB) to further refine and distill features, thereby enhancing the capability of feature extraction. Subsequently, we employ a residual atrous spatial feature aggregate module (RASF) to expand the network's receptive field by incorporating multi-scale feature information. We conducted experiments on five widely recognized databases for retinal vessel segmentation, namely DRIVE, CHASEDB1, STARE, IOSTAR, and LES-AV. The results demonstrate that our proposed FSE-Net outperforms state-of-the-art approaches in terms of segmentation performance. Moreover, we demonstrate the feasibility of achieving superior segmentation performance without employing the traditional U-Net analog network structure.

Highly compressed image representation for classification and content retrieval

In this paper, we propose a new method of representing images using highly compressed features for classification and image content retrieval – called PCA-ResFeats. They are obtained by fusing high- and low-level features from the outputs of ResNet-50 residual blocks and applying to them principal component analysis, which leads to a significant reduction in dimensionality. Further on, by applying a floating-point compression, we are able to reduce the memory required to store a single image by up to 1,200 times compared to jpg images and 220 times compared to features obtained by simple output fusion of ResNet-50. As a result, the representation of a single image from the dataset can be as low as 35 bytes on average. In comparison with the classification results on features from fusion of the last ResNet-50 residual block, we achieve a comparable accuracy (no worse than five percentage points), while preserving two orders of magnitude data compression. We also tested our method in the content-based image retrieval task, achieving better results than other known methods using sparse features. Moreover, our method enables the creation of concise summaries of image content, which can find numerous applications in databases.

Intelligent Gangue Sorting System Based on Dual-Energy X-ray and Improved YOLOv5 Algorithm

Intelligent gangue sorting with high precision is of vital importance for improving coal quality. To tackle the challenges associated with coal gangue target detection, including algorithm performance imbalance and hardware deployment difficulties, in this paper, an intelligent gangue separation system that adopts the elevated YOLO-v5 algorithm and dual-energy X-rays is proposed. Firstly, images of dual-energy X-ray transmission coal gangue mixture under the actual operation of a coal mine were collected, and datasets for training and validation were self-constructed. Then, in the YOLOv5 backbone network, the EfficientNetv2 was used to replace the original cross stage partial darknet (CSPDarknet) to achieve the lightweight of the backbone network; in the neck, a light path aggregation network (LPAN) was designed based on PAN, and a convolutional block attention module (CBAM) was integrated into the BottleneckCSP of the feature fusion block to raise the feature acquisition capability of the network and maximize the learning effect. Subsequently, to accelerate the rate of convergence, an efficient intersection over union (EIOU) was used instead of the complete intersection over union (CIOU) loss function. Finally, to address the problem of low resolution of small targets leading to missed detection, an L2 detection head was introduced to the head section to improve the multi-scale target detection performance of the algorithm. The experimental results indicate that in comparison with YOLOv5-S, the same version of the algorithm proposed in this paper increases by 19.2% and 32.4% on mAP @.5 and mAP @.5:.95, respectively. The number of parameters decline by 51.5%, and the calculation complexity declines by 14.7%. The algorithm suggested in this article offers new ideas for the design of identification algorithms for coal gangue sorting systems, which is expected to save energy and reduce consumption, reduce labor, improve efficiency, and be more friendly to the embedded platform.

Deep Learning for Integrated Speckle Reduction and Super-Resolution in Multi-Temporal SAR

In the domain of synthetic aperture radar (SAR) image processing, a prevalent issue persists wherein research predominantly focuses on single-task learning, often neglecting the concurrent impact of speckle noise and low resolution on SAR images. Currently, there are two main processing strategies. The first strategy involves conducting speckle reduction and super-resolution processing step by step. The second strategy involves performing speckle reduction as an auxiliary step, with a focus on enhancing the primary task of super-resolution processing. However, both of these strategies exhibit clear deficiencies. Nevertheless, both tasks jointly focus on two key aspects, enhancing SAR quality and restoring details. The fusion of these tasks can effectively leverage their task correlation, leading to a significant improvement in processing effectiveness. Additionally, multi-temporal SAR images covering imaging information from different time periods exhibit high correlation, providing deep learning models with a more diverse feature expression space, greatly enhancing the model’s ability to address complex issues. Therefore, this study proposes a deep learning network for integrated speckle reduction and super-resolution in multi-temporal SAR (ISSMSAR). The network aims to reduce speckle in multi-temporal SAR while significantly improving the image resolution. Specifically, it consists of two subnetworks, each taking the SAR image at time 1 and the SAR image at time 2 as inputs. Each subnetwork includes a primary feature extraction block (PFE), a high-level feature extraction block (HFE), a multi-temporal feature fusion block (FFB), and an image reconstruction block (REC). Following experiments on diverse data sources, the results demonstrate that ISSMSAR surpasses speckle reduction and super-resolution methods based on a single task in terms of both subjective perception and objective evaluation metrics regarding the quality of image restoration.

IMSF-Net:An improved multi-scale information fusion network for PPG-based blood pressure estimation

Recently, deep learning (DL) architectures have been widely used for PPG-based blood pressure (BP) monitoring due to their powerful feature extraction ability. However, the DL methods still suffer from limitations. First, the multi-scale time dependency characteristics of PPG require the DL-based methods to have the ability of sufficient feature representation at various scales. Second, the lack of DL model interpretability can lead to concerns about the reliability and generalizability of prediction. In this study, a new PPG-based end-to-end deep learning method, referred to as the IMSF-Net, is proposed for BP estimation, and its advantages are as follows: (1) the improved multi-scale fusion (IMSF) block is designed to extract different-scale feature in a two-channel way, which can obtain the abstract high-level information while avoiding low-level information loss simultaneously; (2) Permutation importance algorithm is firstly adopted to determine what parts of PPG are most useful for automatic BP estimation. On the University of California, Irvine database, the proposed IMSF-Net achieves mean absolute error (MAE) ± standard deviation (STD) of 2.07 ± 2.32 mmHg for systolic blood pressure (SBP) and 1.21 ± 1.51 mmHg for diastolic blood pressure (DBP). On the healthy database, referred to as the VDB, the method achieves MAE ± STD of 1.75 ± 3.98 mmHg for SBP and 1.29 ± 2.3 mmHg for DBP. Both of the performances outperform related previous works. The model interpretation results show that feature grids comprising of Diastolic 1–3, and Systolic 5, which generally cover the waveform from the peak point to the dicrotic notch, have a strong correlation with BP estimation.

U structured network with three encoding paths for breast tumor segmentation

Breast ultrasound segmentation remains challenging because of the blurred boundaries, irregular shapes, and the presence of shadowing and speckle noise. The majority of approaches stack convolutional layers to extract advanced semantic information, which makes it difficult to handle multiscale issues. To address those issues, we propose a three-path U-structure network (TPUNet) that consists of a three-path encoder and an attention-based feature fusion block (AFF Block). Specifically, instead of simply stacking convolutional layers, we design a three-path encoder to capture multiscale features through three independent encoding paths. Additionally, we design an attention-based feature fusion block to weight and fuse feature maps in spatial and channel dimensions. The AFF Block encourages different paths to compete with each other in order to synthesize more salient feature maps. We also investigate a hybrid loss function for reducing false negative regions and refining the boundary segmentation, as well as the deep supervision to guide different paths to capture the effective features under the corresponding receptive field sizes. According to experimental findings, our proposed TPUNet achieves more excellent results in terms of quantitative analysis and visual quality than other rival approaches.

Global semantic-guided network for saliency prediction

The human visual system effectively analyzes scenes based on local, global and semantic properties. Deep learning-based saliency prediction models adopted two-stream networks, leveraged prior knowledge of global semantics, or added long-range dependency modeling structures like transformers to incorporate global saliency information. However, they either brought high complexity to learning local and global features or neglected the design for enhancing local features. In this paper, we propose a Global Semantic-Guided Network (GSGNet), which first enriches global semantics through a modified transformer block and then incorporates semantic information into visual features from local and global perspectives in an efficient way. Multi-head self-attention in transformers captures global features, but lacks information communication within and between feature subspaces (heads) when computing the similarity matrix. To learn global representations and enhance interactions of the subspaces, we propose a Channel-Squeeze Spatial Attention (CSSA) module to emphasize channel-relevant information in a compression manner and learn global spatial relationships. To better fuse local and global contextual information, we propose a hybrid CNN-Transformer block called local–global fusion block (LGFB) for aggregating semantic features simply and efficiently. Experimental results on four public datasets demonstrate that our model achieves compelling performance compared with the state-of-the-art saliency prediction models on various evaluation metrics.

Multi-modal fusion network with intra- and inter-modality attention for prognosis prediction in breast cancer

Accurate breast cancer prognosis prediction can help clinicians to develop appropriate treatment plans and improve life quality for patients. Recent prognostic prediction studies suggest that fusing multi-modal data, e.g., genomic data and pathological images, plays a crucial role in improving predictive performance. Despite promising results of existing approaches, there remain challenges in effective multi-modal fusion. First, albeit a powerful fusion technique, Kronecker product produces high-dimensional quadratic expansion of features that may result in high computational cost and overfitting risk, thereby limiting its performance and applicability in cancer prognosis prediction. Second, most existing methods put more attention on learning cross-modality relations between different modalities, ignoring modality-specific relations that are complementary to cross-modality relations and beneficial for cancer prognosis prediction. To address these challenges, in this study we propose a novel attention-based multi-modal network to accurately predict breast cancer prognosis, which efficiently models both modality-specific and cross-modality relations without bringing in high-dimensional features. Specifically, two intra-modality self-attentional modules and an inter-modality cross-attentional module, accompanied by latent space transformation of channel affinity matrix, are developed to successfully capture modality-specific and cross-modality relations for efficient integration of genomic data and pathological images, respectively. Moreover, we design an adaptive fusion block to take full advantage of both modality-specific and cross-modality relations. Comprehensive experiment demonstrates that our method can effectively boost prognosis prediction performance of breast cancer and compare favorably with the state-of-the-art methods.

SwinHCST: a deep learning network architecture for scene classification of remote sensing images based on improved CNN and Transformer

ABSTRACTRemote sensing image scene classification is a fundamental task in intelligent interpretation of remote sensing images. Although Transformers possess a powerful attention mechanism, they require lengthy training procedures to achieve good performance levels. To address this issue, this paper proposes a novel deep learning network model by combining CNN and Swin Transformer named SwinHCST. Firstly, the model uses Weighted Normalized CNN to quickly extract low-level features of the image. Secondly, the Receptive Field Block module facilitates multi-scale information fusion, Thirdly, the Information Fusion Transformer further excavates the deep-level features of the image. Furthermore, this paper has designed a plug-and-play Cross Spatial Information Fusion Block, which is used to encodes dimensional information and extracts global information to enhance information exchange. The scene classification experiments show that the proposed model outperforms other methods on the three selected datasets and can achieve excellent performance without requiring large amounts of data and training. Specifically, the classification accuracy of the proposed method on the three datasets is 93.76%, 93.60%, and 98.10%, which is 1.7% to 3.71% higher than ResNet50 and 3.7% to 5.7% higher than Swin Transformer.

Attention-Rectified and Texture-Enhanced Cross-Attention Transformer Feature Fusion Network for Facial Expression Recognition

Facial Expression Recognition (FER) in the wild is a challenging task for affective computing in human-machine interaction fields. However, most of existing methods fail to learn the most prominent regions of facial images by simple cross-entropy loss due to the imbalance problem commonly existing in FER datasets, which limits the robustness and interpretability of the model. In addition, these methods only capture local features of original images with multi-size shallow convolution and ignore facial texture characteristics, leading to a suboptimal recognition performance. To address these issues, we propose a novel facial expression recognition network, named the attention-rectified and texture-enhanced cross-attention transformer feature fusion network (AR-TE-CATFFNet). Specifically, an attention-rectified convolution block is first designed to assist multiple convolution heads to focus on critical areas of human faces and improve the model generalization. Second, we investigate a texture enhancement block to capture texture features through local binary pattern and gray-level co-occurrence matrix, which solves the limitation of insufficient texture information. Finally, a cross-attention transformer feature fusion block is employed to deeply integrate RGB features and texture features globally, which is beneficial to boost the accuracy of recognition. Competitive experimental results on three public datasets validate the efficacy of the proposed method, indicating that our proposed method achieves superior classification performance of 89.50% on RAF-DB dataset, 65.66% on AffectNet dataset, and 74.84% on FER2013 dataset against the existing methods. The code of our proposed method will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/smy17/AR-TE-CATFFNet</uri> .

SGA-Net: A Sparse Graph Attention Network for Two-View Correspondence Learning

Establishing reliable correspondences between two images is a fundamental and important task in computer vision. This paper proposes a novel network called Sparse Graph Attention Network (SGA-Net), to capture rich contextual information of sparse graphs for feature matching task. Specifically, a graph attention block is proposed to enhance the representational ability of graph-structured features. The proposed block introduces a novel normalization technique for graph-structured features to embed global information into each edge feature, and it adopts the squeeze-and-excitation mechanism to capture graph-wise contextual information. Meanwhile, to further obtain interesting structural information of sparse graphs, a novel sparse graph transformer is developed based on multi-headed self-attention mechanism, while maintaining permutation-equivariance. Additionally, considering that the graph contexts in shallow layers are not fully exploited, a simple graph-context fusion block is introduced to adaptively capture topological information from different layers by implicitly modeling the interdependence between these graph contexts. The proposed SGA-Net can search dependable candidates among the putative correspondences and simultaneously estimate accurate camera poses for two-view geometry estimation. Extensive experiments on outlier removal and camera pose estimation tasks have demonstrated that the proposed SGA-Net outperforms state-of-the-art methods on both outdoor and indoor benchmarks (i.e., YFCC100M and SUN3D). SGA-Net achieves a mAP5° of 58.88% without RANSAC on the outdoor dataset, and it achieves a precision increase of 13.45% and 7.34% compared with the state-of-the-art result on outdoor and indoor datasets, respectively.