Multiscale Information Fusion Research Articles

HV-YOLOv8 by HDPconv: Better lightweight detectors for small object detection

Accurately identifying and localising small objects within images or videos is a critical challenge in the field of computer vision. It is mostly applied in scenarios that require high real-time performance, such as pedestrian detection and autonomous driving scenarios. These tiny targets generally include small objects at long distances, or objects appearing in low-resolution images, due to which it becomes exceptionally difficult to extract effective feature information. Since YOLOv8 with its large downsampling multiplier leads to deeper feature maps that make it difficult to detect tiny objects, we find that the use of residual structures in the convolution module can enhance the accuracy of small object detection. However, this undoubtedly increases the computational cost, so we lightened the convolution module to make it more suitable for practical applications and named it Halved Deep Pointwise Convolution (HDPConv). A cross-level partial module Variety of View Group Shuffle Cross Stage Partial Network (VOV-GSCSP) is also utilised, using a rational architecture as well as multi-scale information fusion, to ensure that the overall model is lightweight while obtaining rich gradient flows. On this basis, we propose a new network lightweight model HV-YOLOv8. In multiple sets of comparative experiments on two datasets (containing several state-of-the-art solutions as well as classical ones), we demonstrate the superiority of HV-YOLOv8, in particular, the accuracy is improved by 1.4% compared to YOLOv8, while the number of parameters and the amount of computation are drastically reduced.

ESGN: Efficient Stereo Geometry Network for Fast 3D Object Detection

Fast stereo based 3D object detectors have made great progress recently. However, they suffer from the inferior accuracy. We argue that the main reason is due to the poor geometry-aware feature representation in 3D space. To solve this problem, we propose an efficient stereo geometry network (ESGN). The key in our ESGN is an efficient geometry-aware feature generation (EGFG) module. Our EGFG module first uses a stereo correlation and reprojection module to construct multi-scale stereo volumes in camera frustum space, second employs a multi-scale bird&#x2019;s eye view (BEV) projection and fusion module to generate multiple geometry-aware features. In these two steps, we adopt deep multi-scale information fusion for discriminative geometry-aware feature generation, without any complex aggregation networks. In addition, we introduce a deep geometry-aware feature distillation scheme to guide stereo feature learning with a LiDAR-based detector. The experiments are performed on the classical KITTI dataset. On KITTI test set, our ESGN outperforms the fast state-of-art-art detector YOLOStereo3D by 5.14% on mAP_3<i>d</i> at 62<i>ms</i>. To the best of our knowledge, our ESGN achieves a best trade-off between accuracy and speed. We hope that our efficient stereo geometry network can provide more possible directions for fast 3D object detection.

A novel Dual-Branch Asymmetric Encoder–Decoder Segmentation Network for accurate colonic crypt segmentation

Colorectal cancer (CRC) is a leading cause of cancer-related deaths, with colonic crypts (CC) being crucial in its development. Accurate segmentation of CC is essential for decisions CRC and developing diagnostic strategies. However, colonic crypts’ blurred boundaries and morphological diversity bring substantial challenges for automatic segmentation. To mitigate this problem, we proposed the Dual-Branch Asymmetric Encoder–Decoder Segmentation Network (DAUNet), a novel and efficient model tailored for confocal laser endomicroscopy (CLE) CC images. In DAUNet, we crafted a dual-branch feature extraction module (DFEM), employing Focus operations and dense depth-wise separable convolution (DDSC) to extract multiscale features, boosting semantic understanding and coping with the morphological diversity of CC. We also introduced the feature fusion guided module (FFGM) to adaptively combine features from both branches using cross-group spatial and channel attention to improve the model representation in focusing on specific lesion features. These modules are seamlessly integrated into the encoder for effective multiscale information extraction and fusion, and DDSC is further introduced in the decoder to provide rich representations for precise segmentation. Moreover, the local multi-layer perceptron (LMLP) module is designed to decouple and recalibrate features through a local linear transformation that filters out the noise and refines features to provide edge-enriched representation. Experimental evaluations on two datasets demonstrate that the proposed method achieves Intersection over Union (IoU) scores of 81.54% and 84.83%, respectively, which are on par with state-of-the-art methods, exhibiting its effectiveness for CC segmentation. The proposed method holds great potential in assisting physicians with precise lesion localization and region analysis, thereby improving the diagnostic accuracy of CRC.

Cross-scale Vision Transformer for crowd localization

Crowd localization can provide the positions of individuals and the total number of people, which has great application value for security monitoring and public management, meanwhile it meets the challenges of lighting, occlusion and perspective effect. In recent times, Transformer has been applied in crowd localization to overcome these challenges. Yet such kind of methods only consider to integrate the multi-scale information once, which results in incomplete multi-scale information fusion. In this paper, we propose a novel Transformer network named Cross-scale Vision Transformer (CsViT) for crowd localization, which simultaneously fuses multi-scale information during both the encoder and decoder stages and meanwhile building the long-range context dependencies on the combined feature maps. To this end, we design the multi-scale encoder to fuse the feature maps of multiple scales at corresponding positions so as to obtain the combined feature maps, and meanwhile design the multi-scale decoder to integrate the tokens at multiple scales when modeling the long-range context dependencies. Furthermore, we propose Multi-scale SSIM (MsSSIM) loss to adaptively compute head regions and optimize the similarity at multiple scales. Specifically, we set the adaptive windows with different scales for each head and compute the loss values within these windows so as to enhance the accuracy of the predicted distance transform map. We perform comprehensive experiments on five public datasets, and the results obtained validate the effectiveness of our method.

Multiscale Feature Fusion and Semi-Supervised Temporal-Spatial Learning for Performance Monitoring in the Flotation Industrial Process.

This article studies the performance monitoring problem for the potassium chloride flotation process, which is a critical component of potassium fertilizer processing. To address its froth image segmentation problem, this article proposes a multiscale feature extraction and fusion network (MsFEFNet) to overcome the multiscale and weak edge characteristics of potassium chloride flotation froth images. MsFEFNet performs simultaneous feature extraction at multiple image scales and automatically learns spatial information of interest at each scale to achieve efficient multiscale information fusion. In addition, the potassium chloride flotation process is a multistage dynamic process with massive unlabeled data. To overcome its dynamic time-varying and working condition spatial similarity characteristics, a semi-supervised froth-grade prediction model based on a temporal-spatial neighborhood learning network combined with Mean Teacher (MT-TSNLNet) is proposed. MT-TSNLNet designs a new objective function for learning the temporal-spatial neighborhood structure of data. The introduction of Mean Teacher can further utilize unlabeled data to promote the proposed prediction model to better track the concentrate grade. To verify the effectiveness of the proposed MsFEFNet and MT-TSNLNet, froth image segmentation and grade prediction experiments are performed on a real-world potassium chloride flotation process dataset.

MSFANet: multi-scale fusion attention network for mangrove remote sensing lmage segmentation using pattern recognition

Mangroves are ecosystems that grow in the intertidal areas of coastal zones, playing crucial ecological roles and possessing unique economic and social values. They have garnered significant attention and research interest. Semantic segmentation of mangroves is a fundamental step for further investigations. However, mangrove remote sensing images often have large dimensions, with a substantial portion of the image containing mangrove features. Deep learning convolutional kernels may lead to inadequate receptive fields for accurate mangrove recognition. In mangrove remote sensing images, various challenges arise, including the presence of small and intricate details aside from the mangrove regions, which intensify the segmentation complexity. To address these issues, this paper primarily focuses on two key aspects: first, the exploration of methods to achieve a large receptive field, and second, the fusion of multi-scale information. To this end, we propose the Multi-Scale Fusion Attention Network (MSFANet), which incorporates a multi-scale network structure with a large receptive field for feature fusion. We emphasize preserving spatial information by integrating spatial data across different scales, employing separable convolutions to reduce computational complexity. Additionally, we introduce an Attention Fusion Module (AFM). This module helps mitigate the influence of irrelevant information and enhances segmentation quality. To retain more semantic information, this paper introduces a dual channel approach for information extraction through the deep structure of ResNet. We fuse features using the Feature Fusion Module (FFM) to combine both semantic and spatial information for the final output, further enhancing segmentation accuracy. In this study, a total of 230 images with dimensions of 768 pixels in width and height were selected for this experiment, with 184 images used for training and 46 images for validation. Experimental results demonstrate that our proposed method achieves excellent segmentation results on a small sample dataset of remote-sensing images, with significant practical value. This paper primarily focuses on three key aspects: the generation of mangrove datasets, the preprocessing of mangrove data, and the design and training of models. The primary contribution of this paper lies in the development of an effective approach for multi-scale information fusion and advanced feature preservation, providing a novel solution for mangrove remote sensing image segmentation tasks. The best Mean Intersection over Union (MIoU) achieved on the mangrove dataset is 86%, surpassing other existing models by a significant margin.

Location and segmentation of important railway signs based on improved segmentation

Purpose With the rapid development of railway-intelligent video technology, scene understanding is becoming more and more important. Semantic segmentation is a major part of scene understanding. There is an urgent need for an algorithm with high accuracy and real-time to meet the current railway requirements for railway identification. In response to this demand, this paper aims to explore a variety of models, accurately locate and segment important railway signs based on the improved SegNeXt algorithm, supplement the railway safety protection system and improve the intelligent level of railway safety protection. Design/methodology/approach This paper studies the performance of existing models on RailSem19 and explores the defects of each model through performance so as to further explore an algorithm model dedicated to railway semantic segmentation. In this paper, the authors explore the optimal solution of SegNeXt model for railway scenes and achieve the purpose of this paper by improving the encoder and decoder structure. Findings This paper proposes an improved SegNeXt algorithm: first, it explores the performance of various models on railways, studies the problems of semantic segmentation on railways and then analyzes the specific problems. On the basis of retaining the original excellent MSCAN encoder of SegNeXt, multiscale information fusion is used to further extract detailed features such as multihead attention and mask, solving the problem of inaccurate segmentation of current objects by the original SegNeXt algorithm. The improved algorithm is of great significance for the segmentation and recognition of railway signs. Research limitations/implications The model constructed in this paper has advantages in the feature segmentation of distant small objects, but it still has the problem of segmentation fracture for the railway, which is not completely segmented. In addition, in the throat area, due to the complexity of the railway, the segmentation results are not accurate. Social implications The identification and segmentation of railway signs based on the improved SegNeXt algorithm in this paper is of great significance for the understanding of existing railway scenes, which can greatly improve the classification and recognition ability of railway small object features and can greatly improve the degree of railway security. Originality/value This article introduces an enhanced version of the SegNeXt algorithm, which aims to improve the accuracy of semantic segmentation on railways. The study begins by investigating the performance of different models in railway scenarios and identifying the challenges associated with semantic segmentation on this particular domain. To address these challenges, the proposed approach builds upon the strong foundation of the original SegNeXt algorithm, leveraging techniques such as multi-scale information fusion, multi-head attention, and masking to extract finer details and enhance feature representation. By doing so, the improved algorithm effectively resolves the issue of inaccurate object segmentation encountered in the original SegNeXt algorithm. This advancement holds significant importance for the accurate recognition and segmentation of railway signage.

Cervical Intervertebral Disc Segmentation Based on Multi-Scale Information Fusion and Its Application

The cervical intervertebral disc, a cushion-like element between the vertebrae, plays a critical role in spinal health. Investigating how to segment these discs is crucial for identifying abnormalities in cervical conditions. This paper introduces a novel approach for segmenting cervical intervertebral discs, utilizing a framework based on multi-scale information fusion. Central to this approach is the integration of multi-level features, both low and high, through an encoding–decoding process, combined with multi-scale semantic fusion, to progressively refine the extraction of segmentation characteristics. The multi-scale semantic fusion aspect of this framework is divided into two phases: one leveraging convolution for scale interaction and the other utilizing pooling. This dual-phase method markedly improves segmentation accuracy. Facing a shortage of datasets for cervical disc segmentation, we have developed a new dataset tailored for this purpose, which includes interpolation between layers to resolve disparities in pixel spacing along the longitudinal and transverse axes in CT image sequences. This dataset is good for advancing cervical disc segmentation studies. Our experimental findings demonstrate that our network model not only achieves good segmentation accuracy on human cervical intervertebral discs but is also highly effective for three-dimensional reconstruction and printing applications. The dataset will be publicly available soon.

Classification of breast pathological images based on multiscale information interaction and fusion

Classification of breast pathological images based on multiscale information interaction and fusion

A Multiscale Information Fusion Network Based on PixelShuffle Integrated With YOLO for Aerial Remote Sensing Object Detection

A Multiscale Information Fusion Network Based on PixelShuffle Integrated With YOLO for Aerial Remote Sensing Object Detection

Multi-Level Residual Dual Attention Network for Major Cerebral Arteries Segmentation in MRA Toward Diagnosis of Cerebrovascular Disorders.

Segmentation of major brain vessels is very important for the diagnosis of cerebrovascular disorders and subsequent surgical planning. Vessel segmentation is an important preprocessing step for a wide range of algorithms for the automatic diagnosis or treatment of several vascular pathologies and as such, it is valuable to have a well-performing vascular segmentation pipeline. In this article, we propose an end-to-end multiscale residual dual attention deep neural network for resilient major brain vessel segmentation. In the proposed network, the encoder and decoder blocks of the U-Net are replaced with the multi-level atrous residual blocks to enhance the learning capability by increasing the receptive field to extract the various semantic coarse- and fine-grained features. Dual attention block is incorporated in the bottleneck to perform effective multiscale information fusion to obtain detailed structure of blood vessels. The methods were evaluated on the publicly available TubeTK data set. The proposed method outperforms the state-of-the-art techniques with dice of 0.79 on the whole-brain prediction. The statistical and visual assessments indicate that proposed network is robust to outliers and maintains higher consistency in vessel continuity than the traditional U-Net and its variations.

GRNet: Geometry Restoration for G-PCC Compressed Point Clouds Using Auxiliary Density Signaling.

The lossy Geometry-based Point Cloud Compression (G-PCC) inevitably impairs the geometry information of point clouds, which deteriorates the quality of experience (QoE) in reconstruction and/or misleads decisions in tasks such as classification. To tackle it, this work proposes GRNet for the geometry restoration of G-PCC compressed large-scale point clouds. By analyzing the content characteristics of original and G-PCC compressed point clouds, we attribute the G-PCC distortion to two key factors: point vanishing and point displacement. Visible impairments on a point cloud are usually dominated by an individual factor or superimposed by both factors, which are determined by the density of the original point cloud. To this end, we employ two different models for coordinate reconstruction, termed Coordinate Expansion and Coordinate Refinement, to attack the point vanishing and displacement, respectively. In addition, 4-byte auxiliary density information is signaled in the bitstream to assist the selection of Coordinate Expansion, Coordinate Refinement, or their combination. Before being fed into the coordinate reconstruction module, the G-PCC compressed point cloud is first processed by a Feature Analysis Module for multiscale information fusion, in which kNN-based Transformer is leveraged at each scale to adaptively characterize neighborhood geometric dynamics for effective restoration. Following the common test conditions recommended in the MPEG standardization committee, GRNet significantly improves the G-PCC anchor and remarkably outperforms state-of-the-art methods on a great variety of point clouds (e.g., solid, dense, and sparse samples) both quantitatively and qualitatively. Meanwhile, GRNet runs fairly fast and uses a smaller-size model when compared with existing learning-based approaches, making it attractive to industry practitioners.

IMSF-Net:An improved multi-scale information fusion network for PPG-based blood pressure estimation

Recently, deep learning (DL) architectures have been widely used for PPG-based blood pressure (BP) monitoring due to their powerful feature extraction ability. However, the DL methods still suffer from limitations. First, the multi-scale time dependency characteristics of PPG require the DL-based methods to have the ability of sufficient feature representation at various scales. Second, the lack of DL model interpretability can lead to concerns about the reliability and generalizability of prediction. In this study, a new PPG-based end-to-end deep learning method, referred to as the IMSF-Net, is proposed for BP estimation, and its advantages are as follows: (1) the improved multi-scale fusion (IMSF) block is designed to extract different-scale feature in a two-channel way, which can obtain the abstract high-level information while avoiding low-level information loss simultaneously; (2) Permutation importance algorithm is firstly adopted to determine what parts of PPG are most useful for automatic BP estimation. On the University of California, Irvine database, the proposed IMSF-Net achieves mean absolute error (MAE) ± standard deviation (STD) of 2.07 ± 2.32 mmHg for systolic blood pressure (SBP) and 1.21 ± 1.51 mmHg for diastolic blood pressure (DBP). On the healthy database, referred to as the VDB, the method achieves MAE ± STD of 1.75 ± 3.98 mmHg for SBP and 1.29 ± 2.3 mmHg for DBP. Both of the performances outperform related previous works. The model interpretation results show that feature grids comprising of Diastolic 1–3, and Systolic 5, which generally cover the waveform from the peak point to the dicrotic notch, have a strong correlation with BP estimation.

SwinHCST: a deep learning network architecture for scene classification of remote sensing images based on improved CNN and Transformer

ABSTRACTRemote sensing image scene classification is a fundamental task in intelligent interpretation of remote sensing images. Although Transformers possess a powerful attention mechanism, they require lengthy training procedures to achieve good performance levels. To address this issue, this paper proposes a novel deep learning network model by combining CNN and Swin Transformer named SwinHCST. Firstly, the model uses Weighted Normalized CNN to quickly extract low-level features of the image. Secondly, the Receptive Field Block module facilitates multi-scale information fusion, Thirdly, the Information Fusion Transformer further excavates the deep-level features of the image. Furthermore, this paper has designed a plug-and-play Cross Spatial Information Fusion Block, which is used to encodes dimensional information and extracts global information to enhance information exchange. The scene classification experiments show that the proposed model outperforms other methods on the three selected datasets and can achieve excellent performance without requiring large amounts of data and training. Specifically, the classification accuracy of the proposed method on the three datasets is 93.76%, 93.60%, and 98.10%, which is 1.7% to 3.71% higher than ResNet50 and 3.7% to 5.7% higher than Swin Transformer.

DAMS-Net: Dual attention and multi-scale information fusion network for 12-lead ECG classification

Automated 12-lead electrocardiographic (ECG) classification algorithms play an important role in the diagnosis of clinical arrhythmias. Current methods that perform well in the field of automatic ECG classification are usually based on Convolutional Neural Networks (CNN) or Transformer. However, due to the intrinsic locality of convolution operations, CNN can't extract long-dependence between series. On the other side, the Transformer design includes a built-in global self-attention mechanism, but it doesn't pay enough attention to local features. In this paper, we propose DAMS-Net, which combines the advantages of Transformer and CNN, introducing a spatial attention module and a channel attention module using a CNN-Transformer hybrid encoder to adaptively focus on the significant features of global and local parts between space and channels. In addition, our proposal fuses multi-scale information to capture high and low-level semantic information by skip-connections. We evaluate our method on the 2018 Physiological Electrical Signaling Challenge dataset, and our proposal achieves a precision rate of 83.6%, a recall rate of 84.7%, and an F1-score of 0.839. The classification performance is superior to all current single-model methods evaluated in this dataset. The experimental results demonstrate the promising application of our proposed method in 12-lead ECG automatic classification tasks.

Fusing multi-scale fuzzy information to detect outliers

Outlier detection aims to find objects that behave differently from the majority of the data. Existing unsupervised approaches often process data with a single scale, which may not capture the multi-scale nature of the data. In this paper, we propose a novel information fusion model based on multi-scale fuzzy granules and an unsupervised outlier detection algorithm with the fuzzy rough set theory. First, a multi-scale information fusion model is formulated based on fuzzy granules. Then we employ fuzzy approximations to define the outlier factor of multi-scale fuzzy granules centered at each data point. Finally, the outlier score is calculated by aggregating the outlier factors of a set of multi-scale fuzzy granules. Experimental results demonstrate that the proposed method is comparable with or better than the leading outlier detection methods. The codes and datasets are publicly available online at https://github.com/ChenBaiyang/MFIOD.

BertNDA: A Model Based on Graph-Bert and Multi-scale Information Fusion for ncRNA-disease Association Prediction.

Non-coding RNAs (ncRNAs) are a class of RNA molecules that lack the ability to encode proteins in human cells, but play crucial roles in various biological process. Understanding the interactions between different ncRNAs and their impact on diseases can significantly contribute to diagnosis, prevention, and treatment of diseases. However, predicting tertiary interactions between ncRNAs and diseases based on structural information in multiple scales remains a challenging task. To address this challenge, we propose a method called BertNDA, aiming to predict potential relationships between miRNAs, lncRNAs, and diseases. The framework identifies thelocal information through connectionless subgraph, which aggregate neighbor nodes' feature. And global information is extracted by leveraging Laplace transform of graph structures and WL (Weisfeiler-Lehman) absolute role coding. Additionally, an EMLP (Element-wise MLP) structure is designed to fuse pairwise global information. The transformer-encoder is employed as the backbone of our approach, followed by a prediction-layer to output the final correlation score. Extensive experiments demonstrate that BertNDA outperforms state-of-the-art methods in prediction assignment and exhibits significant potential for various biological applications. Moreover, we develop an online prediction platform that incorporates the prediction model, providing users with an intuitive and interactive experience. Overall, our model offers an efficient, accurate, and comprehensive tool for predicting tertiary associations between ncRNAs and diseases. Code of our method can be available in: https://github.com/zhiweining/BertNDA-main.

Mifanet: multi-scale information fusion attention network for determining hatching eggs activity via detecting PPG signals

Mifanet: multi-scale information fusion attention network for determining hatching eggs activity via detecting PPG signals

CloudDeepLabV3+: a lightweight ground-based cloud segmentation method based on multi-scale feature aggregation and multi-level attention feature enhancement

ABSTRACT The segmentation of ground-based cloud images is the basis for obtaining numerous cloud parameters. To achieve high-precision adaptive cloud image segmentation requirements, this study designs a lightweight ground-based cloud image adaptive segmentation method named CloudDeepLabV3+ that integrates multi-scale features aggregation and multi-level attention feature enhancement. Firstly, a novel lightweight EfficientNetV2-S is designed as a feature extraction backbone to reduce network parameters. Secondly, a heterogeneous receptive field splicing atrous spatial pyramid pooling module is designed. It enhances the correlation of information between layers, and realizes multiscale information fusion. The feature enhancement module based on the self-attention mechanism intensifies the representation of local and global features. Thirdly, the feature alignment module based on the attention mechanism is constructed to pull deep and shallow features for alignment. Finally, we implement ablation study on the key components of method and comparison experiment with other advanced methods using five evaluation metrics. Results show that the key components play an important role in multiscale information fusion. It promotes the accuracy of cloud image feature extraction while reducing the loss of detailed features. Generalization performance verification indicates the excellent performance of the proposed model in advanced cloud feature extraction and cloud-mask generation.

UcUNet: A lightweight and precise medical image segmentation network based on efficient large kernel U-shaped convolutional module design

In recent years, precise medical image segmentation methods based on the encoder–decoder structure have attracted much attention, but there are still some limitations. They mainly manifest in two aspects: one is that the network structure becomes increasingly complex in pursuit of segmentation accuracy, and the other is the insufficient multiscale information fusion ability between the encoder and decoder. To address these issues, we designed a novel lightweight precise medical image segmentation network called UcUNet, which has not only a very large receptive field and multiscale information fusion ability but also a low parameter count. Specifically, we first designed an efficient U-shaped convolution module that can increase the network depth with fewer parameters and effectively filter out invalid features while fusing shallow and deep features. Furthermore, we ingeniously introduced large-kernel convolution into this module, which significantly improved the network’s receptive field with very few parameters, providing new ideas for the use of large-kernel convolution in lightweight models. In addition, we designed a self-adjusting multiscale feature fusion module based on large-kernel convolution to replace the original simple skip connections, effectively enhancing the network’s multiscale information extraction and fusion ability. Finally, we conducted experiments on multiple datasets, and the results showed that our method can achieve precise medical image segmentation with a low parameter count.