Multi-layer Fusion Research Articles

Multi-sensor adaptive fusion and convolutional neural network-based acoustic emission diagnosis for initial damage of the engine

Abstract Aiming at the problems of traditional fault diagnosis means that are difficult to identify initial damage, as well as the poor reliability and fault tolerance with a single sensor, an acoustic emission (AE) diagnosis method for initial damage of the engine based on multi-sensor adaptive fusion and convolutional neural network (CNN) is proposed. Firstly, under the premise of utilizing parametric analysis to characterize the multi-sensor AE signals, the feature parameter entropy is used to determine the primary and secondary relationships between multi-sensor signals, and then the AE feature parameter matrix is formed by adaptive fusion. Secondly, CNN is employed to mine and learn the fault feature combinations from the AE feature parameter matrix by multi-layer fusion to realize the identification and diagnosis for initial damage of the engine. Finally, the proposed method is validated on the engine test bench designed for initial damage identification and is compared with conventional methods in terms of diagnostic performance. The results demonstrate that the proposed method can achieve an identification accuracy of 98.83% for initial damage, and has advantages in various aspects such as TAMSE, K, F1mic, and F1mac, which explicitly provides a theoretical and methodological basis for identifying initial faults comprehensively and accurately. This research not only enriches the theory and methods in the field of structural health monitoring, but also provides strong technical support for engine health management, which is expected to play a key role in the maintenance and guarantee of aviation engines in the future.

Real-time multi-sensor fusion for object detection and localization in self-driving cars: A Carla simulation

Research on integrating camera and LiDAR in self-driving car systems has important scientific significance in the context of developing 4.0 technology and applying artificial intelligence. The research contributes to improving the accuracy in recognizing and locating objects in complex environments. This is an important foundation for further research on optimizing response time and improving the safety of self-driving systems. This study proposes a real-time multi-sensor data fusion method, termed "Multi-Layer Fusion," for object detection and localization in autonomous vehicles. The fusion process leverages pixel-level and feature-level integration, ensuring seamless data synchronization and robust performance under adverse conditions. Experiments conducted on the CARLA simulator. The results show that the method significantly improves environmental perception and object localization, achieving a mean detection accuracy of 95% and a mean distance error of 0.54 meters across diverse conditions, with real-time performance at 30 FPS. These results demonstrate its robustness in both ideal and adverse scenarios

Study on the Transformation Process of Thyroid Fine-Needle Aspiration Liquid-Based Cytology to Whole-Slide Image.

Analyse and summarise the reasons for failure in the digital acquisition of thyroid liquid-based cytology (LBC) slides and the technical challenges, and explore methods to obtain reliable and reproducible whole digital slide images for clinical thyroid cytology. Use the glass slide scanning imaging system to acquire whole-slide image (WSI) of thyroid LBC in sdpc format through different. Statistical analysis was conducted on the different acquisition methods, the quality of the glass slides, clinical and pathological characteristics of the case, TBSRTC grading and the quality of WSI. The WSI obtained by different scanning methods showed a high level of consistency in quality (W = 0.325, p < 0.001), especially between fully automatic scanning with different focus densities (W = 0.9, p < 0.001). A total of 2114 images were obtained through different methods of multi-layer fusion and multi-point focusing scanning, with scan success rates of 100.0%, 100.0%, 100.0% and 23.6%, respectively. The correlation between the quality of thyroid LBC glass slides and the image quality of thyroid LBC WSI was statistically significant (p < 0.001). The correlation between TBSRTC grading and the quality of thyroid LBC digital WSI was statistically significant (p < 0.001). Although the quality of glass slides has a significant impact, the success rate and image quality of malignant tumour scanning are both high. Overall, the risk of missed diagnosis of malignant tumours is low. In the future, we also need to improve the performance and algorithm of the scanner in cases of sparse cells.

A novel sensor-independent convolutional neural network for structural damage detection：illustrated by a case study on gantry crane

Structural damage detection is a critical technology for ensuring the safety of engineering structures. In recent years, intelligent structural damage detection algorithms based on machine learning have garnered widespread attention globally, thanks to their efficient and reliable detection performance. However, existing damage detection methods largely depend on the use of sensors, which not only increases operational and maintenance costs but also limits the applicability of these methods—especially in parts of structures where sensors are difficult to install. To overcome this barrier, this paper introduces a novel sensor-independent convolutional neural network (CNN) approach for predicting the maximum response of structures under seismic excitation. This method utilizes IDA method to collect a large number of samples (these samples have been uploaded to Mendeley Data for reference). It employs innovative data processing and grouping methods for feature alignment as well as the division of the test and training sets. By automatically extracting the signal characteristics of seismic excitation and optimizing feature combinations through multi-layer fusion, this method achieves high-precision and robust prediction without relying on sensors. To validate the effectiveness of this method, this study conducted a performance evaluation on a typical gantry crane metal structure. The results show that the proposed sensor-independent CNN algorithm achieves high-precision structural response prediction, with over 99% accuracy for peak responses, demonstrating the robustness and reliability of the model. It also performs excellently in key metrics such as root mean square error, error bias, and learning rate progression, making it a reliable and cost-effective solution for real-time seismic damage detection.

Metaphor recognition based on cross-modal multi-level information fusion

The metaphor is a pervasive linguistic device that has become an active research topic in the computer field because of its essential role in language's cognitive and communicative processes. Currently, the rapid expansion of social media encourages the development of multimodal. As the most popular communication method in social media, memes have attracted the attention of many linguists, who believe that metaphors contain rich metaphorical information. However, multimodal metaphor detection suffers from insufficient information due to the short text of memes and lacks effective multimodal fusion methods. To address these problems, we utilize a single-pass non-autoregressive text generation method to convert images into text to provide additional textual information for the model. In addition, the information of different modes is fused by a multi-layer fusion module consisting of a prefix guide module and a similarity-aware aggregator. The module can reduce the heterogeneity between modes, learn fine-grained information, and better integrate the characteristic information of different modes. We conducted many experiments on the Met-Meme dataset. Compared with the strong baseline model in the experiment, the weighted F1 of our model on three data types of the MET-Meme dataset improved by 1.95%, 1.55%, and 1.72%, respectively. To further demonstrate the effectiveness of the proposed method, we also conducted experiments on a multimodal sarcasm dataset and obtained competitive results.

Multilayer Fused Correntropy Reprsenstation for Fault Diagnosis of Mechanical Equipment.

Fault diagnosis is vital for improving the reliability and safety of mechanical equipment. Existing fault diagnosis methods require a large number of samples for model training. However, in real-world environments, mechanical equipment usually operates under healthy conditions during most of its service life, resulting in a scarcity of fault samples. To solve this problem, a novel multilayer fusion correntropy representation method combined with a support vector machine is proposed for the fault diagnosis of mechanical equipment. First, the monitoring signal is expanded into multilayer signal components using wavelet packet decomposition. Then, the correlation between the signal components of each layer is expressed by correntropy, and the corresponding correntropy matrix is constructed. After performing the matrix logarithm operator, all correntropy matrices composed of correntropy values are fused into a vector, which is viewed as a feature of the signal. Finally, a support vector machine is established using small samples to realize fault classification. The effectiveness of the proposed method is validated on four public datasets. The results indicate that compared with other methods, the proposed method has advantages in terms of diagnosis accuracy and noise immunity ability.

DuDoCFNet: Dual-Domain Coarse-to-Fine Progressive Network for Simultaneous Denoising, Limited-View Reconstruction, and Attenuation Correction of Cardiac SPECT.

Single-Photon Emission Computed Tomography (SPECT) is widely applied for the diagnosis of coronary artery diseases. Low-dose (LD) SPECT aims to minimize radiation exposure but leads to increased image noise. Limited-view (LV) SPECT, such as the latest GE MyoSPECT ES system, enables accelerated scanning and reduces hardware expenses but degrades reconstruction accuracy. Additionally, Computed Tomography (CT) is commonly used to derive attenuation maps ( μ -maps) for attenuation correction (AC) of cardiac SPECT, but it will introduce additional radiation exposure and SPECT-CT misalignments. Although various methods have been developed to solely focus on LD denoising, LV reconstruction, or CT-free AC in SPECT, the solution for simultaneously addressing these tasks remains challenging and under-explored. Furthermore, it is essential to explore the potential of fusing cross-domain and cross-modality information across these interrelated tasks to further enhance the accuracy of each task. Thus, we propose a Dual-Domain Coarse-to-Fine Progressive Network (DuDoCFNet), a multi-task learning method for simultaneous LD denoising, LV reconstruction, and CT-free μ -map generation of cardiac SPECT. Paired dual-domain networks in DuDoCFNet are cascaded using a multi-layer fusion mechanism for cross-domain and cross-modality feature fusion. Two-stage progressive learning strategies are applied in both projection and image domains to achieve coarse-to-fine estimations of SPECT projections and CT-derived μ -maps. Our experiments demonstrate DuDoCFNet's superior accuracy in estimating projections, generating μ -maps, and AC reconstructions compared to existing single- or multi-task learning methods, under various iterations and LD levels. The source code of this work is available at https://github.com/XiongchaoChen/DuDoCFNet-MultiTask.

Research on Feature Fusion Method Based on Graph Convolutional Networks

This paper proposes an enhanced BertGCN-Fusion (BGF) model aimed at addressing the limitations of Graph Convolutional Networks (GCN) in processing global text features for text categorization tasks. While traditional GCN effectively capture local structural features, they face challenges when integrating global semantic features. Issues such as the potential loss of global semantic information due to local feature fusion and limited depth of information propagation are prevalent. To overcome these challenges, the BGF model introduces improvements based on the BertGCN framework: (1) Feature fusion mechanism: Introducing a linear layer to fuse BERT outputs with traditional features facilitates the integration of fine-grained local semantic features from BERT with traditional global features. (2) Multilayer fusion approach: Employing a multilayer fusion technique enhances the integration of textual semantic features, thereby comprehensively and accurately capturing text semantic information. Experimental results demonstrate that the BGF model achieves notable performance improvements across multiple datasets. On the R8 and R52 datasets, the BGF model achieves accuracies of 98.45% and 93.77%, respectively, marking improvements of 0.28% to 0.90% compared to the BertGCN model. These findings highlight the BGF model’s efficacy in overcoming the deficiencies of traditional GCN in processing global semantic features, presenting an efficient approach for handling text data.

Motor imagery electroencephalography channel selection based on deep learning: A shallow convolutional neural network

Electroencephalography (EEG) motor imagery (MI) signals have recently attracted much attention because of their potential to communicate with the surrounding environment in a specific way without the need for muscular and physical movement. Despite these advantages, these signals are difficult to detect due to their low signal to noise (SNR) rate and non-stationary and dynamic nature. Convolutional neural networks (CNN) can extract appropriate spatial and temporal features without the need for separate feature extraction and classification steps. Clean input data for CNN significantly improves its performance, but sometimes, common interference occurs between multi-channel signals. This challenge, along with the noisy EEG signals, degrades the performance of CNN networks. This paper presents a channel selection method based on convolutional neural networks to obviate these challenges. The proposed shallow convolutional neural network (SCNN) is designed with temporal convolution and pointwise convolution (using 1×1 convolution) to select the best channel with minimal computational load. Following the channel selection step, the multi-layer fusion CNN model was used to classify the signals. Choosing suitable and relevant EEG channels for motor imagery tasks makes it possible to create appropriate inputs for the classification model so that the model can be improved in the end. For High Gamma Dataset and Brian-Computer Interface (BCI) Competition IV-2a, the following accuracies were obtained: 81.15% and 72.01%, which are higher than when the other channel selections such as Mutual Information, Sequential Feature Forward Selection (SFFS) and wrapper methods are used.

A Single Image High-Perception Super-Resolution Reconstruction Method Based on Multi-layer Feature Fusion Model with Adaptive Compression and Parameter Tuning

We propose a simple image high-perception super-resolution reconstruction method based on multi-layer feature fusion model with adaptive compression and parameter tuning. The aim is to further balance the high and low-frequency information of an image, enrich the detailed texture to improve perceptual quality, and improve the adaptive optimization and generalization of the model in the process of super-resolution reconstruction. First, an effective multi-layer fusion super-resolution (MFSR) basic model is constructed by the design of edge enhancement, refine layering, enhanced super-resolution generative adversarial network and other sub-models, and effective multi-layer fusion. This further enriches the image representation of features of different scales and depths and improves the feature representation of high and low-frequency information in a balanced way. Next, a total loss function of the generator is constructed with adaptive parameter tuning performance. The overall adaptability of the model is improved through adaptive weight distribution and fusion of content loss, perceptual loss, and adversarial loss, and improving the error while reducing the edge enhancement model. Finally, a fitness function with the evaluation perceptual function as the optimization strategy is constructed, and the model compression and adaptive tuning of MFSR are carried out based on the multi-mechanism fusion strategy. Consequently, the construction of the adaptive MFSR model is realized. Adaptive MFSR can maintain high peak signal to noise ratio and structural similarity on the test sets Set5, Set14, and BSD100, and achieve high-quality reconstructed images with low learned perceptual image patch similarity and perceptual index, while having good generalization capabilities.

YOLOv8MS: Algorithm for Solving Difficulties in Multiple Object Tracking of Simulated Corn Combining Feature Fusion Network and Attention Mechanism

The automatic cultivation of corn has become a significant research focus, with precision equipment operation being a key aspect of smart agriculture’s advancement. This work explores the tracking process of corn, simulating the detection and approach phases while addressing three major challenges in multiple object tracking: severe occlusion, dense object presence, and varying viewing angles. To effectively simulate these challenging conditions, a multiple object tracking dataset using simulated corn was created. To enhance accuracy and stability in corn tracking, an optimization algorithm, YOLOv8MS, is proposed based on YOLOv8. Multi-layer Fusion Diffusion Network (MFDN) is proposed for improved detection of objects of varying sizes, and the Separated and Enhancement Attention Module (SEAM) is introduced to tackle occlusion issues. Experimental results show that YOLOv8MS significantly enhances the detection accuracy, tracking accuracy and tracking stability, achieving a mean average precision (mAP) of 89.6% and a multiple object tracking accuracy (MOTA) of 92.5%, which are 1% and 6.1% improvements over the original YOLOv8, respectively. Furthermore, there was an average improvement of 4% in the identity stability indicator of tracking. This work provides essential technical support for precision agriculture in detecting and tracking corn.

Hi-gMISnet: generalized medical image segmentation using DWT based multilayer fusion and dual mode attention into high resolution pGAN

Objective. Automatic medical image segmentation is crucial for accurately isolating target tissue areas in the image from background tissues, facilitating precise diagnoses and procedures. While the proliferation of publicly available clinical datasets led to the development of deep learning-based medical image segmentation methods, a generalized, accurate, robust, and reliable approach across diverse imaging modalities remains elusive. Approach. This paper proposes a novel high-resolution parallel generative adversarial network (pGAN)-based generalized deep learning method for automatic segmentation of medical images from diverse imaging modalities. The proposed method showcases better performance and generalizability by incorporating novel components such as partial hybrid transfer learning, discrete wavelet transform (DWT)-based multilayer and multiresolution feature fusion in the encoder, and a dual mode attention gate in the decoder of the multi-resolution U-Net-based GAN. With multi-objective adversarial training loss functions including a unique reciprocal loss for enforcing cooperative learning in pGANs, it further enhances the robustness and accuracy of the segmentation map. Main results. Experimental evaluations conducted on nine diverse publicly available medical image segmentation datasets, including PhysioNet ICH, BUSI, CVC-ClinicDB, MoNuSeg, GLAS, ISIC-2018, DRIVE, Montgomery, and PROMISE12, demonstrate the proposed method’s superior performance. The proposed method achieves mean F1 scores of 79.53%, 88.68%, 82.50%, 93.25%, 90.40%, 94.19%, 81.65%, 98.48%, and 90.79%, respectively, on the above datasets, surpass state-of-the-art segmentation methods. Furthermore, our proposed method demonstrates robust multi-domain segmentation capabilities, exhibiting consistent and reliable performance. The assessment of the model’s proficiency in accurately identifying small details indicates that the high-resolution generalized medical image segmentation network (Hi-gMISnet) is more precise in segmenting even when the target area is very small. Significance. The proposed method provides robust and reliable segmentation performance on medical images, and thus it has the potential to be used in a clinical setting for the diagnosis of patients.

Other tokens matter: Exploring global and local features of Vision Transformers for Object Re-Identification

Object Re-Identification (Re-ID) aims to identify and retrieve specific objects from images captured at different places and times. Recently, object Re-ID has achieved great success with the advances of Vision Transformers (ViT). However, the effects of the global–local relation have not been fully explored in Transformers for object Re-ID. In this work, we first explore the influence of global and local features of ViT and then further propose a novel Global–Local Transformer (GLTrans) for high-performance object Re-ID. We find that the features from last few layers of ViT already have a strong representational ability, and the global and local information can mutually enhance each other. Based on this fact, we propose a Global Aggregation Encoder (GAE) to utilize the class tokens of the last few Transformer layers and learn comprehensive global features effectively. Meanwhile, we propose the Local Multi-layer Fusion (LMF) which leverages both the global cues from GAE and multi-layer patch tokens to explore the discriminative local representations. Extensive experiments demonstrate that our proposed method achieves superior performance on four object Re-ID benchmarks. The code is available at https://github.com/AWangYQ/GLTrans.

An multilayer fusion strategy based on improved YOLOv5 for ship detection in SAR images

An multilayer fusion strategy based on improved YOLOv5 for ship detection in SAR images

Semantics-enhanced discriminative descriptor learning for LiDAR-based place recognition

LiDAR-based place recognition (LPR) aims to localize autonomous vehicles and mobile robots relative to pre-built maps or retrieve previously visited places. However, the complexity of real-world scenes and changes in viewpoint are significant challenges for place recognition. As high-level information, semantics makes it easier to distinguish geometrically similar scene situations. Unlike most existing methods that rely solely on a single type of information (geometric or semantic) to construct scene descriptors, we consider the complementary nature of the semantic and geometric information and propose a semantics-enhanced discriminative feature learning method for LPR. Specifically, we first develop a Multi-layer Fusion Feature Extraction Network (MFFEN) based on the transformer encoder to hierarchically fuse local geometric and semantic information and utilize contextual information for extracting discriminative local features. To obtain semantic information, we introduce the dynamic graph convolution network to extract local semantic features with local relations. In addition, to weaken the interference of redundancy and dynamic objects in the scene, we design a semantics-guided local attention network (SLAN) to focus on salient local features that are helpful for recognizing scenes, thereby enhancing the descriptive ability of the global descriptor. Extensive experiments on public datasets KITTI and KITTI-360 demonstrate that the proposed method performs better than recent LiDAR-based methods on the 3D place recognition task. For instance, it achieves the mean F1max score of 96.9% on the KITTI dataset, surpassing the strongest prior model by 2.7%.

A novel predict method for muscular invasion of bladder cancer based on 3D mp-MRI feature fusion

Objective. To assist urologist and radiologist in the preoperative diagnosis of non-muscle-invasive bladder cancer (NMIBC) and muscle-invasive bladder cancer (MIBC), we proposed a combination models strategy (CMS) utilizing multiparametric magnetic resonance imaging. Approach. The CMS includes three components: image registration, image segmentation, and multisequence feature fusion. To ensure spatial structure consistency of T2-weighted imaging (T2WI), diffusion-weighted imaging (DWI), and dynamic contrast-enhanced imaging (DCE), a registration network based on patch sampling normalized mutual information was proposed to register DWI and DCE to T2WI. Moreover, to remove redundant information around the bladder, we employed a segmentation network to obtain the bladder and tumor regions from T2WI. Using the coordinate mapping from T2WI, we extracted these regions from DWI and DCE and integrated them into a three-branch dual-channel input. Finally, to fully fuse low-level and high-level features of T2WI, DWI, and DCE, we proposed a distributed multilayer fusion model for preoperative MIBC prediction with five-fold cross-validation. Main results. The study included 436 patients, of which 404 were for the internal cohort and 32 for external cohort. The MIBC was confirmed by pathological examination. In the internal cohort, the area under the curve, accuracy, sensitivity, and specificity achieved by our method were 0.928, 0.869, 0.753, and 0.929, respectively. For the urologist and radiologist, Vesical Imaging-Reporting and Data System score >3 was employed to determine MIBC. The urologist demonstrated an accuracy, sensitivity, and specificity of 0.842, 0.737, and 0.895, respectively, while the radiologist achieved 0.871, 0.803, and 0.906, respectively. In the external cohort, the accuracy of our method was 0.831, which was higher than that of the urologist (0.781) and the radiologist (0.813). Significance. Our proposed method achieved better diagnostic performance than urologist and was comparable to senior radiologist. These results indicate that CMS can effectively assist junior urologists and radiologists in diagnosing preoperative MIBC.

Multi-Layer Fusion 3D Object Detection via Lidar Point Cloud and Camera Image

Object detection is a key task in automatic driving, and the poor performance of small object detection is a challenge that needs to be overcome. Previously, object detection networks could detect large-scale objects in ideal environments, but detecting small objects was very difficult. To address this problem, we propose a multi-layer fusion 3D object detection network. First, a dense fusion (D-fusion) method is proposed, which is different from the traditional fusion method. By fusing the feature maps of each layer, more semantic information of the fusion network can be preserved. Secondly, in order to preserve small objects at the feature map level, we designed a feature extractor with an adaptive fusion module (AFM), which reduces the impact of the background on small objects by weighting and fusing different feature layers. Finally, an attention mechanism was added to the feature extractor to accelerate the training efficiency and convergence speed of the network by suppressing information that is irrelevant to the task. The experimental results show that our proposed approach greatly improves the baseline and outperforms most state-of-the-art methods on KITTI object detection benchmarks.

High Frequency Component Enhancement Network for Image Manipulation Detection

With the support of deep neural networks, the existing image manipulation detection (IMD) methods can detect manipulated regions within a suspicious image effectively. In general, manipulation operations (e.g., splicing, copy-move, and removal) tend to leave manipulation artifacts in the high-frequency domain of the image, which provides rich clues for locating manipulated regions. Inspired by this phenomenon, in this paper, we propose a High-Frequency Component Enhancement Network, short for HFCE-Net, for image manipulation detection, which aims to fully explore the manipulation artifacts left in the high-frequency domain to improve the localization performance in IMD tasks. Specifically, the HFCE-Net consists of two parallel branches, i.e., the main stream and high-frequency auxiliary branch (HFAB). The HFAB is introduced to fully explore high-frequency artifacts within manipulated images. To achieve this goal, the input image of the HFAB is filtered out of the low-frequency component by the Sobel filter. Furthermore, the HFEB is supervised with the edge information of the manipulated regions. The main stream branch takes the RGB image as input, and aggregates the features learned from the HFAB by the proposed multi-layer fusion (MLF) in a hierarchical manner. We conduct extensive experiments on widely used benchmarks, and the results demonstrate that our HFCE-Net exhibits a strong ability to capture high-frequency information within the manipulated image. Moreover, the proposed HFCE-Net achieves comparable performance (57.3%, 90.9%, and 73.8% F1 on CASIA, NIST, and Coverage datasets) and achieves 1.9%, 9.0%, and 1.5% improvement over the existing method.

MCNMF-Unet: a mixture Conv-MLP network with multi-scale features fusion Unet for medical image segmentation.

Recently, the medical image segmentation scheme combining Vision Transformer (ViT) and multilayer perceptron (MLP) has been widely used. However, one of its disadvantages is that the feature fusion ability of different levels is weak and lacks flexible localization information. To reduce the semantic gap between the encoding and decoding stages, we propose a mixture conv-MLP network with multi-scale features fusion Unet (MCNMF-Unet) for medical image segmentation. MCNMF-Unet is a U-shaped network based on convolution and MLP, which not only inherits the advantages of convolutional in extracting underlying features and visual structures, but also utilizes MLP to fuse local and global information of each layer of the network. MCNMF-Unet performs multi-layer fusion and multi-scale feature map skip connections in each network stage so that all the feature information can be fully utilized and the gradient disappearance problem can be alleviated. Additionally, MCNMF-Unet incorporates a multi-axis and multi-windows MLP module. This module is fully end-to-end and eliminates the need to consider the negative impact of image cropping. It not only fuses information from multiple dimensions and receptive fields but also reduces the number of parameters and computational complexity. We evaluated the proposed model on BUSI, ISIC2018 and CVC-ClinicDB datasets. The experimental results show that the performance of our proposed model is superior to most existing networks, with an IoU of 84.04% and a F1-score of 91.18%.

SpanBERT-based Multilayer Fusion Model for Extractive Reading Comprehension

SpanBERT-based Multilayer Fusion Model for Extractive Reading Comprehension