Fusion Method Research Articles

Determining the geographical origins of goji berries using the Twin-Tower model for Multi-Feature

Gogi berry products, known for their medicinal value, are exclusively available in the Ningxia region of China. However, the nutritional components, active ingredients, and economic value of these berries vary considerably among different cultivation zones in Ningxia. Accurate geographical traceability is essential not only for fostering the sustainable growth of the goji berry industry but also to ensure consumer safety, safeguard regional brands, and prevent fraudulent labelling practices that can erode market confidence and disrupt economic stability. Consequently, identification of the precise geographical origin of goji berries in Ningxia has emerged as a focal point in the progression of the industry. In this study, the Twin-Tower Model (TTM) was employed in conjunction with hyperspectral spectral and image data to establish a classification model for determining the proximate geographical origins of goji berries in Ningxia, including the regions of Zhongning, Guyuan, Tongxin, and Huinong. The results show that the graph-based training method built on TTM improves the accuracy by 3.7% compared to traditional data fusion methods. Notably, it attained a flawless 100% recognition rate for the highest quality Zhongning goji berries. These findings highlight that the implementation of the TTM method for multitask learning signifies a novel direction for transforming the conventional data fusion paradigm. Furthermore, it provides an essential technological means for the robust development of the goji berry industry.

Learning infrared degradations for coherent visible image fusion in the undecimated dual-tree complex wavelet domain

Traditional infrared (IR) and visible (VIS) image fusion methods demand identical resolution levels for source images, which can be problematic due to the inherent low-resolution nature of IR imagery. In this paper, we introduce an innovative image fusion approach that harmonizes resolution across IR-VIS source images, leading to the generation of fused images with higher resolution. We employ a convolutional neural network model to recover the real-time image degradations in IR data, with a particular focus on super-resolution through the multi degradation resolution enhancement network (MDREN). We adopt undecimated dual-tree complex wavelet transform (UDT-CWT) in our fusion process due to its near shift invariance and better directionality capabilities. This results in coherent information of the fused images with minimized noise and loss. Experiments employing five image quality assessment measures are used to compare the proposed method to nine state-of-the-art approaches and show its efficacy.

Robust and accurate visual geo-localization using prior map constructed by handheld LiDAR SLAM with camera image and terrestrial LiDAR point cloud

ABSTRACT Visual geo-localization using prior map of known environments has extensive applications in the fields such as self-driving, augmented reality and navigation. Currently, such prior maps are usually constructed by visual SLAM or SFM. However, the recent advances in LiDAR SLAM technology with RGB camera demonstrate great potential in efficient and accurate prior map construction and visual geo-localization. In this research, we developed a novel prior map construction approach which seamlessly integrates a handheld LiDAR SLAM system, a global LiDAR localization algorithm and a sparse set of terrestrial LiDAR scans as accurate control within a factor graph optimization framework. The developed LiDAR SLAM system augmented by terrestrial LiDAR scans achieves an improved mapping accuracy of 8 cm under GNSS-denied conditions in the areas over 50,000 m2 while the state-of-art FAST-LIO2 alone fails and produces a mapping error over 30 m. The results show that a competitive accuracy of visual geo-localization on a mobile phone using the constructed prior map is achieved (i.e. 60 and 76 cm in two tests respectively). Finally, this work also demonstrates a novel LiDAR point cloud fusion method that produces optimized and consistent coarse-to-fine 3D reconstruction in large and complex scenes.

Enhancing underwater target detection: Fusion of spatio‐temporal incompletely‐aligned AIS and sonar information via DTW and multi‐head attention mechanism

AbstractIn the field of underwater target detection, the passive sonar is an important means of long‐distance target detection. The sonar detection information typically includes both surface and underwater targets, whereas it is a great challenge on effectively distinguishing between surface and underwater targets solely based on sonar information. Effective fusion of sonar and AIS (Automatic Identification System) data can leverage their complementary nature to compensate for the limitation of sonar information. However, the sonar information and AIS information are acquired based on different detection principles and systems, which are essentially multi‐source heterogeneous information with obvious spatio‐temporal misalignment in nature. Existing fusion methods normally struggle to effectively align sonar and AIS data in both time and space subject to the complexity of the problem. In this study, the Dynamic Time Warping (DTW) algorithm is applied to align sonar and AIS data in the time domain. In addition, a deep learning algorithm with multi‐head attention mechanism is proposed to achieve the spatial alignment of sonar and AIS data, where the matching between the surface targets in AIS data and the same surface targets in sonar data can also be successfully achieved. It provides a priori knowledge to enhance the underwater target detection of the passive sonar by eliminating the interference of the surface targets. Based on the attention mechanism, the abstract features extracted from the intermediate‐layer of the neural networks are found to be effective to represent the typical features of the target motion trajectories, which also demonstrates the effectiveness of the attention mechanism. The experiment results show that the proposed method can successfully achieve a MatchingSucccessRate of over 95% between the AIS targets and sonar detection targets.

FGYOLO: An Integrated Feature Enhancement Lightweight Unmanned Aerial Vehicle Forest Fire Detection Framework Based on YOLOv8n

To address the challenges of complex backgrounds and small, easily confused fire and smoke targets in Unmanned Aerial Vehicle (UAV)-based forest fire detection, we propose an improved forest smoke and fire detection algorithm based on YOLOv8. Considering the limited computational resources of UAVs and the lightweight property of YOLOv8n, the original model of YOLOv8n is improved, the Bottleneck module is reconstructed using Group Shuffle Convolution (GSConv), and the residual structure is improved, thereby enhancing the model’s detection capability while reducing network parameters. The GBFPN module is proposed to optimize the neck layer network structure and fusion method, enabling the more effective extraction and fusion of pyrotechnic features. Recognizing the difficulty in capturing the prominent characteristics of fire and smoke in a complex, tree-heavy environment, we implemented the BiFormer attention mechanism to boost the model’s ability to acquire multi-scale properties while retaining fine-grained features. Additionally, the Inner-MPDIoU loss function is implemented to replace the original CIoU loss function, thereby improving the model’s capacity for detecting small targets. The experimental results of the customized G-Fire dataset reveal that FGYOLO achieves a 3.3% improvement in mean Average Precision (mAP), reaching 98.8%, while reducing the number of parameters by 26.4% compared to the original YOLOv8n.

Data fusion method for aircraft surveillance in flight zone based on Trans-Attention

Aiming at the problem of low monitoring accuracy and position jump of single monitoring source of aircraft in the flight zone, a method of aircraft monitoring data fusion based on Transformer and attention mechanism is proposed. Firstly, the encoder structure of Transformer is used to extract features of each monitoring source data respectively, and then weight values are assigned to different monitoring sources through the attention mechanism. Finally, regression calculation is performed through the fully connected network to obtain the final fusion result. The monitoring data of the surface surveillance radar and the broadcast automatic dependent surveillance system are selected as the fusion source, and the multi-point positioning data is used as the true label. The experimental results show that this method effectively reduces the monitoring error of a single monitoring source, and the fusion effect is better than the long short-term memory network, recurrent neural network and extended Kalman filter fusion method based on the attention mechanism, and the mean absolute error is improved by 2.20%、14.32%and respectively 33.94%.

Innovative SVM optimization with differential gravitational fireworks for superior air pollution classification

Amid escalating tension between environmental conservation and economic development, the imperative to enhance air quality has become increasingly urgent. This study elucidates a sophisticated approach for the assessment and remediation of air pollution issues through the integration of an enhanced particle swarm optimization algorithm and a differential gravitational fireworks algorithm-optimized support vector machine (SVM). In the initial phase of this research, a series of intricate data preprocessing and augmentation procedures were conducted, and the differential evolution algorithm played a pivotal role. The differential gravitational fireworks algorithm was subsequently introduced to optimize the SVM parameter settings, thereby bolstering classification accuracy and mitigating issues such as overfitting. Through rigorous and meticulous empirical testing, the augmented SVM model demonstrated notable performance in terms of classification accuracy and sequential and nonsequential data fusion, surpassing conventional SVM techniques. Notably, our sequential fusion method achieved an accuracy of up to 91%, at least 3% higher than that of nonsequential techniques. In conclusion, this study reveals an innovative and enhanced technological approach that is highly effective for the precise measurement and control of air pollution levels.

ESFuse: Weak Edge Structure Perception Network for Infrared and Visible Image Fusion

Infrared and visible image fusion (IVIF) fully integrates the complementary features of different modal images, and the fused image provides a more comprehensive and objective interpretation of the scene compared to each source image, thus attracting extensive attention in the field of computer vision in recent years. However, current fusion methods usually center their attention on the extraction of prominent features, falling short of adequately safeguarding subtle and diminutive structures. To address this problem, we propose an end-to-end unsupervised IVIF method (ESFuse), which effectively enhances fine edges and small structures. In particular, we introduce a two-branch head interpreter to extract features from source images of different modalities. Subsequently, these features are fed into the edge refinement module with the detail injection module (DIM) to obtain the edge detection results of the source image, improving the network’s ability to capture and retain complex details as well as global information. Finally, we implemented a multiscale feature reconstruction module to obtain the final fusion results by combining the output of the DIM with the output of the head interpreter. Extensive IVIF fusion experiments on existing publicly available datasets show that the proposed ESFuse outperforms the state-of-the-art(SOTA) methods in both subjective vision and objective evaluation, and our fusion results perform well in semantic segmentation, target detection, pose estimation and depth estimation tasks. The source code has been availabled.

Lightweight Multi-Domain Fusion Model for Through-Wall Human Activity Recognition Using IR-UWB Radar

Impulse radio ultra-wideband (IR-UWB) radar, operating in the low-frequency band, can penetrate walls and utilize its high range resolution to recognize different human activities. Complex deep neural networks have demonstrated significant performance advantages in classifying radar spectrograms of various actions, but at the cost of a substantial computational overhead. In response, this paper proposes a lightweight model named TG2-CAFNet. First, clutter suppression and time–frequency analysis are used to obtain range–time and micro-Doppler feature maps of human activities. Then, leveraging GhostV2 convolution, a lightweight feature extraction module, TG2, suitable for radar spectrograms is constructed. Using a parallel structure, the features of the two spectrograms are extracted separately. Finally, to further explore the correlation between the two spectrograms and enhance the feature representation capabilities, an improved nonlinear fusion method called coordinate attention fusion (CAF) is proposed based on attention feature fusion (AFF). This method extends the adaptive weighting fusion of AFF to a spatial distribution, effectively capturing the subtle spatial relationships between the two radar spectrograms. Experiments showed that the proposed method achieved a high degree of model lightweightness, while also achieving a recognition accuracy of 99.1%.

Strong Robust LIO System Based on Event Camera Assistance

Abstract. At present, the LiDAR system can achieve high precision and is extensively used for indoor and outdoor mobile positioning and mapping. However, LiDAR systems still face issues in cluttered environments where strong features are absent, leading to a degradation of the LiDAR-based solution. When the carrier movement involves high-speed or prolonged exposure to the mirror wall, it can cause severe degradation issues or even positioning failures in the laser slam system. Event cameras are vision sensors inspired by biology that exhibit strong robustness in high dynamic and low texture environments, potentially leading to better performance in such environments. However, some issues with event cameras remain unresolved. In this paper, a multi-source fusion method based on EVIO, LIO and IMU trajectory layer post-processing method is proposed, which will fully consider the robustness of event camera in high dynamic environment and the high precision advantage of Lidar in conventional environment, and use an algorithm based on normalized uncertainty. The elastic multi-source fusion of event camera and LiDAR is realized and tested in realde environment. Experimental results show that the proposed algorithm can effectively improve the accuracy of event camera and LiDAR. Compared to the current more advanced algorithms, the proposed algorithm effectively addresses the problem of LIO in degraded environments. Additionally, it mitigates the scale inaccuracy and divergence of the EVIO trajectory to some extent. Compared to LIO, the algorithm can reduce the maximum position error by approximately 30% and increase the overall position accuracy by 32%. Additionally, it can significantly constrain the divergence of errors in the Y direction, improving its accuracy by about 75% and 65% compared to the LIO and EVIO algorithms, respectively.

Use of aqueous polyvinyl alcohol in binder jetting of Inconel 718

AbstractBinders used in binder jetting often pose health and environmental risks during processing and post processing operations. The print-heads which are used to deposit binder selectively on the feedstock are prone to clogging, despite the trend of print-heads being highly customised to suit different kinds of binders. These factors often hide the advantages of binder jetting as an additive manufacturing process, especially its scalability and its faster printing rates in comparison to powder bed fusion methods. The work presented here takes a step back and focuses on the development of an aqueous, polyvinyl alcohol (PVA)-based liquid binder that is easy to manufacture and store, safe to handle, and can be reliably jetted to print parts. The feedstock considered was Inconel 718, a nickel-based super alloy which can be effectively processed by binder jetting without niobium segregation. PVA was added to the Inconel 718 powder in dry, granular form to manufacture a modified feedstock. The study also investigated the role of molecular weight of the PVA used, sintering environments and post-processing methods like hot isostatic pressing (HIP) on process responses like part densification, tensile strength, and hardness. Three different types of PVA were chosen which had molecular weights 10,000 g/mol (low molecular weight or LMW), 26,000 g/mol (medium molecular weight or MMW), and 84,000 g/mol (high molecular weight or HMW). The compatibility of the liquid, aqueous PVA-based binders with virgin Inconel 718 was examined by measuring the contact angle. The liquid, aqueous binder having MMW PVA reported better wetting with the Inconel 718 powder with a wetting angle of 26.6 which was lower than the wetting angle of 42.4°, seen in case of a commercial resin-based binder. The green strength reported by the MMW PVA liquid binder was 220 kPa which was higher than the other two PVA-based liquid binders. Green parts, upon successful printing, were sintered at 1260 °C. It was observed that a part printed using MMW PVA had a densification of 96.16% when sintered in 99.98% by volume argon gas, which increased to 98.96% after undergoing HIP. The same part reported a densification of 88.69% when sintered in a 95% by volume N2 and 5% by volume H2 gaseous environment, which was later attributed to the uptake of nitrogen by the chromium present in Inconel 718, which prevented necking between particles. Tensile specimens printed using MMW PVA, sintered in a 99.98% argon environment, showed the highest ultimate tensile strength of 220 MPa, which increased to 1010 MPa after the HIP process, which can be compared to commercially available Inconel 718.

Updating assembly parameters for spacecraft assembly process state changes: based on data fusion method

Thermal protection systems (TPSs) are important components of reusable spacecraft, and their assembly quality has a crucial impact on flight safety. Owing to the complex assembly process and variable states of spacecraft thermal protection systems, assembly parameters may vary under different assembly states. Therefore, to obtain assembly parameters accurately and efficiently under different assembly states, in this study, 3D point cloud data and fiber optic sensor data were fused to develop an assembly parameter update method for assembly process state changes. Firstly, based on the measured data of thermal protection components and load-bearing structure, the gap, flush and matching parameters solution model are proposed. Secondly, to address the deformation problem of the load-bearing structure caused by changes in assembly status, a fusion method based on laser scanning and sensor detection was devised to achieve deformation prediction of the assembly structure during the assembly process. Thirdly, based on the assembly parameter solution model and point cloud prediction model, a constraint-based assembly parameter optimisation model was established, and an improved quantum particle swarm optimisation (LQPSO) algorithm was employed to achieve assembly parameter updates oriented toward changes in assembly status. Finally, an experimental system for array-based thermal protection structure simulation was established to validate the proposed method. The results show that the proposed parameter update method can achieve ideal results for different assembly state simulation components.

Recent advances in data-driven fusion of multi-modal imaging and genomics for precision medicine

Imaging genomics is poised to revolutionize clinical practice by providing deep insights into the genetic underpinnings of disease, enabling early detection, and facilitating personalized treatment strategies. The field has seen remarkable advancements, with significant momentum fueled by cutting-edge imaging techniques, sophisticated data-driven fusion methods, and extensive large cohort datasets. Originally centered on the brain, imaging genomics has now expanded to encompass other organs throughout the body. Due to the highly interdisciplinary nature involving medical imaging, genetics, machine learning, and clinical medicine, readers who wish to conduct research in this field urgently need a comprehensive review. This survey provides an overview of recent advancements in data-driven fusion of multi-modal imaging and genomics, covering applications in the brain, heart, lungs, breasts, abdomen, and bones. We summarize three primary fusion strategies: correlation analysis, causal analysis, and machine learning, discussing their respective application scenarios. Additionally, we explore clinical applications that integrate imaging datasets and genomic data across six major organ systems, and present available open datasets featuring both modalities. Finally, we summarize the challenges and future directions in imaging genomics, which include improving data representation, integrating other omics data, conducting cross-dataset analyses, advancing machine learning algorithms, and investigating organ interactions. This survey aims to review the latest developments in data-driven fusion for precision medicine while providing insights into the future of this evolving field.

A deep multimodal fusion method for personality traits prediction

A deep multimodal fusion method for personality traits prediction

A novel Bi-LSTM method fusing current and historical data for tunnelling parameters of shield tunnel

Reasonable shield tunnelling parameters play a crucial role in controlling ground stability and enhancing tunnelling efficiency. Predicting shield tunnelling parameters before excavation is of paramount importance. A novel deep learning method is introduced, integrating bidirectional long short-term memory (Bi-LSTM) layers, and fully connected (FC) layers to fuse current and historical data for shield tunnelling parameters prediction. Historical data captures the impact of excavated sections on the current predicted ring, while current data considers present conditions. A feature fusion method eliminates dimensional differences between historical and current data. The resulting tensor, encompassing both data types, is fed into the FC layer to generate predictions. The effectiveness of the method is demonstrated by predicting shield cutter head torque for Qingdao Metro Line 4 in China, outperforming traditional Bi-LSTM, MLP and RF methods significantly. Ablation studies further analyze the impact of different component modules and structural parameters on model performance. Overall, this innovative approach offers accurate shield tunnelling parameters prediction, enhancing ground stability and tunnelling efficiency.

A method for predicting remaining useful life using enhanced Savitzky–Golay filter and improved deep learning framework

Ensuring operational integrity in large-scale equipment hinges on effective fault prediction and health management. Prognostics and health management (PHM) face the challenge of accurately predicting remaining useful life (RUL) using multivariate sensor data. Traditional methods often require extensive prior knowledge for indicator construction and processing. Deep learning offers a promising alternative. This study presents a multi-channel multi-scale deep learning approach. Initially, an improved Savitzky‒Golay filter (ISG) addresses challenges posed by large and rapidly changing data volumes, enhancing data preprocessing. Subsequently, a framework integrates convolutional neural networks (CNNs) with long short-term memory (LSTM) to capture hierarchical signal information and make integrated predictions. The CNN extracts spatial features from multi-channel input data, while the LSTM captures temporal dependencies. By fusing outputs from both components, the framework enhances predictive accuracy and robustness for complex operational datasets. Experimental validation on the C-MAPSS dataset tests various fusion methods and CNN depths, determining parameters and evaluating filtering effectiveness. Comparative analyses show promising performance, particularly under dynamic conditions. While not optimal for predicting multiple fault types, it outperforms classical algorithms, especially in single fault type prediction tasks.

Significant feature suppression and cross-feature fusion networks for fine-grained visual classification

The technique of extracting different distinguishing features by locating different part regions to achieve fine-grained visual classification (FGVC) has made significant improvements. Utilizing attention mechanisms for feature extraction has become one of the mainstream methods in computer vision, but these methods have certain limitations. They typically focus on the most discriminative regions and directly combine the features of these parts, neglecting other less prominent yet still discriminative regions. Additionally, these methods may not fully explore the intrinsic connections between higher-order and lower-order features to optimize model classification performance. By considering the potential relationships between different higher-order feature representations in the object image, we can enable the integrated higher-order features to contribute more significantly to the model’s classification decision-making capabilities. To this end, we propose a saliency feature suppression and cross-feature fusion network model (SFSCF-Net) to explore the interaction learning between different higher-order feature representations. These include (1) an object-level image generator (OIG): the intersection of the output feature maps of the last two convolutional blocks of the backbone network is used as an object mask and mapped to the original image for cropping to obtain an object-level image, which can effectively reduce the interference caused by complex backgrounds. (2) A saliency feature suppression module (SFSM): the most distinguishing part of the object image is obtained by a feature extractor, and the part is masked by a two-dimensional suppression method, which improves the accuracy of feature suppression. (3) A cross-feature fusion method (CFM) based on inter-layer interaction: the output feature maps of different network layers are interactively integrated to obtain high-dimensional features, and then the high-dimensional features are channel compressed to obtain the inter-layer interaction feature representation, which enriches the output feature semantic information. The proposed SFSCF-Net can be trained end-to-end and achieves state-of-the-art or competitive results on four FGVC benchmark datasets.

Multi‐domain pseudo‐reference quality evaluation for infrared and visible image fusion

AbstractInfrared and visible image fusion involves merging the advantages of infrared and visible images to generate a composite image that encompasses thermal radiation as well as intricate texture details. Infrared and visible image fusion has garnered increasing attention, with numerous fusion methods proposed. However, how to fairly perceive the performance of fused image remains a contentious topic. This paper is dedicated to solving this problem from two perspectives (e.g., subjective and objective aspects). Firstly, an infrared and visible fusion image quality assessment dataset was constructed, including 60 pairs of infrared and visible images captured in various scenes, along with 540 fusion images with different types and degrees of distortions. Additionally, a subjective evaluation dataset of 16,200 subjective scores by 30 participants was further provided for the fused image. Secondly, to overcome the challenging assessment for infrared and visible fusion images without a real reference image, an interesting multi‐domain pseudo‐reference image quality assessment model (MPIQAM) is proposed, by comprehensively considering the thermal radiation information distortion, texture information distortion, and overall naturalness of the fused image. The proposed MPIQAM was compared with 18 mainstream objective metrics, and the experimental findings showcased a commendable level of competitiveness.

Lightweight Detection of Broccoli Heads in Complex Field Environments Based on LBDC-YOLO

Robotically selective broccoli harvesting requires precise lightweight detection models to efficiently detect broccoli heads. Therefore, this study introduces a lightweight and high-precision detection model named LBDC-YOLO (Lightweight Broccoli Detection in Complex Environment—You Look Only Once), based on the improved YOLOv8 (You Look Only Once, Version 8). The model incorporates the Slim-neck design paradigm based on GSConv to reduce computational complexity. Furthermore, Triplet Attention is integrated into the backbone network to capture cross-dimensional interactions between spatial and channel dimensions, enhancing the model’s feature extraction capability under multiple interfering factors. The original neck network structure is replaced with a BiFPN (Bidirectional Feature Pyramid Network), optimizing the cross-layer connection structure, and employing weighted fusion methods for better integration of multi-scale features. The model undergoes training and testing on a dataset constructed in real field conditions, featuring broccoli images under various influencing factors. Experimental results demonstrate that LBDC-YOLO achieves an average detection accuracy of 94.44% for broccoli. Compared to the original YOLOv8n, LBDC-YOLO achieves a 32.1% reduction in computational complexity, a 47.8% decrease in parameters, a 44.4% reduction in model size, and a 0.47 percentage point accuracy improvement. When compared to models such as YOLOv5n, YOLOv5s, and YOLOv7-tiny, LBDC-YOLO exhibits higher detection accuracy and lower computational complexity, presenting clear advantages for broccoli detection tasks in complex field environments. The results of this study provide an accurate and lightweight method for the detection of broccoli heads in complex field environments. This work aims to inspire further research in precision agriculture and to advance knowledge in model-assisted agricultural practices.

Infrared and Visible Image Fusion via Sparse Representation and Guided Filtering in Laplacian Pyramid Domain

The fusion of infrared and visible images together can fully leverage the respective advantages of each, providing a more comprehensive and richer set of information. This is applicable in various fields such as military surveillance, night navigation, environmental monitoring, etc. In this paper, a novel infrared and visible image fusion method based on sparse representation and guided filtering in Laplacian pyramid (LP) domain is introduced. The source images are decomposed into low- and high-frequency bands by the LP, respectively. Sparse representation has achieved significant effectiveness in image fusion, and it is used to process the low-frequency band; the guided filtering has excellent edge-preserving effects and can effectively maintain the spatial continuity of the high-frequency band. Therefore, guided filtering combined with the weighted sum of eight-neighborhood-based modified Laplacian (WSEML) is used to process high-frequency bands. Finally, the inverse LP transform is used to reconstruct the fused image. We conducted simulation experiments on the publicly available TNO dataset to validate the superiority of our proposed algorithm in fusing infrared and visible images. Our algorithm preserves both the thermal radiation characteristics of the infrared image and the detailed features of the visible image.