Robust Fusion Research Articles

RCRFNet: Enhancing Object Detection with Self-Supervised Radar-Camera Fusion and Open-Set Recognition.

Robust object detection in complex environments, poor visual conditions, and open scenarios presents significant technical challenges in autonomous driving. These challenges necessitate the development of advanced fusion methods for millimeter-wave (mmWave) radar point cloud data and visual images. To address these issues, this paper proposes a radar-camera robust fusion network (RCRFNet), which leverages self-supervised learning and open-set recognition to effectively utilise the complementary information from both sensors. Specifically, the network uses matched radar-camera data through a frustum association approach to generate self-supervised signals, enhancing network training. The integration of global and local depth consistencies between radar point clouds and visual images, along with image features, helps construct object class confidence levels for detecting unknown targets. Additionally, these techniques are combined with a multi-layer feature extraction backbone and a multimodal feature detection head to achieve robust object detection. Experiments on the nuScenes public dataset demonstrate that RCRFNet outperforms state-of-the-art (SOTA) methods, particularly in conditions of low visual visibility and when detecting unknown class objects.

Bayesian multi-exposure image fusion for robust high dynamic range ptychography.

The limited dynamic range of the detector can impede coherent diffractive imaging (CDI) schemes from achieving diffraction-limited resolution. To overcome this limitation, a straightforward approach is to utilize high dynamic range (HDR) imaging through multi-exposure image fusion (MEF). This method involves capturing measurements at different exposure times, spanning from under to overexposure and fusing them into a single HDR image. The conventional MEF technique in ptychography typically involves subtracting the background noise, ignoring the saturated pixels and then merging the acquisitions. However, this approach is inadequate under conditions of low signal-to-noise ratio (SNR). Additionally, variations in illumination intensity significantly affect the phase retrieval process. To address these issues, we propose a Bayesian MEF modeling approach based on a modified Poisson distribution that takes the background and saturation into account. The expectation-maximization (EM) algorithm is employed to infer the model parameters. As demonstrated with synthetic and experimental data, our approach outperforms the conventional MEF method, offering superior phase retrieval under challenging experimental conditions. This work underscores the significance of robust multi-exposure image fusion for ptychography, particularly in imaging shot-noise-dominated weakly scattering specimens or in cases where access to HDR detectors with high SNR is limited. Furthermore, the applicability of the Bayesian MEF approach extends beyond CDI to any imaging scheme that requires HDR treatment. Given this versatility, we provide the implementation of our algorithm as a Python package.

SiamDCFF: Dynamic Cascade Feature Fusion for Vision Tracking.

Establishing an accurate and robust feature fusion mechanism is key to enhancing the tracking performance of single-object trackers based on a Siamese network. However, the output features of the depth-wise cross-correlation feature fusion module in fully convolutional trackers based on Siamese networks cannot establish global dependencies on the feature maps of a search area. This paper proposes a dynamic cascade feature fusion (DCFF) module by introducing a local feature guidance (LFG) module and dynamic attention modules (DAMs) after the depth-wise cross-correlation module to enhance the global dependency modeling capability during the feature fusion process. In this paper, a set of verification experiments is designed to investigate whether establishing global dependencies for the features output by the depth-wise cross-correlation operation can significantly improve the performance of fully convolutional trackers based on a Siamese network, providing experimental support for rational design of the structure of a dynamic cascade feature fusion module. Secondly, we integrate the dynamic cascade feature fusion module into the tracking framework based on a Siamese network, propose SiamDCFF, and evaluate it using public datasets. Compared with the baseline model, SiamDCFF demonstrated significant improvements.

Efficient and Accurate pH Determination with pH Test Strips Based on Machine Learning.

The determination of pH values is crucial in various fields, such as analytical chemistry, medical diagnostics, and biochemical research. pH test strips, renowned for their convenience and cost-effectiveness, are commonly utilized for pH qualitative estimation. Recently, quantitative methods for determining pH values using pH test strips have been developed. However, these methods can be prone to errors due to environmental factors, such as lighting conditions, which affect the imaging quality of the pH test strips. To address these challenges, we developed an innovative approach that combines machine learning techniques with pH test strips for the quantitative determination of pH values. Our method involves extracting artificial features from the pH test strip images and combining them across multiple dimensions for comprehensive analysis. To ensure optimal feature selection, we developed a feature selection strategy based on SHAP importance. This strategy helps in identifying the most relevant features that contribute to accurate pH prediction. Furthermore, we integrated multiple machine learning algorithms, employing a robust stacking fusion strategy to establish a highly reliable pH value prediction model. Our proposed method automates the determination of pH values through pH test strips, effectively overcoming the limitations associated with environmental lighting interference. Experimental results demonstrate that this method is convenient, effective, and highly reliable for the determination of pH values.

Design of first experiment to achieve fusion target gain > 1

A decades-long quest to achieve fusion energy target gain and ignition in a controlled laboratory experiment, dating back to 1962, has been realized at the National Ignition Facility (NIF) on December 5, 2022 [Abu-Shawareb et al., Phys. Rev. Lett. 132, 065102 (2024)] where an imploded pellet of deuterium and tritium (DT) fuel generated more fusion energy (3.15 MJ) than laser energy incident on the target (2.05 MJ). In these experiments, laser beams incident on the inside of a cylindrical can (Hohlraum) generate an intense ∼3 × 106 million degree x-ray radiation bath that is used to spherically implode ∼2 mm diameter pellets containing frozen deuterium and tritium. The maximum fusion energy produced in this configuration to date is 3.88 MJ using 2.05 MJ of incident laser energy and 5.2 MJ using 2.2 MJ of incident laser energy, producing a new record target gain of ∼2.4×. This paper describes the physics (target and laser) design of this platform and follow-on experiments that show increased performance. We show robust megajoule fusion energy output using this design as well as explore design modification using radiation hydrodynamic simulations benchmarked against experimental data, which can further improve the performance of this platform.

An OpenFOAM solver for multiphysics modeling of fusion reactor design: The nemoFoam code

This paper introduces a multiphysics code, named nemoFoam, designed within the OpenFOAM environment to address the coupling of neutron and photon transport with thermal-hydraulics in nuclear reactor simulations. The code, conceived both for fusion and fission applications, employs a modular approach where the neutronic module is currently based on multi-group neutron diffusion equations, including a mono-kinetic treatment for photons. The thermal-hydraulic module is built on the standard OpenFOAM solver. The paper focuses on presenting the key features of nemoFoam and discussing the structure of the code, the nuclear and thermal-hydraulic models, and the coupling strategy between them.In order to assess the performance of the neutronic module, the code is applied to a fusion case study, indeed a benchmark against the Serpent Monte Carlo code for neutron and photon transport is performed, applying nemoFoam to the Affordable, Robust and Compact (ARC) fusion reactor design. Simulation results demonstrate good agreement with Monte Carlo benchmarks, emphasizing the potential of the code for multiphysics simulations. The modular design of nemoFoam allows users to extend the implemented model equations to study additional phenomena, focus only on selected aspects independently and, potentially, to add new solvers in each module.

Toward digital design at the exascale: An overview of project ICECap

High performance computing has entered the Exascale Age. Capable of performing over 1018 floating point operations per second, exascale computers, such as El Capitan, the National Nuclear Security Administration's first, have the potential to revolutionize the detailed in-depth study of highly complex science and engineering systems. However, in addition to these kind of whole machine “hero” simulations, exascale systems could also enable new paradigms in digital design by making petascale hero runs routine. Currently, untenable problems in complex system design, optimization, model exploration, and scientific discovery could all become possible. Motivated by the challenge of uncovering the next generation of robust high-yield inertial confinement fusion (ICF) designs, project ICECap (Inertial Confinement on El Capitan) attempts to integrate multiple advances in machine learning (ML), scientific workflows, high performance computing, GPU-acceleration, and numerical optimization to prototype such a future. Built on a general framework, ICECap is exploring how these technologies could broadly accelerate scientific discovery on El Capitan. In addition to our requirements, system-level design, and challenges, we describe some of the key technologies in ICECap, including ML replacements for multiphysics packages, tools for human-machine teaming, and algorithms for multifidelity design optimization under uncertainty. As a test of our prototype pre-El Capitan system, we advance the state-of-the art for ICF hohlraum design by demonstrating the optimization of a 17-parameter National Ignition Facility experiment and show that our ML-assisted workflow makes design choices that are consistent with physics intuition, but in an automated, efficient, and mathematically rigorous fashion.

MRFTrans: Multimodal Representation Fusion Transformer for monocular 3D semantic scene completion

The complete understanding of 3D scenes is crucial in robotic visual perception, impacting tasks such as motion planning and map localization. However, due to the limited field of view and scene occlusion constraints of sensors, inferring complete scene geometry and semantic information from restricted observations is challenging. In this work, we propose a novel Multimodal Representation Fusion Transformer framework (MRFTrans) that robustly fuses semantic, geometric occupancy, and depth representations for monocular-image-based scene completion. MRFTrans centers on an affinity representation fusion transformer, integrating geometric occupancy and semantic relationships within a transformer architecture. This integration enables the modeling of long-range dependencies within scenes for inferring missing information. Additionally, we present a depth representation fusion method, efficiently extracting reliable depth knowledge from biased monocular estimates. Extensive experiments demonstrate MRFTrans’s superiority, setting a new benchmark on SemanticKITTI and NYUv2 datasets. It significantly enhances completeness and accuracy, particularly in large structures, movable objects, and scene components with major occlusions. The results underscore the benefits of the affinity-aware transformer and robust depth fusion in monocular-image-based completion.

Robust DOA Estimation Using Multi-Scale Fusion Network with Attention Mask

To overcome the limitations of traditional methods in reverberant and noisy environments, a robust multi-scale fusion neural network with attention mask is designed to improve direction-of-arrival (DOA) estimation accuracy for acoustic sources. It combines the benefits of deep learning and complex-valued operations to effectively deal with the interference of reverberation and noise in speech signals. The unique properties of complex-valued signals are exploited to fully capture inherent features and rich information is preserved in the complex field. An attention mask module is designed to generate distinct masks for selectively focusing and masking based on the input. After that, the multi-scale fusion block efficiently captures multi-scale spatial features by stacking complex-valued convolutional layers with small size kernels, and reduces the module complexity through special branching operations. Experimental results demonstrate that the model achieves significant improvements over other methods for speaker localization in reverberant and noisy environments. It provides a new solution for DOA estimation for acoustic sources in different scenarios, which has significant theoretical and practical implications.

Highly robust and accurate multi-sensor fusion localization system for complex and challenging scenarios

Highly robust and accurate localization using on-board sensors in the absence of GNSS is a key issue for long-term autonomous navigation of vehicles. However, the limited sensing capabilities of a single sensor cannot resist the effects of feature degradation and high-speed motion in complex and challenging scenarios, which results in the fragile of existing LiDAR-based and visual-based localization algorithms. To solve these problems, we propose a highly robust and accurate localization system by fusing information from multi-sensors. Firstly, we propose a global localization method that leverages deep learning models for processing visual information to provide an accurate initial position for vehicles, which solves the problem of global localization of vehicles in the absence of GNSS. Then, we propose a highly robust and accurate LiDAR-Vision-IMU localization method based on pose graph optimization, which integrates LiDAR-based poses and visual-inertial-based poses. The proposed method detects sensor degradation in real time to dynamically adjust the fusion weights, which solves the problem of low localization performance of a single sensor. Finally, we conduct experiments in complex and challenging scenarios, such as urban, tunnels, and unstructured mountains. Experimental results show that our method can achieve a global localization error of less than 0.5 m in complex scenes. In unstructured mountainous scenarios of length 21.1 km, the root mean square errors (RMSE) of translation and rotation are 0.67 m and 0.048rad, respectively. Compared to most existing algorithms, our method achieves the most robust and accurate localization performance.

FGO-MFI: factor graph optimization-based multi-sensor fusion and integration for reliable localization

Reliable and precise information pertaining to the position, velocity, and attitude is essential for automated driving. This paper proposes FGO-MFI, a cost-effective and robust multi-sensor fusion and integration localization framework that utilizes factor graph optimization. Firstly, a tightly coupled Global Navigation Satellite Systems (GNSS)/on-board sensor fusion localization framework is established to estimate vehicle states, including position, velocity, and attitude. To address the large drift rate of the Inertial Measurement Unit (IMU), this study introduces a novel IMU/Dynamics pre-integration method based on the vehicle dynamics model. We establish a two-degree-of-freedom vehicle dynamics model utilizing measurements from the wheel speed sensor and steering wheel angle sensor. The IMU/Dynamics factor is devised through a close integration of the model output and IMU pre-integration, enabling the construction of precise odometry with low-cost on-board sensors. Then, to address the issue of the non-Gaussian distribution of GNSS pseudorange error, this paper employs a Gaussian Mixture Model (GMM) to characterize the pseudorange noise, which is then applied to further sensor fusion. Given the time-varying nature of pseudorange noise, the expectation maximization algorithm is utilized to estimate GMM parameters online, leveraging the pseudorange residuals within a sliding window. Comprehensive experiments, inclusive of challenging scenarios such as urban canyons, tunnels, and wooded areas, have been carried out. They affirm the superior performance of the proposed method. Experiments have shown that our method demonstrates reliable localization across different statuses of GNSS signal, exhibiting a 39.4% improvement in the root mean square error of position error when compared to the state-of-the-art. Additionally, this FGO-MFI is a general sensor data fusion framework and is able to incorporate diverse sensor measurements, for example, from cameras and LiDARs to provide more reliable and accurate localization information.

Boosting Robust Multi-Focus Image Fusion With Frequency Mask and Hyperdimensional Computing

Boosting Robust Multi-Focus Image Fusion With Frequency Mask and Hyperdimensional Computing

Geometric Correspondence-Based Multimodal Learning for Ophthalmic Image Analysis.

Color fundus photography (CFP) and Optical coherence tomography (OCT) images are two of the most widely used modalities in the clinical diagnosis and management of retinal diseases. Despite the widespread use of multimodal imaging in clinical practice, few methods for automated diagnosis of eye diseases utilize correlated and complementary information from multiple modalities effectively. This paper explores how to leverage the information from CFP and OCT images to improve the automated diagnosis of retinal diseases. We propose a novel multimodal learning method, named geometric correspondence-based multimodal learning network (GeCoM-Net), to achieve the fusion of CFP and OCT images. Specifically, inspired by clinical observations, we consider the geometric correspondence between the OCT slice and the CFP region to learn the correlated features of the two modalities for robust fusion. Furthermore, we design a new feature selection strategy to extract discriminative OCT representations by automatically selecting the important feature maps from OCT slices. Unlike the existing multimodal learning methods, GeCoM-Net is the first method that formulates the geometric relationships between the OCT slice and the corresponding region of the CFP image explicitly for CFP and OCT fusion. Experiments have been conducted on a large-scale private dataset and a publicly available dataset to evaluate the effectiveness of GeCoM-Net for diagnosing diabetic macular edema (DME), impaired visual acuity (VA) and glaucoma. The empirical results show that our method outperforms the current state-of-the-art multimodal learning methods by improving the AUROC score 0.4%, 1.9% and 2.9% for DME, VA and glaucoma detection, respectively.

Highly compact adaptive network based on transformer for RGBT tracking

RGBT tracking is a challenging task that requires robust fusion of visible images (RGB) and thermal infrared images (TIR) to handle various scenarios, such as illumination changes, occlusions, and camouflage. The current popular RGBT trackers mostly based on two stream Siamese trackers. They tend to separately extracts template and search images region features, which neglects relationship between target and background. Moreover, they do not fuse RGB and TIR features properly, limiting their ability to utilize complementary features. To address these issues, we introduce a highly compact adaptive transformer-based network that unifies the process of feature extracting and correlation between template and search images. In this way, the compact network has dual-branches. They can combine feature extraction and correlation for different modalities of template images and search images. Meanwhile, we introduce a cross-modal weight redistribution module (CMWR) for multi-modal fusion. This adaptive fusion scheme learns discriminative features of RGB and TIR data and assigns weights to them, enabling them to complement each other. Furthermore, to address the issue of tracking targets of different scales, we design scale-adaptive optimization pyramid module (SAOP) that adapt to objects of different sizes. Our method achieves exceptional performance on the GTOT, RGBT234 and LasHeR datasets, surpassing most of the existing methods. The results are consistent across multiple datasets, demonstrating the effectiveness and superiority of our approach. And our code is released at: https://github.com/ELOESZHANG/HCANet.

A Robust Medical Image Fusion Based on Synthetic Focusing Degree Criterion and Special Kernel Set for Clinical Diagnosis

A Robust Medical Image Fusion Based on Synthetic Focusing Degree Criterion and Special Kernel Set for Clinical Diagnosis

1078 Performance of Improved Forms of Demineralized Bone Fiber Allografts in Rabbit Posterolateral Spine Fusion Model

INTRODUCTION: Biologic fusion is often critical for successful spine procedures. Autograft remains the Gold Standard bone graft due to osteogenic properties but has limitations in practice. New bone graft forms that efficiently incorporate autograft or bone marrow aspirate (BMA) and provide osteoinductive and osteoconductive properties were evaluated in the current study. Grafts were made using demineralized bone fibers (DBF) produced by a patented method that provides long fibers that can effectively entangle autograft and be formed into open cylinders (PAK) that contain and prevent autograft migration and absorb BMA. METHODS: DBF was produced from rabbit cortical bone using the same processes as human clinical product. Pak grafts were made with 1.5cc Fibers formed into a cylinder with 1.5cc volume. Implants into the rabbit PLF model were harvested after 8 weeks and evaluated for fusion by manual palpation and x-rays. RESULTS: All graft forms demonstrated robust new bone formation. Each group demonstrated high fusion rates. Across the groups 10/11 levels were 100% fused (91%) with the remaining 1/11 being 95% reduced motion (functionally fused). Note radiolucency of Group 3 at post-op and appearance of new bone at 8 weeks. CONCLUSIONS: The new graft forms demonstrated robust bone formation and fusion in PLF model. The DBF Fibers provided excellent entanglement of autograft to prevent migration. The PAK form eliminated autograft migration, and absorbed BMA providing osteogenesis to the osteoconductive and osteoinductive graft.

Mapping Geothermal Indicator Minerals Using Fusion of Target Detection Algorithms

Mineral mapping from satellite images provides valuable insights into subsurface mineral alteration for geothermal exploration. In previous studies, eight fundamental algorithms were used for mineral mapping utilizing USGS spectra, a collection of reflectance spectra containing samples of minerals, rocks, and soils created by the USGS. We used an ASD FieldSpec 4 Hi-RES NG portable spectrometer to collect spectra for analyzing ASTER images of the Coso Geothermal Field. Then, we established the ground-truth information and the spectral library by analyzing 97 samples. Samples collected from the field were analyzed using the CSIRO TSG (The Spectral Geologist of the Commonwealth Scientific and Industrial Research Organization). Based on the mineralogy study, multiple high-purity spectra of geothermal alteration minerals were selected from collected data, including alunite, chalcedony, hematite, kaolinite, and opal. Eight mineral spectral target detection algorithms were applied to the preprocessed satellite data with a proposed local spectral library. We measured the highest overall accuracy of 87% for alunite, 95% for opal, 83% for chalcedony, 60% for hematite, and 96% for kaolinite out of these eight algorithms. Three, four, five, and eight algorithms were fused to extract mineral alteration with the obtained target detection results. The results prove that the fusion of algorithms gives better results than using individual ones. In conclusion, this paper discusses the significance of evaluating different mapping algorithms. It proposes a robust fusion approach to extract mineral maps as an indicator for geothermal exploration.

Depth-Guided Robust and Fast Point Cloud Fusion NeRF for Sparse Input Views

Novel-view synthesis with sparse input views is important for real-world applications like AR/VR and autonomous driving. Recent methods have integrated depth information into NeRFs for sparse input synthesis, leveraging depth prior for geometric and spatial understanding. However, most existing works tend to overlook inaccuracies within depth maps and have low time efficiency. To address these issues, we propose a depth-guided robust and fast point cloud fusion NeRF for sparse inputs. We perceive radiance fields as an explicit voxel grid of features. A point cloud is constructed for each input view, characterized within the voxel grid using matrices and vectors. We accumulate the point cloud of each input view to construct the fused point cloud of the entire scene. Each voxel determines its density and appearance by referring to the point cloud of the entire scene. Through point cloud fusion and voxel grid fine-tuning, inaccuracies in depth values are refined or substituted by those from other views. Moreover, our method can achieve faster reconstruction and greater compactness through effective vector-matrix decomposition. Experimental results underline the superior performance and time efficiency of our approach compared to state-of-the-art baselines.

A Robust Mutual-Reinforcing Framework for 3D Multi-Modal Medical Image Fusion Based on Visual-Semantic Consistency

This work proposes a robust 3D medical image fusion framework to establish a mutual-reinforcing mechanism between visual fusion and lesion segmentation, achieving their double improvement. Specifically, we explore the consistency between vision and semantics by sharing feature fusion modules. Through the coupled optimization of the visual fusion loss and the lesion segmentation loss, visual-related and semantic-related features will be pulled into the same domain, effectively promoting accuracy improvement in a mutual-reinforcing manner. Further, we establish the robustness guarantees by constructing a two-level refinement constraint in the process of feature extraction and reconstruction. Benefiting from full consideration for common degradations in medical images, our framework can not only provide clear visual fusion results for doctor's observation, but also enhance the defense ability of lesion segmentation against these negatives. Extensive evaluations of visual fusion and lesion segmentation scenarios demonstrate the advantages of our method in terms of accuracy and robustness. Moreover, our proposed framework is generic, which can be well-compatible with existing lesion segmentation algorithms and improve their performance. The code is publicly available at https://github.com/HaoZhang1018/RMR-Fusion.

A robust data-model dual-driven fusion with uncertainty estimation for LiDAR–IMU localization system

Accurate and robust localization is a critical requirement for autonomous driving and intelligent robots, particularly in complex dynamic environments and various motion scenarios. However, existing LiDAR odometry methods often struggle to promptly respond to changes in the surroundings and motion conditions with fixed parameters through execution, hindering their ability to adaptively adjust system model parameters. Additionally, current localization techniques frequently overlook the confidence level associated with their pose results, leading the autonomous systems to unconditionally accept estimated outputs, even when they may be erroneous. In this paper, we propose a robust data-model dual-driven fusion with uncertainty estimation for the LiDAR–IMU localization system, which integrates the advantages of data-driven and model-driven approaches. We introduce data-driven feature encoder modules for LiDAR and IMU raw data, enabling the system to detect changes in the environment and motion status. Subsequently, these data-driven findings are incorporated into a filtering based model, allowing for the adaptive refinement of system model parameters. Furthermore, we refine the representation of uncertainty based on the Extended-Kalman-Filter model covariance, integrating uncertainty from sensor data and model parameters, which helps to evaluate the confidence of fusion system results. We conducted extensive experiments on two publicly available datasets and one dataset we collected with three sequences, verifying the accuracy of our method. In addition, we have demonstrated the robustness of our method in different motion states and scenarios through comparative experiments, as well as the effectiveness of our refined uncertainty estimation, compared with existing great methods, such as Fast-LIO2 and LIO-SAM, the localization accuracy has been improved by 8.3%.