Multiple Sensor Fusion Research Articles

Deep learning in standard least-squares theory of linear models: Perspective, development and vision

Inspired by the attractive features of least-squares theory in many practical applications, this contribution introduces least-squares-based deep learning (LSBDL). Least-squares theory connects explanatory variables to predicted variables, called observations, through a linear(ized) model in which the unknown parameters of this relation are estimated using the principle of least-squares. Conversely, deep learning (DL) methods establish nonlinear relationships for applications where predicted variables are unknown (nonlinear) functions of explanatory variables. This contribution presents the DL formulation based on least-squares theory in linear models. As a data-driven method, a network is trained to construct an appropriate design matrix of which its entries are estimated using two descent optimization methods: steepest descent and Gauss–Newton. In conjunction with interpretable and explainable artificial intelligence, LSBDL leverages the well-established least-squares theory for DL applications through the following three-fold objectives: (i) Quality control measures such as covariance matrix of predicted outcome can directly be determined. (ii) Available least-squares reliability theory and hypothesis testing can be established to identify mis-specification and outlying observations. (iii) Observations’ covariance matrix can be exploited to train a network with inconsistent, heterogeneous and statistically correlated data. Three examples are presented to demonstrate the theory. The first example uses LSBDL to train coordinate basis functions for a surface fitting problem. The second example applies LSBDL to time series forecasting. The third example showcases a real-world application of LSBDL to downscale groundwater storage anomaly data. LSBDL offers opportunities in many fields of geoscience, aviation, time series analysis, data assimilation and data fusion of multiple sensors.

Spatial coding metastructure for single-sensor impact region recognition

Abstract The rapid development of aircraft has created a strong demand for structural health monitoring, but current methods that rely on multiple sensor fusion suffer from complex hardware systems. Computational sensing with metastructures provides a promising approach to reduce sensing cost, but the lack of calibrated information makes it challenging to identify impact regions. In this study, we propose a concept of spatial coding metastructure for impact region recognition with a single sensor. Owing to the multi-order local resonance effect, the metastructures are capable of producing multiple vibration modulations over a wide frequency band. We demonstrate that the frequency-dependent vibration modulation effects of the metastructures on different test regions are distinguishable, a characteristic referred to as spatial coding. This characteristic enables impact regions to be accurately recognized with only a single sensor by using machine learning methods. Our work not only presents promising application prospects for condition monitoring of aircraft and other mechanical systems, but also inspires the development of safer and more efficient systems in various industries.

Rapid SLAM Method for Star Surface Rover in Unstructured Space Environments

The space environment is characterized by unstructured features, sparsity, and poor lighting conditions. The difficulty in extracting features from the visual frontend of traditional SLAM methods results in poor localization and time-consuming issues. This paper proposes a rapid and real-time localization and mapping method for star chart surveyors in unstructured space environments. Improved localization is achieved using multiple sensor fusion to sense the space environment. We replaced the traditional feature extraction module with an enhanced SuperPoint feature extraction network to tackle the challenge of challenging feature extraction in unstructured space environments. By dynamically adjusting detection thresholds, we achieved uniform detection and description of image keypoints, ultimately resulting in robust and accurate feature association information. Furthermore, we minimized redundant information to achieve precise positioning with high efficiency and low power consumption. We established a star surface rover simulation system and created simulated environments resembling Mars and the lunar surface. Compared to the LVI-SAM system, our method achieved a 20% improvement in localization accuracy for lunar scenarios. In Mars scenarios, our method achieved a positioning accuracy of 0.716 m and reduced runtime by 18.682 s for the same tasks. Our approach exhibits higher localization accuracy and lower power consumption in unstructured space environments.

Smart Nursing Wheelchairs: A New Trend in Assisted Care and the Future of Multifunctional Integration.

As a significant technological innovation in the fields of medicine and geriatric care, smart care wheelchairs offer a novel approach to providing high-quality care services and improving the quality of care. The aim of this review article is to examine the development, applications and prospects of smart nursing wheelchairs, with particular emphasis on their assistive nursing functions, multiple-sensor fusion technology, and human-machine interaction interfaces. First, we describe the assistive functions of nursing wheelchairs, including position changing, transferring, bathing, and toileting, which significantly reduce the workload of nursing staff and improve the quality of care. Second, we summarized the existing multiple-sensor fusion technology for smart nursing wheelchairs, including LiDAR, RGB-D, ultrasonic sensors, etc. These technologies give wheelchairs autonomy and safety, better meeting patients' needs. We also discussed the human-machine interaction interfaces of intelligent care wheelchairs, such as voice recognition, touch screens, and remote controls. These interfaces allow users to operate and control the wheelchair more easily, improving usability and maneuverability. Finally, we emphasized the importance of multifunctional-integrated care wheelchairs that integrate assistive care, navigation, and human-machine interaction functions into a comprehensive care solution for users. We are looking forward to the future and assume that smart nursing wheelchairs will play an increasingly important role in medicine and geriatric care. By integrating advanced technologies such as enhanced artificial intelligence, intelligent sensors, and remote monitoring, we expect to further improve patients' quality of care and quality of life.

PPP based on factor graph optimization

Abstract Kalman or Kalman-related filtering methods are routinely applied in precise point positioning (PPP). However, in robot simultaneous localization and mapping (SLAM) systems, the factor graph optimization (FGO) has proved advantages over filtering methods in recent years, e.g., reducing the linearization errors and support of plug-and-play feature for multiple sensor fusion. Therefore, it would be interesting to apply the FGO to PPP. In addition, it will also facilitate the tight integration of PPP with Visual/LiDAR SLAM. In this work, PPP is solved under the factor graph optimization framework. A factor graph for PPP has been constructed. Results from 268 IGS-MGEX stations show that the factor graph optimization method can achieve a similar performance with that of Kalman filtering. First, the positioning accuracy in the convergence period can be improved for PPP based on factor graph optimization because it optimizes the entire state variables based on all the available observations. For applications that do not require real-time processing, the observation after the current states, e.g., future observations, can also be used to enhance the current state estimation. Second, the accuracy of static PPP is almost the same for the two methods with millimeter-accuracy for horizontal directions and centimeter-accuracy for vertical directions. Third, the kinematic PPP for both methods can achieve centimeter-level accuracy in horizontal directions and decimeter-level accuracy in vertical directions. Although the performance is comparable, it is noted that the computational efficiency of factor graph optimization method is still a problem. For each epoch, the average of elapsed time for Kalman filtering is 132 microseconds, while that of factor graph optimization method is 9664 microseconds. The elapsed time of factor graph optimization method can be further improved if the fix-window optimization technique is applied, which will be investigated in the future.

IRBEVF-Q: Optimization of Image-Radar Fusion Algorithm Based on Bird's Eye View Features.

In autonomous driving, the fusion of multiple sensors is considered essential to improve the accuracy and safety of 3D object detection. Currently, a fusion scheme combining low-cost cameras with highly robust radars can counteract the performance degradation caused by harsh environments. In this paper, we propose the IRBEVF-Q model, which mainly consists of BEV (Bird's Eye View) fusion coding module and an object decoder module.The BEV fusion coding module solves the problem of unified representation of different modal information by fusing the image and radar features through 3D spatial reference points as a medium. The query in the object decoder, as a core component, plays an important role in detection. In this paper, Heat Map-Guided Query Initialization (HGQI) and Dynamic Position Encoding (DPE) are proposed in query construction to increase the a priori information of the query. The Auxiliary Noise Query (ANQ) then helps to stabilize the matching. The experimental results demonstrate that the proposed fusion model IRBEVF-Q achieves an NDS of 0.575 and a mAP of 0.476 on the nuScenes test set. Compared to recent state-of-the-art methods, our model shows significant advantages, thus indicating that our approach contributes to improving detection accuracy.

Smartphone-based Vision/MEMS-IMU/GNSS tightly coupled seamless positioning using factor graph optimization

With the diversification of functionalities and enhancement of performance in smartphones, low-cost GNSS chips, Micro-Electro-Mechanical System (MEMS) inertial measurement units (IMU), and cameras have become essential hardware components. Moreover, the computational power provided by smartphones for data processing has also become more robust. The fusion of multiple sensors in smartphones for positioning has emerged as a trend. In order to further enhance the positioning performance of smartphones in complex environments, this paper proposes a smartphone-based Vision/MEMS-IMU/GNSS tightly coupled integration for indoor-outdoor seamless positioning. Factor graph optimization is employed to combine observations from monocular vision, MEMS-IMU, and GNSS. To solve the unstable timing of current smartphone sensors, we present a GNSS-to-VIO time synchronization method. Furthermore, an adaptive positioning mechanism is devised based on the presence or absence of GNSS signals, allowing indoor and outdoor environment to switch. The proposed method is validated by integrating smartphone VIO with both smartphone built-in GNSS and high-performance geodetic GNSS receivers. The test results of the complex outdoor environments demonstrate that the proposed method significantly enhances GNSS positioning performance. The positioning accuracy of the proposed method using geodetic GNSS receiver is superior to that using smartphone GNSS. In outdoor environments, our method enables positioning accuracy of smartphone to improve 29.4, 53.0 and 69.9 % in three directions compared to single point positioning. Indoor and outdoor tests verified that the proposed method can provide continuous and reliable positioning in alternating interrupted and restored GNSS signals.

Sequential centralized fusion of multiple passive acoustic sensors with unknown propagation delays

In this paper, we address the problem of target tracking using multiple acoustic sensors to observe the target state with unknown propagation delays. This problem occurs because the measurements received by multiple acoustic sensors located at different positions are from different unknown emission times of the acoustic signal even if the sensors receive measurements simultaneously; thus, they cannot be stacked up directly for centralized fusion as usual. However, the target states at different unknown emission times can be aligned to a common measurement received time by the retrodiction of state prediction. On this basis, herein we propose the centralized fusion of multiple acoustic sensors via sequential processing, namely, sequential centralized fusion (SCF). First, the measurement received time is chosen as the target state time, and the target state is predicted to this time for tracking. Second, state prediction is retrodicted to the signal emission times by solving augmented implicit nonlinear equations through Wegstein's method. Third, the state prediction is updated with acoustic measurements sequentially at measurement received time. Compared with the existing distributed fusion methods, our proposed SCF method has smaller computational complexity and better tracking performance. Illustrative examples demonstrate that SCF outperforms covariance intersection and the largest ellipsoid approximation.

Sensor Fusion-Based Anthropomorphic Control of a Robotic Arm.

The main goal of this research is to develop a highly advanced anthropomorphic control system utilizing multiple sensor technologies to achieve precise control of a robotic arm. Combining Kinect and IMU sensors, together with a data glove, we aim to create a multimodal sensor system for capturing rich information of human upper body movements. Specifically, the four angles of upper limb joints are collected using the Kinect sensor and IMU sensor. In order to improve the accuracy and stability of motion tracking, we use the Kalman filter method to fuse the Kinect and IMU data. In addition, we introduce data glove technology to collect the angle information of the wrist and fingers in seven different directions. The integration and fusion of multiple sensors provides us with full control over the robotic arm, giving it flexibility with 11 degrees of freedom. We successfully achieved a variety of anthropomorphic movements, including shoulder flexion, abduction, rotation, elbow flexion, and fine movements of the wrist and fingers. Most importantly, our experimental results demonstrate that the anthropomorphic control system we developed is highly accurate, real-time, and operable. In summary, the contribution of this study lies in the creation of a multimodal sensor system capable of capturing and precisely controlling human upper limb movements, which provides a solid foundation for the future development of anthropomorphic control technologies. This technology has a wide range of application prospects and can be used for rehabilitation in the medical field, robot collaboration in industrial automation, and immersive experience in virtual reality environments.

Hierarchical Attention-Based Sensor Fusion Strategy for Depth Estimation in Diverse Weather

In this paper, we propose a hierarchical attention-based sensor fusion strategy for depth estimation under various weather conditions. Multiple-sensor fusion is proven as a promising solution for predicting accurate depth maps in diverse weather conditions, especially for extreme weather conditions. However, most of the current studies simply fuse the information from different sensors without jointly considering the difference in their performance at the sensor level and feature level. To fill this gap, our hierarchical attention-based fusion strategy uses two attention mask-generation modules that weigh sensor data from branches (i.e. different sensors) and features. With the cooperation of these two masks, our system is able to determine the adaptive contribution of each sensor as well as the individual contribution of each feature in the sensor regarding their performance in different weather. We compare the proposed methods with the baseline, i.e. the late fusion Sparse-to-Dense model, and two extended models individually with the branch-wise-only and feature-wise-only masks. The results show a robust and superior performance of our methods even in clear environments where the baseline already performs well enough. Moreover, we investigate the performance of RGB camera, radar, and LiDAR in foggy environments comprehensively by visualizing the generated mask. Our results show a significantly increased importance of radar sensors in extreme weather conditions, e.g. dense fog.

Optimal Linear Multilateration Combined With the Kalman Filter for Range-Only Tracking

This paper presents the optimal estimation for the extensively concerned linear multilateral positioning issues and further improves the accuracy of range-only tracking significantly. The traditional linear multilateration method has been used to achieve target positioning for range-only measurements, but the completeness of the theory and implementation effect are limited due to the imperfect model and the lack of proper statistical error analysis. Moreover, the performance of the representative nonlinear filters is not satisfactory because of the strong nonlinearity between the range observations and kinematic states of the moving target. This paper reveals the essence of the linear fusion of multiple distance sensors from a geometric perspective. The multi-sensor fusion model is reconstructed in the linear feature space by transforming the likelihood function into the probability function and separating the expectation of the compound random variable and residual. After that, a new linear multilateration method is developed based on the precise derivation of statistical characteristics corresponding to the linear fusion model, and also minimizing the squared Mahalanobis distance is introduced to replace minimizing the mean square error. The first-order moment, as a pseudo-measurement used to improve the positioning accuracy, and the second-order moment, as the covariance of the pseudo-measurement noise, are provided by this method, which has been proved to be best linear unbiased according to the Gauss–Markov theorem. Combined with the pseudo-measurement, a standard linear Kalman filter is capable of tracking the maneuvering target. Several simulations and experiments verify the superior performance of the proposed method.

Multiple‐model generalised labelled multi‐Bernoulli filter with distributed sensors for tracking manoeuvring targets using belief propagation

AbstractThe multi‐sensor multiple‐model generalised labelled multi‐Bernoulli filter (MS‐MM‐GLMB) is presented for tracking multiple manoeuvring targets. And we develop efficient implementation for computing multi‐target posterior. The proposed implementation of the MS‐MM‐GLMB filter is composed of two main parts including centralised fusion with distributed sensors and posterior calculation using belief propagation. Specifically, the centralised fusion with distributed sensors is performed based on common predicted information to enable a full parallelisation. This fusion strategy yields the exact solution along with independence of the sensor ordering. More importantly, this fusion allows multiple sensor fusion with the MM‐GLMB filters for iterative update step for synchronous measurements. Subsequently, the belief propagation algorithm is adopted to implement the MS‐MM‐GLMB filter by efficiently calculating the marginal posterior distributions of GLMB form with superior tracking performance. The belief propagation solution to solving data association is presented for the computationally efficient centralised tracking of the manoeuvring targets. Compared with Gibbs‐based implementation, the proposed belief‐propagation‐based method performs well or better, especially in more challenging scenario.

A Clinical Perspective on Bespoke Sensing Mechanisms for Remote Monitoring and Rehabilitation of Neurological Diseases: Scoping Review.

Neurological diseases including stroke and neurodegenerative disorders cause a hefty burden on the healthcare system. Survivors experience significant impairment in mobility and daily activities, which requires extensive rehabilitative interventions to assist them to regain lost skills and restore independence. The advent of remote rehabilitation architecture and enabling technology mandates the elaboration of sensing mechanisms tailored to individual clinical needs. This study aims to review current trends in the application of sensing mechanisms in remote monitoring and rehabilitation in neurological diseases, and to provide clinical insights to develop bespoke sensing mechanisms. A systematic search was performed using the PubMED database to identify 16 papers published for the period between 2018 to 2022. Teleceptive sensors (56%) were utilized more often than wearable proximate sensors (50%). The most commonly used modality was infrared (38%) and acceleration force (38%), followed by RGB color, EMG, light and temperature, and radio signal. The strategy adopted to improve the sensing mechanism included a multimodal sensor, the application of multiple sensors, sensor fusion, and machine learning. Most of the stroke studies utilized biofeedback control systems (78%) while the majority of studies for neurodegenerative disorders used sensors for remote monitoring (57%). Functional assessment tools that the sensing mechanism may emulate to produce clinically valid information were proposed and factors affecting user adoption were described. Lastly, the limitations and directions for further development were discussed.

Extending the dynamic response of structural input estimation by combining sensors with unmatched bandwidth

The estimation of external loads acting on mechanic structures is of paramount importance for the design, operation, and control of these systems. In many cases, external loads cannot be measured directly due to the intrusive nature or the high costs of adequate sensors. Model-based estimation techniques yield the possibility to estimate external loads in an indirect manner using indirect measurements. These methods usually assume an optimal sensor behaviour over the entire frequency range of the estimator application, which not necessarily has to be the case due to sensor bandwidth limitations. Therefore, model-based sensor fusion is constrained to sensors and applications that share the same bandwidth. This paper proposes a novel non-intrusive approach to estimating external loading in the presence of incongruent measurement regimes of the involved sensors using Kalman filters. The method uses existing model-based estimation techniques to establish a broadly applicable framework for the fusion of multiple sensors with non-coinciding measurement bandwidths. A promising application of this new approach is not only the combination of sensors that measure the same quantity in different frequency regimes but also the combination of sensor types which are not being used together due to different measurement regimes. The newly formulated approach is tested both numerically and with a validation experiment in which accelerometer and gyroscope measurement are combined. In both cases, the new method shows improved behaviour in terms of estimations quality of an unknown force over traditional Kalman filter formulations in the presence of prominent sensor dynamics and over a wide frequency range.

Multi-sensor based strategy learning with deep reinforcement learning for unmanned ground vehicle

As intelligent Unmanned Ground Vehicles (UGVs) find broader applications in areas such as transportation and logistics. The fusion of multiple sensors becomes crucial, since it not only amplifies UGV perception in dynamic scenarios but also underpins their autonomous decision-making capabilities. However, many existing methods only focus on single-sensor data, overlooking the multi-sensor data integration, thereby limiting UGV's scalability and adaptability. In this paper, we introduce the Multi-Sensor Collaborative Decision Network (MSCDN) for autonomous multi-sensor fusion policy learning designed specifically for UGVs. MSCDN is dedicated to integrate the data collected by multi-sensors in simulation environment and can be migrate to real environment. Firstly, a simulation environment mirroring real environment is created, using a framework that transfers UGV decision-making from simulated to real environment with deep reinforcement learning. Secondly, MSCDN uses a multi-sensor attention fusion network to adaptively integrate sensor data, refining UGV responses in dynamic settings. Thirdly, MSCDN's efficacy is tested on both simulated and real UGV lane-keeping tasks, showcasing its superior performance in comparative experiments. Compared to baseline methods, MSCDN reduces training steps and achieves a 35.71 % higher success rate and a 37.5 % quicker task completion time, underlining its proficient multi-sensor data fusion capability.

Full-body pose estimation for excavators based on data fusion of multiple onboard sensors

To reduce machine-related accidents on sites, automatically monitoring the full-body poses of operating heavy machines is crucial. Conventional pose estimation systems relying on homogeneous sensors are vulnerable to negative environmental impacts, leading to inaccurate and unstable estimation of machine states. Hence, a full-body pose estimation framework is proposed for excavators, with a data fusion strategy to utilize different types of onboard sensors for enhanced accuracy and robustness. Specifically, a non-invasive onboard visual-inertial sensor system is designed for data fusion. Then, through competitive and complementary data fusion, the keypoints describing the full-body poses of the excavator are tracked in 3D space. Especially, an EKF-based localization algorithm is developed for optimized multi-keypoint tracking, which is verified to improve the accuracy and robustness of pose estimation by a real-world excavator case study. The proposed sensor-fusion method can effectively improve operational safety, by accurately monitoring the motion of heavy machines operating on construction sites.

Adaptive wavelet pyramid for noisy machinery fault diagnosis with multiple sensors

The fusion of multiple monitoring sensors is crucial to improve the accuracy and robustness of machinery fault diagnosis. However, existing fault diagnosis methods may underestimate the interference of noise in the multi-sensor fusion process, leading to unsatisfied performance. To handle this problem, this paper proposes a deep model based on the frequency adaptive wavelet pyramid. First, an adaptive frequency selection strategy is designed to prune the seriously polluted frequencies and only retain some key frequencies. Then, the self-attention mechanism is used to perform information fusion on the selected frequency bands of different sensors. Finally, a wavelet fusion pyramid is adopted by repeating the fusion process at multiple wavelet decomposition levels. In this way, different sensors can be fused in a more fine-grained manner. The experimental results on two multi-sensor-based fault diagnosis datasets demonstrate the anti-noise capability of the proposed method.

Multilayer Low-Cost Sensor Local-Global Filtering Fusion Integrated Navigation of Small UAV

Aimed at improving the nonlinear integrated navigation solution performance of multiple low-cost sensors fusion, this paper presents a multilayer loosely-coupled, local-global, and step-optimized MF5DCKF (Multisensor Federated fifth-degree Cubature Kalman filter) state estimation algorithm for the small unmanned aerial vehicle (UAV). This method establishes a multilayer nonlinear integrated navigation model composed of the nonlinear attitude and heading reference system (AHRS) error model, strapdown inertial navigation system/global positioning system (SINS/GPS) error model, and strapdown inertial navigation system/barometer (SINS/BARO) error model to enhance the robustness and richness of the navigation module. Further, based on the above navigation models, a loosely-coupled error state fusion frame is designed to obtain the local convergent state vector. Simultaneously, a three-layer fifth-degree Cubature Kalman filter is proposed to improve the local state estimation accuracy. Subsequently, to optimize the estimated local state, this paper presents a novel distributed MF5DCKF scheme fusing the local state vector to calculate the global optimal state parameters in a step-optimized process. The experimental flight test results show that the proposed algorithm achieves a higher state solution accuracy and a better convergent performance compared with some conventional multisensor fusion algorithms. The new algorithm framework can provide applicability and reliability for the small UAV during the flight.

A Sensor Fusion Method Using Transfer Learning Models for Equipment Condition Monitoring.

Sensor fusion is becoming increasingly popular in condition monitoring. Many studies rely on a fusion-level strategy to enable the most effective decision-making and improve classification accuracy. Most studies rely on feature-level fusion with a custom-built deep learning architecture. However, this may limit the ability to use the widely available pre-trained deep learning architectures available to users today. This study proposes a new method for sensor fusion based on concepts inspired by image fusion. The method enables the fusion of multiple and heterogeneous sensors in the time-frequency domain by fusing spectrogram images. The method’s effectiveness is tested with transfer learning (TL) techniques on four different pre-trained convolutional neural network (CNN) based model architectures using an original test environment and data acquisition system. The results show that the proposed sensor fusion technique effectively classifies device faults and the pre-trained TL models enrich the model training capabilities.

An encoded reconfigurable RFID strain sensor and its information fusion method

Chip-based radio frequency identification (RFID) sensor systems have been investigated for structural health monitoring applications. However, the use of chips in sensor tags has drawbacks in terms of cost, durability, and detection capability. In this paper, we propose a reconfigurable RFID-based chipless tag sensor for strain detection, which detects structural strain by observing the offset of the resonant frequency of the antenna and incorporates an encoding unit for tag identification. To realize the reconfigurability of the antenna, a PIN diode is added to control the antenna, and the frequency reconfigurability of the antenna is realized by controlling the turn on/off of the PIN diodes. The Kalman filter algorithm is used to realize the information fusion of multiple sensors to improve the sensor detection accuracy, and the background noise data enhancement is used to expand the original data samples to fuse the information of the three sensors’ data, and the relative error after fusion is reduced by about 36% compared with the relative error before fusion compared with separate measurements. The antenna was fabricated by an etching process and measured using vector network analyzer to verify the accuracy of the antenna simulation results.