Sensor Fusion Problem Research Articles

RoVaR

The plethora of sensors in our commodity devices provides a rich substrate for sensor-fused tracking. Yet, today's solutions are unable to deliver robust and high tracking accuracies across multiple agents in practical, everyday environments - a feature central to the future of immersive and collaborative applications. This can be attributed to the limited scope of diversity leveraged by these fusion solutions, preventing them from catering to the multiple dimensions of accuracy, robustness (diverse environmental conditions) and scalability (multiple agents) simultaneously. In this work, we take an important step towards this goal by introducing the notion of dual-layer diversity to the problem of sensor fusion in multi-agent tracking. We demonstrate that the fusion of complementary tracking modalities, - passive/relative (e.g. visual odometry) and active/absolute tracking (e.g.infrastructure-assisted RF localization) offer a key first layer of diversity that brings scalability while the second layer of diversity lies in the methodology of fusion, where we bring together the complementary strengths of algorithmic (for robustness) and data-driven (for accuracy) approaches. ROVAR is an embodiment of such a dual-layer diversity approach that intelligently attends to cross-modal information using algorithmic and data-driven techniques that jointly share the burden of accurately tracking multiple agents in the wild. Extensive evaluations reveal ROVAR'S multi-dimensional benefits in terms of tracking accuracy, scalability and robustness to enable practical multi-agent immersive applications in everyday environments.

Sequential State and Unknown Parameter Estimation Strategy and its Application to a Sensor Fusion Problem

Soft-sensing and state-space-based methods play crucial roles in fault-tolerant sensor fusion schemes. Kalman filtering is one of the most employed sensor fusion methods. In practice, filtering (or state estimation) is sensitive to initial conditions, variations in model parameters, and faults in sensors and actuators. Cumulatively, these uncertainties that affect the state estimator can be modeled as unknown parameters. In this article, we present the development of a discrete-time minimum variance unbiased estimator (MVUE) that is robust to a class of unknown parameters. The estimation gain of MVUE is designed to facilitate unbiased state estimates in the presence of unknown parameters. We then use the innovations that are generated by the proposed state estimator (MVUE) to estimate the unknown parameters, thus proposing a sequential state and unknown parameter estimation strategy. The development of the decoupled state and unknown parameter estimation strategy is motivated by the fault-tolerant sensor fusion exercise for railway track geometry inspection. This exercise aims to formulate the state-space realization of the track geometry inspection problem to enable sequential state and unknown parameter estimation. The track geometry parameters, initial conditions, and sensor faults are considered unknown parameters for the application. The simulation analysis shows the importance of developing the trinity of the state-space realization for track geometry inspection, the state estimation framework, and the decoupled unknown parameter estimator.

Optimized data rate allocation for dynamic sensor fusion over resource constrained communication networks

AbstractThis article presents a new method to solve a dynamic sensor fusion problem. We consider a large number of remote sensors which measure a common Gauss–Markov process. Each sensor encodes and transmits its measurement to a data fusion center through a resource restricted communication network. The communication cost incurred by a given sensor is quantified as the expected bitrate from the sensor to the fusion center. We propose an approach that attempts to minimize a weighted sum of these communication costs subject to a constraint on the state estimation error at the fusion center. We formulate the problem as a difference‐of‐convex program and apply the convex‐concave procedure (CCP) to obtain a heuristic solution. We consider a 1D heat transfer model and a model for 2D target tracking by a drone swarm for numerical studies. Through these simulations, we observe that our proposed approach has a tendency to assign zero data rate to unnecessary sensors indicating that our approach is sparsity‐promoting, and an effective sensor selection heuristic.

Learning a Joint Embedding of Multiple Satellite Sensors: A Case Study for Lake Ice Monitoring

Fusing satellite imagery acquired with different sensors has been a long-standing challenge of Earth observation, particularly across different modalities such as optical and Synthetic Aperture Radar (SAR) images. Here, we explore the joint analysis of imagery from different sensors in the light of representation learning: we propose to learn a joint embedding of multiple satellite sensors within a deep neural network. Our application problem is the monitoring of lake ice on Alpine lakes. To reach the temporal resolution requirement of the Swiss Global Climate Observing System (GCOS) office, we combine three image sources: Sentinel-1 SAR (S1-SAR), Terra MODIS, and Suomi-NPP VIIRS. The large gaps between the optical and SAR domains and between the sensor resolutions make this a challenging instance of the sensor fusion problem. Our approach can be classified as a late fusion that is learned in a data-driven manner. The proposed network architecture has separate encoding branches for each image sensor, which feed into a single latent embedding. I.e., a common feature representation shared by all inputs, such that subsequent processing steps deliver comparable output irrespective of which sort of input image was used. By fusing satellite data, we map lake ice at a temporal resolution of < 1.5 days. The network produces spatially explicit lake ice maps with pixel-wise accuracies > 91% (respectively, mIoU scores > 60%) and generalises well across different lakes and winters. Moreover, it sets a new state-of-the-art for determining the important ice-on and ice-off dates for the target lakes, in many cases meeting the GCOS requirement.

A Quaternion Gated Recurrent Unit Neural Network for Sensor Fusion

Recurrent Neural Networks (RNNs) are known for their ability to learn relationships within temporal sequences. Gated Recurrent Unit (GRU) networks have found use in challenging time-dependent applications such as Natural Language Processing (NLP), financial analysis and sensor fusion due to their capability to cope with the vanishing gradient problem. GRUs are also known to be more computationally efficient than their variant, the Long Short-Term Memory neural network (LSTM), due to their less complex structure and as such, are more suitable for applications requiring more efficient management of computational resources. Many of such applications require a stronger mapping of their features to further enhance the prediction accuracy. A novel Quaternion Gated Recurrent Unit (QGRU) is proposed in this paper, which leverages the internal and external dependencies within the quaternion algebra to map correlations within and across multidimensional features. The QGRU can be used to efficiently capture the inter- and intra-dependencies within multidimensional features unlike the GRU, which only captures the dependencies within the sequence. Furthermore, the performance of the proposed method is evaluated on a sensor fusion problem involving navigation in Global Navigation Satellite System (GNSS) deprived environments as well as a human activity recognition problem. The results obtained show that the QGRU produces competitive results with almost 3.7 times fewer parameters compared to the GRU.

Encoder-Camera-Ground Penetrating Radar Sensor Fusion: Bimodal Calibration and Subsurface Mapping

In this article, we report system and algorithmic developments for a sensing suite comprising a camera and a ground penetrating radar (GPR) with a wheel encoder designed for both surface and subsurface infrastructure inspection, which is a multimodal mapping task. To fuse different sensor modalities properly, we solve a novel GPR-camera calibration problem and a synchronization-challenged sensor fusion problem. First, we design a calibration rig, model the GPR imaging process, introduce a mirror to obtain the joint coverage between the camera and the GPR, and employ the maximum-likelihood estimator to estimate the relative pose between the camera and the GPR with error analysis. Second, we propose a data collection scheme using the customized artificial landmarks to synchronize camera images (temporally evenly spaced) and GPR/encoder data (spatially evenly spaced). We also employ pose graph optimization with location discrepancy as penalty functions to perform data fusion for 3-D reconstruction. We have tested our system in physical experiments. The results show that our system successfully fuses encoder-camera-GPR sensory data and accomplishes a metric 3-D reconstruction. Moreover, our sensor fusion approach reduces the end-to-end distance error from 6.4 to 0.7 cm in a real bridge inspection experiment if comparing to the counterpart that only uses encoder measurements.

Heterogeneous sensor fusion based on Copula theory and importance sampling

This study deals with the problem of heterogeneous sensor fusion based on Copula theory and importance sampling. The proposed fusion algorithm is grounded on Copula statistical modelling and Bayesian probabilistic theory. The distinctive advantage of this Copula-based methodology is that it formulates the internal dependency between the local sensors' data, which is usually unknown but essential for accurate track fusion. To this end, a joint distribution of the local sensors' observations is constructed based on Copula functions, and the corresponding fusion rule is derived with a specific correlation term. In addition, a Monte–Carlo importance sampling technique is adopted to improve the computational efficiency by drawing less random samples from the local estimates to be fused. After that, a procedure of Kernel density estimation is applied to learn a Gaussian approximation of the fused density. In the end, extensive Monte–Carlo simulations are conducted to evaluate the proposed sensor fusion method in a distributed target-tracking scenario.

A Joint Optimization Approach of LiDAR-Camera Fusion for Accurate Dense 3-D Reconstructions

Fusing data from LiDAR and camera is conceptually attractive because of their complementary properties. For instance, camera images are of higher resolution and have colors, while LiDAR data provide more accurate range measurements and have a wider field of view. However, the sensor fusion problem remains challenging since it is difficult to find reliable correlations between data of very different characteristics (geometry versus texture, sparse versus dense). This letter proposes an offline LiDAR-camera fusion method to build dense, accurate 3-D models. Specifically, our method jointly solves a bundle adjustment problem and a cloud registration problem to compute camera poses and the sensor extrinsic calibration. In experiments, we show that our method can achieve an average accuracy of 2.7 mm and resolution of 70 points/cm 2 by comparing to the ground truth data from a survey scanner. Furthermore, the extrinsic calibration result is discussed and shown to outperform the state-of-the-art method.

Cooperative Visual-Inertial Sensor Fusion: The Analytic Solution

This letter analyzes the visual–inertial sensor fusion problem in the cooperative case of two agents, and proves that this sensor fusion problem is equivalent to a simple polynomial equations system that consists of several linear equations and three polynomial equations of second degree. The analytic solution of this polynomial equations system is easily obtained by using an algebraic method. In other words, this letter provides the analytic solution to the visual–inertial sensor fusion problem in the case of two agents. The power of the analytic solution is twofold. From one side, it allows us to determine the relative state between the agents (i.e., relative position, speed, and orientation) without the need of an initialization. From another side, it provides fundamental insights into all the theoretical aspects of the problem. This letter mainly focuses on the first issue. However, the analytic solution is also exploited to obtain basic structural properties of the problem that characterize the observability of the absolute scale and the relative orientation. Extensive simulations and real experiments show that the solution is successful in terms of precision and robustness.

ConFusion: Sensor Fusion for Complex Robotic Systems Using Nonlinear Optimization

We present ConFusion, an open-source package for online sensor fusion for robotic applications. ConFusion is a modular framework for fusing measurements from many heterogeneous sensors within a moving horizon estimator. ConFusion offers greater flexibility in sensor fusion problem design than filtering-based systems and the ability to scale the online estimate quality with the available computing power. We demonstrate its performance in comparison to an iterated extended Kalman filter in visual-inertial tracking, and show its versatility through whole-body sensor fusion on a mobile manipulator.

Cooperative visual-inertial sensor fusion: fundamental equations and state determination in closed-form

This paper investigates the visual and inertial sensor fusion problem in the cooperative case and provides new theoretical and basic results. Specifically, the case of two agents is investigated. Each agent is equipped with inertial sensors (accelerometer and gyroscope) and with a monocular camera. By using the monocular camera, each agent can observe the other agent. No additional camera observations (e.g., of external point features in the environment) are considered. First, the entire observable state is analytically derived. This state contains the relative position between the two agents (which includes the absolute scale), the relative velocity, the three Euler angles that express the rotation between the two local frames and all the accelerometer and gyroscope biases. Then, the basic equations that describe this system are analytically obtained. The last part of the paper describes the use of these equations to obtain a closed-form solution that provides the observable state in terms of the visual and inertial measurements provided in a short time interval. This last contribution is the extension of the results presented in Kaiser et al. (IEEE Robot Autom Lett 2(1):18–25, 2017), Martinelli (IEEE Trans Robot 28(1):44–60, 2012; Int J Comput Vis 106(2):138–152, 2014) to the cooperative case. The impact of the presence of the bias on the performance of this closed-form solution is also investigated and a simple and effective method to obtain the gyroscope bias is proposed. Extensive simulations clearly show that the proposed method is successful. It is worth noting that it is possible to automatically retrieve the absolute scale and simultaneously calibrate the gyroscopes not only without any prior knowledge (as in Kaiser et al. IEEE Robot Autom Lett 2(1):18–25, 2017), but also without external point features in the environment.

Distributed Joint Probabilistic Data Association Filter With Hybrid Fusion Strategy

This paper investigates the problem of distributed multitarget tracking (MTT) over a large-scale sensor network, consisting of low-cost sensors. Each local sensor runs a joint probabilistic data association filter to obtain local estimates and communicates with its neighbors for information fusion. The conventional fusion strategies, i.e., consensus on measurement (CM) and consensus on information (CI), are extended to MTT scenarios. This means that data association uncertainty and sensor fusion problems are solved simultaneously. Motivated by the complementary characteristics of these two different fusion approaches, a novel distributed MTT algorithm using a hybrid fusion strategy, e.g., a mix of CM and CI, is proposed. A distributed counting algorithm is incorporated into the tracker to provide the knowledge of the total number of sensor nodes. The new algorithm developed shows advantages in preserving boundedness of local estimates, guaranteeing global convergence to the optimal centralized version and being implemented without requiring no global information, compared with other fusion approaches. Simulations clearly demonstrate the characteristics and tracking performance of the proposed algorithm.

Asynchronous multi-rate multi-sensor fusion based on random finite set

Abstract This paper considers the asynchronous sensor fusion problem for an arbitrary number of sensors with different sampling rates in the framework of the random finite set (RFS) theory. By sequentially predicting and updating the posteriors with measurements according to the arrival time sequence of measurement sets, a centralized asynchronous fusion algorithm, centralized Sequential Processing (SP), is proposed first. It is optimal due to the usage of original measurement information. Considering the reliability, survivability and communication bandwidth as well as flexibility of output time, two distributed asynchronous fusion algorithms are also proposed by assuming that the process of tracking among sensors is independent. The first distributed asynchronous fusion algorithm, namely Batch Generalization Covariance Intersection (B-GCI), utilizes all the predicted local posteriors (LPs) from different sensors and fuses them simultaneously at the fusion center (FC) based on GCI rule, which avoids the complicated calculations of cross-covariance among the LPs of sensors. Considering the large computational burden of the B-GCI fusion algorithm, another one method, Sequential GCI (S-GCI) fusion algorithm, is proposed. The method sequentially fuses the predicted LPs in pairs based on GCI rule at the FC. Performance of proposed three algorithms, including a centralized fusion algorithm and two distributed fusion algorithms, is realized by using Gaussian Mixture Probability Hypothesis Density (GM-PHD) filter and some numerical simulations are given.

Hierarchical fully adaptive radar

By emulating the cognitive perception-action cycle believed to be at the core of animal cognition, cognitive radars promise to improve radar performance over standard systems. The fully adaptive radar (FAR) framework provides a generalised approach to implementing a single cognitive perception-action cycle for radar systems, but complex adaptive problems necessitate the interaction of multiple perception-action cycles. This study describes the general form of the hierarchical FAR (HFAR) framework. The HFAR framework is applied to a single-target tracking, sensor fusion problem, and real-time experimental results demonstrate the efficacy of the proposed architecture for handling problems of varying scales in a consistent, adaptive fashion.

A Bregman Splitting Scheme for Distributed Optimization Over Networks

We consider distributed optimization problems, in which a group of agents are to collaboratively seek the global optimum through peer-to-peer communication networks. The problem arises in various application areas, such as resource allocation, sensor fusion, and distributed learning. We present a general algorithmic framework based on the Bregman method and operator splitting, which allows us to easily recover most of the existing distributed algorithms. Under this framework, we propose a general efficient distributed algorithm—distributed forward–backward Bregman splitting (D-FBBS)—to simultaneously solve the above primal problem as well as its dual. The proposed algorithm allows agents to communicate asynchronously and, thus, lends itself to stochastic networks. This algorithm is shown to have close connections with some existing well-known algorithms when dealing with fixed networks. However, we will show that it is generally different from the existing ones due to its effectiveness in handling stochastic networks. With proper assumptions, we establish a nonergodic convergence rate of $O(1/k)$ in terms of fixed-point residuals over fixed networks both for D-FBBS and its inexact version (ID-FBBS) that is more computationally efficient and an ergodic convergence rate of $O(1/k)$ for D-FBBS over stochastic networks. We also apply the proposed algorithm to sensor fusion problems to show its superior performance compared to existing methods.

Multirotor UAV sensor fusion for precision landing

PurposeIn relation to rapid development of possible applications of unmanned vehicles, new opportunities for their use are emerging. Among the most dynamic, we can distinguish package shipments, rescue and military applications, autonomous flights and unattended transportation. However, most of the UAV solutions have limitations related to their power supplies and the field of operation. Some of these restrictions can be overcome by implementing the cooperation between unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs). The purpose of this paper is to explore the problem of sensor fusion for autonomous landing of a UAV on the UGV by comparing the performance of precision landing algorithms using different sensor fusions to have precise and reliable information about the position and velocity.Design/methodology/approachThe difficulties in this scenario, among others, are different coordination systems and necessity for sensor data from air and ground. The most suitable solution seems to be the use of widely available Global Navigational Satellite System (GNSS) receivers. Unfortunately, the position measurements obtained from cheap receivers are encumbered with errors when desiring precision. The different approaches are based on the usage of sensor fusion of Inertial Navigation System and image processing. However most of these systems are very vulnerable to lightning.FindingsIn this paper, methods based on an exchange of telemetry data and sensor fusion of GNSS, infrared markers detection and others are used. Different methods are compared.Originality/valueThe subject of sensor fusion and high-precision measurements in reference to the autonomous vehicle cooperation is very important because of the increasing popularity of these vehicles. The proposed solution is efficient to perform autonomous landing of UAV on the UGV.

Convergence of Asynchronous Distributed Gradient Methods Over Stochastic Networks

We consider distributed optimization problems in which a number of agents are to seek the global optimum of a sum of cost functions through only local information sharing. In this paper, we are particularly interested in scenarios, where agents are operating asynchronously over stochastic networks subject to random failures. Most existing algorithms require coordinated and decaying stepsizes to ensure zero gap between the estimated value of each agent and the exact optimum, restricting it from asynchronous implementation and resulting in slower convergence results. To deal with this issue, we develop a new asynchronous distributed gradient method (AsynDGM) based on consensus theory. The proposed algorithm not only allows for asynchronous implementation in a completely distributed manner but also, most importantly, is able to seek the exact optimum even with constant stepsizes. We will show that the assumption of boundedness of gradients, which is widely used in the literature, can be dropped by instead imposing the standard Lipschitz continuity condition on gradients. Moreover, we derive an upper bound of stepsize within which the proposed AsynDGM can achieve a linear convergence rate for strongly convex functions with Lipschitz gradients. A canonical example of sensor fusion problems is provided to illustrate the effectiveness of the proposed algorithm.

Mitigating Quantization Effects on Distributed Sensor Fusion: A Least Squares Approach

In this paper, we consider the problem of sensor fusion over networks with asymmetric links, where the common goal is linear parameter estimation. For the scenario of bandwidth-constrained networks, existing literature shows that nonvanishing errors always occur, which depend on the quantization scheme. To tackle this challenging issue, we introduce the notion of virtual measurements and propose a distributed solution LS-DSFS, which is a combination of a quantized consensus algorithm and the least squares approach. We provide detailed analysis of the LS-DSFS on its performance in terms of unbiasedness and mean square property. Analytical results show that the LS-DSFS is effective in smearing out the quantization errors, and achieving the minimum mean square error (MSE) among the existing centralized and distributed algorithms. Moreover, we characterize its rate of convergence in the mean square sense and that of the mean sequence. More importantly, we find that the LS-DSFS outperforms the centralized approaches within a moderate number of iterations in terms of MSE, and will always consume less energy and achieve more balanced energy expenditure as the number of nodes in the network grows. Simulation results are presented to validate theoretical findings and highlight the improvements over existing algorithms.

A deep learning based fusion of RGB camera information and magnetic localization information for endoscopic capsule robots

A reliable, real time localization functionality is crutial for actively controlled capsule endoscopy robots, which are an emerging, minimally invasive diagnostic and therapeutic technology for the gastrointestinal (GI) tract. In this study, we extend the success of deep learning approaches from various research fields to the problem of sensor fusion for endoscopic capsule robots. We propose a multi-sensor fusion based localization approach which combines endoscopic camera information and magnetic sensor based localization information. The results performed on real pig stomach dataset show that our method achieves sub-millimeter precision for both translational and rotational movements.

Eliminating the Effect of Magnetic Disturbances on the Inclination Estimates of Inertial Sensors

Inertial measurement units (IMUs) are a promising tool for realtime analysis of human motion in many application domains. We consider the standard sensor fusion problem in IMU orientation estimation. We highlight the malicious effects that inhomogeneous magnetic fields, which are often found in indoor environments, can have on the inclination portions (roll and pitch) of the orientation estimate. We then propose a novel method that uses an analytical solution of the sensor fusion problem and purely horizontal magnetometer-based corrections. The method assures that magnetic field measurements affect only the heading (yaw) component of the orientation estimate. Furthermore, we parametrize the algorithm such that the user can choose the time constant and aggressiveness with which the algorithm balances between gyroscope drift compensation and rejection of disturbances caused by inhomogeneous magnetic fields or by velocity changes.