Sensor Fusion Framework Research Articles

Scene adaptation in adverse conditions: a multi-sensor fusion framework for roadside traffic perception

Robust roadside traffic perception requires integrating the strengths of multi-source sensors under various adverse conditions, which is challenging but indispensable for formulating effective traffic management strategies. One limitation of existing radar-camera perception systems is that they focus on integrating multi-source information without directly considering scene information, leading to difficulties in achieving scene adaptive fusion. How to establish the connection between scene information and multi-source information is the key challenge to solving this problem. In this article, we propose a Scene adaptive Sensor Fusion (SSF) framework that characterizes scene information and integrates it into radar-camera fusion schemes, aiming to achieve high-quality roadside traffic perception. Specifically, we introduce a multi-source object association method that accurately associates multi-source sensor information on the roadside. We then utilize coding techniques to characterize the scene information, including visibility characterization regarding lighting and weather conditions, and road characterization regarding sensor viewpoint. By incorporating sensor and scene information into the fusion model, the SSF framework effectively establishes the connection between them. We evaluate the SSF framework on the Roadside Radar and Video Dataset (RRVD) and the Traffic flow Parameter Estimation Dataset (TPED), both collected from real-world traffic scenarios. Experiments demonstrate that SSF significantly improves vehicle detection accuracy under various adverse conditions compared to traditional single-source sensing methods and other state-of-the-art fusion techniques. Furthermore, vehicle trajectories based on SSF detection results enable accurate traffic parameter estimation, such as volume, speed, and density, in complex and dynamic environments.

Learnable fusion mechanisms for multimodal object detection in autonomous vehicles

AbstractPerception systems in autonomous vehicles need to accurately detect and classify objects within their surrounding environments. Numerous types of sensors are deployed on these vehicles, and the combination of such multimodal data streams can significantly boost performance. The authors introduce a novel sensor fusion framework using deep convolutional neural networks. The framework employs both camera and LiDAR sensors in a multimodal, multiview configuration. The authors leverage both data types by introducing two new innovative fusion mechanisms: element‐wise multiplication and multimodal factorised bilinear pooling. The methods improve the bird's eye view moderate average precision score by +4.97% and +8.35% on the KITTI dataset when compared to traditional fusion operators like element‐wise addition and feature map concatenation. An in‐depth analysis of key design choices impacting performance, such as data augmentation, multi‐task learning, and convolutional architecture design is offered. The study aims to pave the way for the development of more robust multimodal machine vision systems. The authors conclude the paper with qualitative results, discussing both successful and problematic cases, along with potential ways to mitigate the latter.

A laser-ultrasonic based sensor fusion framework for height measurement during the stone crushing process

It is a challenging work to monitor the height of the material pile via laser measurement due to the influence of high-concentration dust in a moving crushing line. At present, although the error of using only ultrasonic radar is small, the monitoring range is limited, while using only laser radar to monitor materials is highly affected by dust environments and may fail in high dust environments. To overcome this limitation, a height measurement based on the laser-ultrasonic radar fusion method was proposed to monitor material height in mobile crushing line. Firstly, the proposed method used laser scanning to obtain the three-dimensional point cloud of the material surface through point cloud preprocessing and calculated the material height. Secondly, considering that the concentration of dust has a great impact on the monitoring of the material level height, the experimental data on the change of the material level height with the concentration of dust were obtained, and the monitoring error was fitted to improve the accuracy of the material level height detection. Finally, in order to further reduce the error, the monitoring data of the ultrasonic radar was integrated to reduce the influence of dust on the detection of the material level height. The results showed that the average error was reduced to 22.65 mm when the dust concentration was below 200 mg m−3, and the average error was reduced to 32.14 mm when the dust concentration was higher than 200 mg m−3. The experimental results in different dust concentration environments demonstrated that the proposed method can effectively detect the point cloud shape and material level height of the material in real applications, which effectively improves the accuracy of material height detection by combining the three-dimensional laser points of the laser radar and the penetrability of the ultrasonic radar in high-concentration dust.

Efficient Human Gait Activity Recognition Based on Sensor Fusion and Intelligent Stacking Framework

Efficient Human Gait Activity Recognition Based on Sensor Fusion and Intelligent Stacking Framework

Fusion of Vision Deep-Learning Person Detection and Radio Beacon Ranging for Identifying and Locating Factory Workers

Assessment of an individual factory worker's productivity and safety has been increasingly demanded in the manufacturing industry. Although the long-term accuracy of identifying and locating people are key for such applications, recent technologies such as vision-based person re-identification (Re-ID) and radio-based Real Time Location Systems (RTLS) do not meet the requirements in open-world settings. In addition, vision-based person Re-ID might cause privacy concerns as it utilizes a person's appearance for identification. This article explains a sensor fusion framework that combines vision deep-learning person detection and radio beacon ranging to match the requirements for open-world settings and preserves privacy through an opt-out option. Its scalability is evaluated via Rank-1 person identification accuracy with images generated by a three-dimensional (3D) game engine to conduct hard-to-reproduce extreme scenarios. The representative Rank-1 is 88.7% with 4 people for low-latency applications ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> personalized geofencing) and 90.8% with 8 people for latency-tolerant applications ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> individual worker's productivity analysis).

Multi-modal sensor fusion towards three-dimensional airborne sonar imaging in hydrodynamic conditions

Analogous to how aerial imagery of above-ground environments transformed our understanding of the earth’s landscapes, remote underwater imaging systems could provide us with a dramatically expanded view of the ocean. However, maintaining high-fidelity imaging in the presence of ocean surface waves is a fundamental bottleneck in the real-world deployment of these airborne underwater imaging systems. In this work, we introduce a sensor fusion framework which couples multi-physics airborne sonar imaging with a water surface imager. Accurately mapping the water surface allows us to provide complementary multi-modal inputs to a custom image reconstruction algorithm, which counteracts the otherwise detrimental effects of a hydrodynamic water surface. Using this methodology, we experimentally demonstrate three-dimensional imaging of an underwater target in hydrodynamic conditions through a lab-based proof-of-concept, which marks an important milestone in the development of robust, remote underwater sensing systems.

Multilevel Data and Decision Fusion Using Heterogeneous Sensory Data for Autonomous Vehicles

Autonomous vehicles (AVs) are predicted to change transportation; however, it is still difficult to maintain robust situation awareness in a variety of driving situations. To enhance AV perception, methods to integrate sensor data from the camera, radar, and LiDAR sensors have been proposed. However, due to rigidity in their fusion implementations, current techniques are not sufficiently robust in challenging driving scenarios (such as inclement weather, poor light, and sensor obstruction). These techniques can be divided into two main groups: (i) early fusion, which is ineffective when sensor data are distorted or noisy, and (ii) late fusion, which is unable to take advantage of characteristics from numerous sensors and hence yields sub-optimal estimates. In this paper, we suggest a flexible selective sensor fusion framework that learns to recognize the present driving environment and fuses the optimum sensor combinations to enhance robustness without sacrificing efficiency to overcome the above-mentioned limitations. The proposed framework dynamically simulates early fusion, late fusion, and mixtures of both, allowing for a quick decision on the best fusion approach. The framework includes versatile modules for pre-processing heterogeneous data such as numeric, alphanumeric, image, and audio data, selecting appropriate features, and efficiently fusing the selected features. Further, versatile object detection and classification models are proposed to detect and categorize objects accurately. Advanced ensembling, gating, and filtering techniques are introduced to select the optimal object detection and classification model. Further, innovative methodologies are proposed to create an accurate context and decision rules. Widely used datasets like KITTI, nuScenes, and RADIATE are used in experimental analysis to evaluate the proposed models. The proposed model performed well in both data-level and decision-level fusion activities and also outperformed other fusion models in terms of accuracy and efficiency.

Evaluation of UKF-Based Fusion Strategies for Autonomous Underwater Vehicles Multisensor Navigation

In the underwater domain, guaranteeing accurate navigation for an autonomous underwater vehicle (AUV) is a complex but fundamental task to be achieved. As a matter of fact, only by ensuring a correct AUV localization, it is possible to accomplish surveillance, monitoring, and inspection missions. Most of the navigation filters for AUVs are based on Bayesian estimators, such as the Kalman filter, the extended Kalman filter (EKF), or the unscented Kalman filter (UKF), and employ different instruments, including the Doppler velocity log to perform the localization task. Recently, the use of payload sensors, such as cameras or forward-looking SONARs, in navigation-aiding has arisen as an interesting research field in the attempt to reduce the localization error drift. Such sensors, if used simultaneously, can provide multiple observations, which can be combined in a Kalman filtering framework to increase navigation robustness against noise sources. Navigation techniques that employ multiple devices can provide a high improvement of the estimation quality, but they can also cause an increase in terms of computational load. Consequently, strategies that can represent a tradeoff between these two conflicting goals have to be investigated. In this contribution, two different frameworks have been implemented and compared: on the one hand, a centralized iterative UKF-based navigation approach and, on the other hand, a sensor fusion framework with parallel local UKFs. The sequential (or iterated) UKF, where the correction step is iteratively performed for each available measurement, belongs to the first class of filters. The federated and the consensus-based decentralized UKFs can be categorized as the second class and they differ in the employed fusion strategy. Experimental navigation data obtained during sea trials performed at Vulcano Island, Messina, Italy has been used for offline validation. The results analysis focuses on both the navigation quality and the filter robustness against the reduction of the available measurements.

A Secure Sensor Fusion Framework for Connected and Automated Vehicles Under Sensor Attacks

As typical applications of cyber–physical systems (CPSs), connected and automated vehicles (CAVs) are able to measure the surroundings and share local information with the other vehicles by using multimodal sensors and wireless networks. CAVs are expected to increase safety, efficiency, and capacity of our transportation systems. However, the increasing usage of sensors has also increased the vulnerability of CAVs to sensor faults and adversarial attacks. Anomalous sensor values resulting from malicious cyberattacks or faulty sensors may cause severe consequences or even fatalities. In this article, we increase the resilience of CAVs to faults and attacks by using multiple sensors for measuring the same physical variable to create redundancy. We exploit this redundancy and propose a sensor fusion algorithm for providing a robust estimate of the correct sensor information with bounded errors independent of the attack signals, and for attack detection and isolation. The proposed sensor fusion framework is applicable to a large class of security-critical CPSs. To minimize the performance degradation resulting from the usage of the estimation for control, we provide an <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$H_{\infty }$ </tex-math></inline-formula> controller for cooperative adaptive cruise control-equipped CAVs. The designed controller is capable of stabilizing the closed-loop dynamics of each vehicle in the platoon while reducing the joint effect of estimation errors and communication channel noise on the tracking performance and string behavior of the vehicle platoon. Numerical examples are presented to illustrate the effectiveness of our methods.

Adaptive Band Extraction Based on Low Rank Approximated Nonnegative Tucker Decomposition for Anti-Friction Bearing Faults Diagnosis Using Measured Vibration Data

Condition monitoring and fault diagnosis are topics of growing interest for improving the reliability of modern industrial systems. As critical structural components, anti-friction bearings often operate under harsh conditions and are contributing factors of system failures. Efforts have been cast on bearing diagnostics under the sensor fusion and machine learning framework, whilst challenges remain open on the identification of incipient faults. In this paper, exploiting multi-way representations and decompositions of measured vibration data, a novel band separation method based on the factorization of spectrogram tensors using the low rank approximated nonnegative Tucker decomposition (LRANTD) is proposed and applied to identify detailed fault signatures from the spectral, temporal, and spatial dimensions, flexible for extracting multi-sensor features and multi-dimensional correlations. With the proposed method, informative frequency bands of the latent vibrational components can be automatically extracted, in accordance with the inherent temporal patterns that can be conveniently fed for spectral analysis and fault discrimination. Furthermore, an improved cross-spectrum can be calculated from multi-channel vibrations via LRANTD with enhanced fault features. Based on the real-world vibration data of the accelerated bearing life tests, detailed experimental studies and thorough comparisons to the conventional benchmarks have verified the effectiveness of the reported diagnostic methodology. The proposed method significantly improves the presence of the bearing frequency peaks distinctly over the background noises in the spectrum and hence improves the bearing defect detection process.

Fusion VIO: An Extendable Sensor Fusion Framework for Unmaned Vehicles

Real-time perception based on the simultaneous localization and mapping (SLAM) technology is of immense application potential in the industry filed. In this work, we provide an extended Kalman filter (EKF) based visual SLAM framework named Fusion VIO, which contains HDR imaging system, hardware synchronization, object detection, filter based sensor fusion, and semantic fusion. Notably, Fusion VIO well fuses loop closure information, which takes effects in solving the error problem during operation of the robot and improving the robustness of the system. Besides, experimental validation shows that Fusion VIO not only maintains high accuracy but also cost one-third resources compared to the common VINS fusion. Our work demonstrates a novel sensor fusion framework for unmaned vehicles.

A Credible and Robust Approach to Ego-Motion Estimation Using an Automotive Radar

Consistent motion estimation is fundamental for all mobile autonomous systems. While this sounds like an easy task, often, it is not the case because of changing environmental conditions affecting odometry obtained from vision, Lidar, or the wheels themselves. Unsusceptible to challenging lighting and weather conditions, radar sensors are an obvious alternative. Usually, automotive radars return a sparse point cloud, representing the surroundings. Utilizing this information to motion estimation is challenging due to unstable and phantom measurements, which result in a high rate of outliers. We introduce a credible and robust probabilistic approach to estimate the ego-motion based on these challenging radar measurements; intended to be used within a loosely-coupled sensor fusion framework. Compared to existing solutions, evaluated on the popular nuScenes dataset and others, we show that our proposed algorithm is more credible while not depending on explicit correspondence calculation.

Nonlinear unknown input observability and unknown input reconstruction: The general analytical solution

Observability is a fundamental structural property of any dynamic system and describes the possibility of reconstructing the state that characterizes the system from fusing the observations of its inputs and outputs. Despite the effort made to study this property and to introduce analytical criteria capable of verifying whether or not a dynamic system satisfies this property, there is no general analytical criterion to obtain the observability of the state when the dynamics are also driven by unknown inputs. Here, we introduce the general analytical solution of this fundamental open problem, often called the unknown input observability problem. We provide the systematic procedure, based on automatic calculation (differentiation and determination of the matrix rank), which allows us to check the observability of the state even in the presence of unknown inputs. One of the fundamental ingredients to obtain this solution is the characterization of the group of invariance of observability. We have very recently introduced this group, together with a new set of tensor fields with respect to this group of transformations (Martinelli, 2020). The analytical solution of the unknown input observability problem is expressed in terms of these tensor fields. In Martinelli (2020) we provided the solution by restricting our investigation to systems that satisfy a special assumption that is called canonicity with respect to the unknown inputs. Here, after an exhaustive characterization of the concept of canonicity, we also account for the case when this assumption is not satisfied and we provide the general solution. This solution is also provided in the form of a new algorithm. In addition, even in the canonic case dealt with in Martinelli (2020), here we provide a new fundamental result that regards the convergence properties of the solution. Finally, as a consequence of the results obtained here, we also provide the condition to reconstruct the unknown inputs, and, when this condition is not met, what can be reconstructed on the unknown inputs. We illustrate the implementation of the new algorithm by studying the observability properties of a nonlinear system in the framework of visual-inertial sensor fusion, whose dynamics are driven by two unknown inputs and one known input. In particular, for this system, we follow step by step the algorithm introduced by this paper, which solves the unknown input observability problem in the most general case.

Multimodal sensor fusion framework for residential building occupancy detection

For several years now, smart building energy systems have been a research area of intensive activity. In light of the increasing need for sustainable buildings and energy systems, this trend motivates an increasing need for a solution to reduce carbon dioxide emissions and improve energy efficiency. This paper proposes a high-performing and transferable occupancy detection framework that combines sensor data from different data modalities, including time series environmental data (temperature, humidity, and illuminance), image data, and acoustic energy data using ensemble method. To draw out the best prediction performance in each modality, the proposed framework was developed, including various models that were designed to learn the occupancy patterns reflected in the physical data streams. To tackle the time series environmental data, we designed two variants of an occupancy detection spatiotemporal pattern network (Occ-STPN) that performs both feature level and decision level fusion, respectively. We also propose a new metric; the fading memory mean square error (FMMSE), that provides a fair evaluation and penalization of delayed occupancy predictions. Multiple open-sourced datasets, including the Electricity Consumption and Occupancy and the University of California, Irvine’s (UCI) building occupancy detection dataset, along with our own real data collected from six different houses, were used to validate the algorithms’ performance. The experimental results presented herein break down the performance for each sensing modality, and a detailed analysis of the performance is also discussed.

Deep Visible and Thermal Camera-based Optimal Semantic Segmentation using Semantic Forecasting

Abstract Visible camera-based semantic segmentation and semantic forecasting are important perception tasks in autonomous driving. In semantic segmentation, the current frame's pixel level labels are estimated using the current visible frame. In semantic forecasting, the future frame's pixel-level labels are predicted using the current and the past visible frames and pixel-level labels. While reporting state-of-the-art accuracy, both of these tasks are limited by the visible camera's susceptibility to varying illumination, adverse weather conditions, sunlight and headlight glare etc. In this work, we propose to address these limitations using the deep sensor fusion of the visible and the thermal camera. The proposed sensor fusion framework performs both semantic forecasting as well as an optimal semantic segmentation within a multi-step iterative framework. In the first or forecasting step, the framework predicts the semantic map for the next frame. The predicted semantic map is updated in the second step, when the next visible and thermal frame is observed. The updated semantic map is considered as the optimal semantic map for the given visible-thermal frame. The semantic map forecasting and updating are iteratively performed over time. The estimated semantic maps contain the pedestrian behavior, the free space and the pedestrian crossing labels. The pedestrian behavior is categorized based on their spatial, motion and dynamic orientation information. The proposed framework is validated using the public KAIST dataset. A detailed comparative analysis and ablation study is performed using pixel-level classification and IOU error metrics. The results show that the proposed framework can not only accurately forecast the semantic segmentation map but also accurately update them.

Incremental learning of LSTM framework for sensor fusion in attitude estimation.

This paper presents a novel method for attitude estimation of an object in 3D space by incremental learning of the Long-Short Term Memory (LSTM) network. Gyroscope, accelerometer, and magnetometer are few widely used sensors in attitude estimation applications. Traditionally, multi-sensor fusion methods such as the Extended Kalman Filter and Complementary Filter are employed to fuse the measurements from these sensors. However, these methods exhibit limitations in accounting for the uncertainty, unpredictability, and dynamic nature of the motion in real-world situations. In this paper, the inertial sensors data are fed to the LSTM network which are then updated incrementally to incorporate the dynamic changes in motion occurring in the run time. The robustness and efficiency of the proposed framework is demonstrated on the dataset collected from a commercially available inertial measurement unit. The proposed framework offers a significant improvement in the results compared to the traditional method, even in the case of a highly dynamic environment. The LSTM framework-based attitude estimation approach can be deployed on a standard AI-supported processing module for real-time applications.

Failure Detection Methods for Pipeline Networks: From Acoustic Sensing to Cyber-Physical Systems.

Pipeline networks have been widely utilised in the transportation of water, natural gases, oil and waste materials efficiently and safely over varying distances with minimal human intervention. In order to optimise the spatial use of the pipeline infrastructure, pipelines are either buried underground, or located in submarine environments. Due to the continuous expansion of pipeline networks in locations that are inaccessible to maintenance personnel, research efforts have been ongoing to introduce and develop reliable detection methods for pipeline failures, such as blockages, leakages, cracks, corrosion and weld defects. In this paper, a taxonomy of existing pipeline failure detection techniques and technologies was created to comparatively analyse their respective advantages, drawbacks and limitations. This effort has effectively illuminated various unaddressed research challenges that are still present among a wide array of the state-of-the-art detection methods that have been employed in various pipeline domains. These challenges include the extension of the lifetime of a pipeline network for the reduction of maintenance costs, and the prevention of disruptive pipeline failures for the minimisation of downtime. Our taxonomy of various pipeline failure detection methods is also presented in the form of a look-up table to illustrate the suitability, key aspects and data or signal processing techniques of each individual method. We have also quantitatively evaluated the industrial relevance and practicality of each of the methods in the taxonomy in terms of their respective deployability, generality and computational cost. The outcome of the evaluation made in the taxonomy will contribute to our future works involving the utilisation of sensor fusion and data-centric frameworks to develop efficient, accurate and reliable failure detection solutions.

Mapping in unstructured natural environment: a sensor fusion framework for wearable sensor suites

We present a generalized mapping framework that can withstand the challenges incurred by working in unstructured outdoor environments, such as a snowy forest. The proposed method takes advantage of a sensor fusion scheme, where sensors such as cameras and lidars are used in order to reconstruct the surrounding natural environment. Although mapping techniques such as SLAM and ICP cannot themselves properly handle the complexity of natural scenes, they do have the potential to contribute to the global solution in a proposed sensor fusion scheme, based on a factor graph architecture. In this paper, we propose an innovative map registration scheme for visual maps, and show how it can improve the reconstruction quality after data fusion. We also analyze the behavior and sensitivity of factor graphs to uncertainties, by comparing the residual error with different parameter combinations such as variances, using an exhaustive grid search with ground truth comparison. Finally, we suggest an ICP-inferred loop closure, capable of compensating position and attitude drift. The experiments are carried out by recording in a snowy forest using a wearable sensor suite. In the experiments, ground truth was acquired using a millimeter-accurate total station. The proposed framework is shown to be robust and likewise capable of providing estimates that are otherwise unattainable using classic techniques, such as visual SLAM and ICP for lasers. Finally, a visible improvement in the map reconstruction quality is shown, and the proposed framework achieves a translation error of 0.36 m.

Deep Feature-Level Sensor Fusion Using Skip Connections for Real-Time Object Detection in Autonomous Driving

Object detection is an important perception task in autonomous driving and advanced driver assistance systems. The visible camera is widely used for perception, but its performance is limited by illumination and environmental variations. For robust vision-based perception, we propose a deep learning framework for effective sensor fusion of the visible camera with complementary sensors. A feature-level sensor fusion technique, using skip connection, is proposed for the sensor fusion of the visible camera with the millimeter-wave radar and the thermal camera. The two networks are called the RV-Net and the TV-Net, respectively. These networks have two input branches and one output branch. The input branches contain separate branches for the individual sensor feature extraction, which are then fused in the output perception branch using skip connections. The RVNet and the TVNet simultaneously perform sensor-specific feature extraction, feature-level fusion and object detection within an end-to-end framework. The proposed networks are validated with baseline algorithms on public datasets. The results obtained show that the feature-level sensor fusion is better than baseline early and late fusion frameworks.

An Adaptive Sensor Fusion Framework for Pedestrian Indoor Navigation in Dynamic Environments

Indoor navigation is a representative application of an indoor positioning system that uses a variety of equipment, including smartphones with various sensors. Many indoor navigation systems utilize Wi-Fi signals, as well as a variety of inertial sensors, such as a 3D accelerometer, digital compass, gyroscope, and barometer, to improve the accuracy of user location tracking. The inertial sensors are vulnerable to changes in the surrounding environments and sensitive to users behavior, but little research has been conducted on sensor fusion under these conditions. In this paper, we propose a dynamic sensor fusion framework (DSFF) that provides accurate user tracking results by dynamically calibrating inertial sensor readings in a sensor fusion process. The proposed method continually learns the errors and biases of each sensor due to the changes in user behavior patterns and surrounding environments. The learned patterns are then dynamically applied to the user tracking process to yield accurate results. The results of experiments conducted in both a single-story and a multi-story building confirm that DSFF provides accurate tracking results. The scalability of the DSFF will enable it to provide more accurate tracking results with various sensors, both existing and under development.