Multi-sensor Fusion System Research Articles

A Survey on Vision-Based Anti Unmanned Aerial Vehicles Methods

The rapid development and widespread application of Unmanned Aerial Vehicles (UAV) have raised significant concerns about safety and privacy, thus requiring powerful anti-UAV systems. This survey provides an overview of anti-UAV detection and tracking methods in recent years. Firstly, we emphasize the key challenges of existing anti-UAV and delve into various detection and tracking methods. It is noteworthy that our study emphasizes the shift toward deep learning to enhance detection accuracy and tracking performance. Secondly, the survey organizes some public datasets, provides effective links, and discusses the characteristics and limitations of each dataset. Next, by analyzing current research trends, we have identified key areas of innovation, including the progress of deep learning techniques in real-time detection and tracking, multi-sensor fusion systems, and the automatic switching mechanisms that adapt to different conditions. Finally, this survey discusses the limitations and future research directions. This paper aims to deepen the understanding of innovations in anti-UAV detection and tracking methods. Hopefully our work can offer a valuable resource for researchers and practitioners involved in anti-UAV research.

Development of multi-sensors fusion monitoring system for shaft wall deformation

Ensuring shaft safety is crucial for underground coal mining. The prevailing method for monitoring shaft deformation involves manual measurements conducted regularly, necessitating numerous surveyors to descend into the shaft. This approach is laborious and time-consuming, and its reliance on human intervention significantly affects accuracy. An emerging alternative employs an enhanced laser slam scanner comprising laser ranging and inertial navigation angle measurement modules. The accuracy of the inertial navigation angle often requires refinement through laser point cloud registration to mitigate errors. However, existing research has not explored the utilization of a single inertial navigation sensor within a shaft, which potentially influences measurement outcomes. To investigate the impact of inertial navigation angle measurement errors from lidar scanners on shaft interior measurements and propose corresponding error reduction methods, this study developed a multisensor fusion system for monitoring shaft wall deformations. Three experiments were conducted to assess the accuracy of this monitoring system: a laboratory experiment, a simulated experiment, and a real shaft experiment. The results demonstrated that the maximum mean absolute errors (MAEs) in the laboratory and simulated shaft experiments were 0.06 cm and 0.30 cm, respectively, validating the reliability of the point cloud processing algorithm. In the real shaft experiment, the maximum MAE was 0.48 cm, confirming the measurement accuracy of the monitoring system. Compared to manual measurements, the system reduced the measurement time by 48% and streamlined the process from 12 to 3 steps. These findings highlight the monitoring system’s centimeter-level accuracy, efficiency in reducing measurement time, simplification of the measurement process, and elimination of the need for human entry into the shaft.

GRVINS: Tightly Coupled GNSS-Range-Visual-Inertial System

Bridge inspection is currently a labor intensive task. Utilizing unmanned aerial vehicles (UAVs) to assist in inspection tasks is a promising direction. However, enabling UAVs for autonomous inspection involves the UAV state estimation problems. Since parts of UAV sensors could be unavailable, how to estimate states via sensor fusion is the key. In this paper, we propose a tightly-coupled nonlinear optimization-based system that integrates four kinds of sensors: camera, IMU, Ultra-wideband (UWB) range measurements, and global navigation satellite system (GNSS). Due to the tightly-coupled multi-sensor fusion method and system design, the system takes the advantage of the four sensors, and can seamlessly respond to indoor and outdoor GNSS and UWB loss or reacquisition. It can effectively reduce the long-term trajectory drift and provide smooth and continuous state estimation. The experimental results show that the proposed method outperforms the state-of-the-art approaches.

VELIE: A Vehicle-Based Efficient Low-Light Image Enhancement Method for Intelligent Vehicles.

In Advanced Driving Assistance Systems (ADAS), Automated Driving Systems (ADS), and Driver Assistance Systems (DAS), RGB camera sensors are extensively utilized for object detection, semantic segmentation, and object tracking. Despite their popularity due to low costs, RGB cameras exhibit weak robustness in complex environments, particularly underperforming in low-light conditions, which raises a significant concern. To address these challenges, multi-sensor fusion systems or specialized low-light cameras have been proposed, but their high costs render them unsuitable for widespread deployment. On the other hand, improvements in post-processing algorithms offer a more economical and effective solution. However, current research in low-light image enhancement still shows substantial gaps in detail enhancement on nighttime driving datasets and is characterized by high deployment costs, failing to achieve real-time inference and edge deployment. Therefore, this paper leverages the Swin Vision Transformer combined with a gamma transformation integrated U-Net for the decoupled enhancement of initial low-light inputs, proposing a deep learning enhancement network named Vehicle-based Efficient Low-light Image Enhancement (VELIE). VELIE achieves state-of-the-art performance on various driving datasets with a processing time of only 0.19 s, significantly enhancing high-dimensional environmental perception tasks in low-light conditions.

Anomaly Detection Against GPS Spoofing Attacks on Connected and Autonomous Vehicles Using Learning From Demonstration

GPS spoofing attacks pose great challenges to connected vehicle (CVs) safety applications and localization of autonomous vehicles (AVs). In this paper, we propose to utilize transportation and vehicle engineering domain knowledge to detect GPS spoofing attacks towards CVs and AVs. A novel detection method using learning from demonstration is developed, which can be implemented in both vehicles and at the transportation infrastructure. A computational-efficient driving model, which can be learned from historical trajectories of the vehicles, is constructed to predict normal driving behaviors. Then a statistical method is developed to measure the dissimilarities between the observed trajectory and the predicted normal trajectory for anomaly detection. We validate the proposed method using two threat models (i.e., attacks targeting the multi-sensor fusion system of AVs and attacks targeting the intersection movement assist application of CVs) on two real-world datasets (i.e., KAIST and Michigan roundabout dataset). Results show that the proposed model is able to detect almost all of the attacks in time with low false positive and false negative rates.

A wavelet-based real-time fire detection algorithm with multi-modeling framework

This paper presents a wavelet-based real-time automated fire detection algorithm that takes into consideration the multi-resolution property of the wavelet transforms. Unlike conventional fire detection algorithms, which fail to capture temporal dependency within the fire sensor signals, the proposed wavelet-based features characterize temporal dynamics of chemical sensor signals generated from various types of fire, such as flaming, heating and smoldering fires. We propose a new feature selection technique based on types of fire to select the best features that can effectively discriminate between normal and various fire conditions. Then, a real-time fire detection algorithm with a multi-modeling framework is developed to effectively utilize the selected features and construct multiple fire detectors that are sensitive in monitoring various kinds of fires without prior knowledge. In addition, we develop a novel multi-sensor fusion system that incorporates various chemical sensors and collects an accurate and reliable fire dataset from different real-life fire scenarios in order to validate the performance of the proposed and existing fire detection algorithms. The experimental results with real-life and public fire data show that the proposed algorithm outperforms others with early detection time with a reasonable false alarm rate regardless of the type of fire.

Condition Monitoring and Fault Detection of Wind Turbine Driveline With the Implementation of Deep Residual Long Short-Term Memory Network

In view of the wide application of wind power, it is critically essential to develop solutions to the high fault rate and the long mean time to repair (MTTR). Multi-sensor fusion methods have been widely applied in condition monitoring and anomaly detection of industrial installation. Aimed at the wind turbine driveline, a new method utilizing the deep residual LSTM network with Attention model (ResLSTM-AM) is proposed in this paper. Firstly, data from multi-sensor fusion systems like supervisory control and data acquisition (SCADA) is preprocessed by quartile data cleaning and corresponding selection using the calculation of maximal information coefficient (MIC) to enhance its reliability. Secondly, we establish the neural network by the deep residual network (ResNet), long short-term memory network (LSTM), and attention mechanism (AM) for time series forecasting of wind turbines. The newly added residual connecting architecture in LSTM layers ensures a higher learning rate in feature extraction and provides a more flexible data flow both in forward and backward signals. Additionally, the attention mechanism after a LSTM layer amplifies the influence of vital data. The model gets validated based on historical SCADA data from actual wind turbines containing healthy conditions and anomaly conditions, showing ResNet contributes more. Based on the root-mean-square error (RMSE), the alarming threshold is calculated by an exponential weighted moving average (EWMA). For two chosen variables, the proposed method is confirmed to be efficient and reliable with the accuracy of 0.9945 and 0.9880 in the results of trials and comparisons with other existing models.

Set-membership multi-sensor secure fusion estimation against two-channel malicious attacks

In this study, the set-membership secure fusion estimation problem against malicious attacks is addressed. In practice, adversaries are able to tamper with the measurement outputs and local estimates from different communication channels. By rewriting the attacked information into two parts that are related and unrelated to the initial value, a novel secure estimation model is developed to describe the multi-sensor fusion systems that are subject to two-channel bounded attacks. An existence condition of the set-membership secure fusion estimator is presented to guarantee that the secure fusion estimation errors are always confined to expected ellipsoids. Subsequently, the secure fusion weight matrices are determined by solving a constructed convex optimization problem constrained by a linear matrix inequality condition. Lastly, the rationality of the designed set-membership secure fusion estimation algorithm is verified using two simulation examples.

Performance Evaluation of Stereo Vision Aided Loosely Coupled GNSS/SINS Integration for Land Vehicle Navigation in Different Urban Environments

In urban environments, the positioning performance of global navigation satellite system (GNSS) degrades severely due to the harsh environment, which will be improved after fused with a strap-down inertial navigation system (SINS). However, due to the error accumulation, the positioning ability of GNSS/SINS integration will also decrease significantly during frequent GNSS outages. To improve the positioning accuracy, we use stereo vision to enhance the performance of GNSS/SINS integration, where GNSS and stereo vision are loosely coupled and tightly coupled with SINS under the Earth-centered Earth-fixed (ECEF) frame, respectively. To evaluate the positioning performance of the multisensor fusion system, two in-field experiments were conducted to assess the contribution of stereo vision. The results show that with the support of stereo vision and SINS, GNSS accuracy can be considerably improved by 60%–80% in complex environments, which also outperforms the GNSS/SINS integration. Besides, detailed analyses in various urban environments further demonstrate that stereo vision can significantly improve the performance of GNSS/SINS integration when a long-term GNSS anomaly occurs, but it shows no improvement for a short-term GNSS anomaly.

MVIL-Fusion: Monocular Visual-Inertial-LiDAR Simultaneous Localization and Mapping in Challenging Environments

We propose mVIL-Fusion, a three-level multisensor fusion system that is able to achieve robust state estimation and globally consistent mapping in perceptually degraded environments. First, LiDAR depth-assisted visual-inertial odometry (VIO) with LiDAR odometry (LO) synchronous prediction and distortion correction functions is proposed as the frontend of our system. Second, a novel double-sliding-window-based optimization of midend joints of LiDAR scan-to-scan translation constraints (VIO status detection function) and scan-to-map rotation constraints (local mapping function) is used to enhance the accuracy and robustness of the state estimation. In the backend, loop closures of local-map-based keyframes are identified with altitude verification, and the global map is generated by incremental smoothing of a pose-only factor graph with altitude prior. The performance of our system is verified on both a public dataset and several self-collected sequences in challenging environments. To benefit the robotics community, our implementation is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Stan994265/mVIL-Fusion</uri> .

Selection of Areas for Effective GNSS Spoofing Attacks to a Vehicle-mounted MSF System based on Scenario Classification Models

The inherent vulnerability of the Global Navigation Satellite System (GNSS) leads to the ease of implementation of spoofing attacks. The latest GNSS spoofing attack schemes still suffer from low success rate, long attack time, and poor concealment. To improve the success rate, an efficient GNSS spoofing attack method for a vehicle-mounted Multi Sensors Fusion (MSF) system is proposed based on the scenario classification models with a spatial database. Firstly, the influence of the two typical urban scenarios, which are 1) the road with buildings on both sides and 2) tunnels, on the GNSS spoofing attack is analyzed. Then a dynamic Bayesian network model considering the sky visibility generated with the 3D building models and tunnel models inside the spatial database is established to quantify the difficulty of the attack. Furthermore, the scenarios of the victim can be classified into high-risk and low-risk scenarios. When the vehicle is just out of the tunnel or in open scenarios, attackers can select these high-risk scenarios and implement aggressive spoofing attacks. Then the efficiency of the GNSS spoofing attack can be significantly improved. Finally, the proposed attack scheme is demonstrated by actual world data with simulated spoofing attacks in urban areas.

Cyber-Physical Production and Multi-Sensor Fusion Systems, 3D Digital Twin Modeling and Image-based Visual Computing Technologies, and Virtual Machine and Environment Mapping Algorithms in the Immersive Industrial Metaverse

Cyber-Physical Production and Multi-Sensor Fusion Systems, 3D Digital Twin Modeling and Image-based Visual Computing Technologies, and Virtual Machine and Environment Mapping Algorithms in the Immersive Industrial Metaverse

Distributed Event-Triggered Nonlinear Fusion Estimation under Resource Constraints

This paper studies the event-triggered distributed fusion estimation problems for a class of nonlinear networked multi-sensor fusion systems without noise statistical properties. In practice, the sensor-to-remote estimator channel and the smart sensor-to-fusion center channel of the communication network will be faced with some resource constraint problems. To meet the finite communication resources during the information transmission, an event-triggered strategy and a dimensionality reduction strategy are introduced in a unified network framework to reduce communication traffic. Since the reduction of communication information will inevitably degenerate the estimation performance, two kinds of compensation strategies in terms of a unified model are proposed to restructure the untransmitted information. Then, the local/fusion estimators are designed based on the compensation information. Moreover, the linearization errors caused by the Taylor expansion are modeled by the state-dependent matrices with uncertain parameters when establishing estimation error systems, and then different robust recursive optimization problems are established to determine the estimator gains and the fusion criteria. Meanwhile, the stability conditions are also presented such that the square errors of the designed nonlinear estimators are asymptotically bounded. Finally, a vehicle localization system is employed to demonstrate the effectiveness and advantages of the proposed methods.

Research on Multi-Sensor Fusion SLAM Algorithm Based on Improved Gmapping

Simultaneous Localization and Mapping (SLAM) is the core technology of the intelligent robot system, and it is also the basis for its autonomous movement. In recent years, it has been found that SLAM using a single sensor has certain limitations, such as Inertial Measurement Unit (IMU) noise and serious drift, and 2D radar can only detect environmental information on the same horizontal plane. In this regard, this paper constructs a multi-sensor back-end fusion SLAM algorithm that combines vision, laser, encoder and IMU information. Experiments have proved that compared with using a single sensor, the application of a multi-sensor fusion system makes the edges of the constructed map clearer and the noise reduced. Aiming at the problem of increased calculation caused by particle degradation and too many particles, this paper improves the Gmappping algorithm, and uses the combination of selective resampling and Kullback-Leibler Distance (KLD) sampling to complete resampling. It has been proved by experiments that compared with the original algorithm of Gmapping, the application of the improved algorithm increases the particle convergence speed by 39.85% in the process of indoor mapping. Aiming at the problems that the traditional loop detection algorithm is easily affected by environmental factors, resulting in low detection accuracy, and the loop detection algorithm based on deep convolutional neural network has a large amount of calculation and takes a long time to detect. The main research of this paper is to apply a deep learning-based loop detection algorithm on the multi-sensor fusion framework, and use the combination of high-dimensional and low-dimensional features of the image for loop detection. This paper uses different algorithms to conduct comparative experiments on the dataset CityCentre. The experimental results show that compared with the traditional algorithms Bag of Words (BoW), AlexNet algorithm, VGG19 algorithm, and ResNet32 algorithm, the accuracy of the algorithm proposed in this paper has increased by 31.26%, 14.21%, 3.05%, and 1.56%, respectively. In addition, the comparison experiment results of SLAM mapping with the original Real-Time Appearance-Based Mapping (RTAB-MAP) algorithm prove that the loop closure detection algorithm based on deep learning proposed in this paper can enable the system to better build a globally consistent map, including more environmental information.

Multi-Level Fusion Optimization in Cyber-Physical Systems Using Computer Vision-Based Fault Detection

The healthcare sector's use of cyber-physical systems to provide high-quality patient treatment highlights the need for sophisticated security solutions due to the wide range of attack surfaces from medical and mobile devices, as well as body sensor nodes. Cyber-physical systems have various processing technologies to choose from, but these technical methods are as varied. Existing technologies are not well-suited for managing complex information about problem identification and diagnosis, which is distinct from technology. To address this issue, intelligent techniques for fusion processing, such as multi-sensor fusion system architectures and fusion optimization, can be used to improve fusion score and decision-making. Additionally, the use of deep learning models and multimedia data fusion applications can help to combine multiple models for intelligent systems and enhance machine learning for data fusion in E-Systems and cloud environments. Fuzzy approaches and optimization algorithms for data fusion can also be applied to robotics and other applications.. In this paper, a computer vision technology-based fault detection (CVT-FD) framework has been suggested for securely sharing healthcare data. When utilizing a trusted device like a mobile phone, end-users can rest assured that their data is secure. Cyber-attack behavior can be predicted using an artificial neural network (ANN), and the analysis of this data can assist healthcare professionals in making decisions. The experimental findings show that the model outperforms with current detection accuracy (98.3%), energy consumption (97.2%), attack prediction (96.6%), efficiency (97.9%), and delay ratios (35.6%) over existing approaches.

Fusion of Heterogenous Sensor Data in Border Surveillance.

Wide area surveillance has become of critical importance, particularly for border control between countries where vast forested land border areas are to be monitored. In this paper, we address the problem of the automatic detection of activity in forbidden areas, namely forested land border areas. In order to avoid false detections, often triggered in dense vegetation with single sensors such as radar, we present a multi sensor fusion and tracking system using passive infrared detectors in combination with automatic person detection from thermal and visual video camera images. The approach combines weighted maps with a rule engine that associates data from multiple weighted maps. The proposed approach is tested on real data collected by the EU FOLDOUT project in a location representative of a range of forested EU borders. The results show that the proposed approach can eliminate single sensor false detections and enhance accuracy by up to 50%.

Predicting aircraft maintenance timetofailure using the interval gradient regression tree algorithm

The operation and maintenance of modern aircraft multi-sensor data fusion systems generate vast amounts of numerical and symbolic data. Learning useful and non-trivial insights from this data may lead to considerable savings, and detection and reduction of the number of faults, as result increasing the overall level of aircraft safety. Several machine learning techniquesexist to learn from big amounts of data. However, the use of thesetechniques to infer the desired readable and accurate interval regression tree models from the data obtained during theoperation and maintenance of aircraft is extremely challenging. Difficulties that need to be addressed include: data warehouse collection and preprocessing, data labeling, machine learning model readability, setup, evaluation and maintenance. This paper presents the Interval Gradient Prediction Tree algorithm INGPRET, which addresses these issues. As shown by our empirical evaluation of a real aircraft multi-sensor data set, the INGPRET algorithm provides better readability and similar performance in comparison to other regression tree machine learning algorithms.

An Enhanced Hybrid Visual-Inertial Odometry System for Indoor Mobile Robot.

As mobile robots are being widely used, accurate localization of the robot counts for the system. Compared with position systems with a single sensor, multi-sensor fusion systems provide better performance and increase the accuracy and robustness. At present, camera and IMU (Inertial Measurement Unit) fusion positioning is extensively studied and many representative Visual–Inertial Odometry (VIO) systems have been produced. Multi-State Constraint Kalman Filter (MSCKF), one of the tightly coupled filtering methods, is characterized by high accuracy and low computational load among typical VIO methods. In the general framework, IMU information is not used after predicting the state and covariance propagation. In this article, we proposed a framework which introduce IMU pre-integration result into MSCKF framework as observation information to improve the system positioning accuracy. Additionally, the system uses the Helmert variance component estimation (HVCE) method to adjust the weight between feature points and pre-integration to further improve the positioning accuracy. Similarly, this article uses the wheel odometer information of the mobile robot to perform zero speed detection, zero-speed update, and pre-integration update to enhance the positioning accuracy of the system. Finally, after experiments carried out in Gazebo simulation environment, public dataset and real scenarios, it is proved that the proposed algorithm has better accuracy results while ensuring real-time performance than existing mainstream algorithms.

Unifying Obstacle Detection, Recognition, and Fusion Based on the Polarization Color Stereo Camera and LiDAR for the ADAS.

The perception module plays an important role in vehicles equipped with advanced driver-assistance systems (ADAS). This paper presents a multi-sensor data fusion system based on the polarization color stereo camera and the forward-looking light detection and ranging (LiDAR), which achieves the multiple target detection, recognition, and data fusion. The You Only Look Once v4 (YOLOv4) network is utilized to achieve object detection and recognition on the color images. The depth images are obtained from the rectified left and right images based on the principle of the epipolar constraints, then the obstacles are detected from the depth images using the MeanShift algorithm. The pixel-level polarization images are extracted from the raw polarization-grey images, then the water hazards are detected successfully. The PointPillars network is employed to detect the objects from the point cloud. The calibration and synchronization between the sensors are accomplished. The experiment results show that the data fusion enriches the detection results, provides high-dimensional perceptual information and extends the effective detection range. Meanwhile, the detection results are stable under diverse range and illumination conditions.

Design of Intelligent Perception Module Based on Wireless Sensor Network and Basketball Sports Attitude

In recent years, along with microelectromechanical (MEMS) technique and wireless body area network (WBAN), wearable health monitoring systems have emerged. With sustainability posture detection technique, people pay more and more attention to mankind posture detection. Human posture detection technique has been widely used in medical, film and television, industry, sports, and other fields. Motion capture is a technique for measuring the motion of moving objects in three-dimensional space with great accuracy. It is the most efficient method for producing computer 3D animation and collecting human motion data. The use of motion capture systems in animation is becoming increasingly common. The problem of a single sensor having a large error in monitoring human motion and attitude is addressed. The use of multisensor data fusion technology is used to propose a human motion pattern recognition method based on data fusion of accelerometer and gyroscope. The system must effectively integrate the information of various sensors in order to achieve the goal of accuracy, timeliness, and reliability processing, and multisensor information fusion systems for various complex application objects are constantly emerging.