Sensor Fusion Research Articles

Deep Learning Algorithm for Optimized Sensor Data Fusion in Fault Diagnosis and Tolerance

Environmental perception is one of the key technologies to realize autonomous vehicles. The fault diagnosis process involves identifying the fault that occurred or the cause of the out-of-control condition. Here, the major objective is to locate problems in detection by analysing previous data or sequential patterns of data that cause failure. This study evaluates the use of deep learning for improved sensor data fusion in fault identification and tolerance using the KITTI dataset. The input video from the dataset has been transformed to frames through median filtering. Next, feature extraction is applied to a preprocessed image, resulting in the fusion of sensor data. Data fusion is then carried out utilizing an enhanced RPN (region proposal network). The enhanced RPN also has a loss function (object detection loss, bounding box loss and target classification loss), an estimate of ROI and feature extraction network (FEN). Through the use of the COOT connected blue monkey optimization (CCBMO) model, the weight of the optimally enhanced RPN is established. Next, using global non-maximum suppression with both global and local confidence, fault identification and tolerance are carried out. From the analysis, it clearly shows that proposed method accomplished better results in terms of accuracy, precision and specificity of 97.78%, 93.76% and 93.43%, respectively, when compared with various conventional models with respect to diverse performance measures.

HSF-IBI: A Universal Framework for Extracting Inter-Beat Interval from Heterogeneous Unobtrusive Sensors

Heartbeat inter-beat interval (IBI) extraction is a crucial technology for unobtrusive vital sign monitoring, yet its precision and robustness remain challenging. A promising approach is fusing heartbeat signals from different types of unobtrusive sensors. This paper introduces HSF-IBI, a novel and universal framework for unobtrusive IBI extraction using heterogeneous sensor fusion. Specifically, harmonic summation (HarSum) is employed for calculating the average heart rate, which in turn guides the selection of the optimal band selection (OBS), the basic sequential algorithmic scheme (BSAS)-based template group extraction, and the template matching (TM) procedure. The optimal IBIs are determined by evaluating the signal quality index (SQI) for each heartbeat. The algorithm is morphology-independent and can be adapted to different sensors. The proposed algorithm framework is evaluated on a self-collected dataset including 19 healthy participants and an open-source dataset including 34 healthy participants, both containing heterogeneous sensors. The experimental results demonstrate that (1) the proposed framework successfully integrates data from heterogeneous sensors, leading to detection rate enhancements of 6.25 % and 5.21 % on two datasets, and (2) the proposed framework achieves superior accuracy over existing IBI extraction methods, with mean absolute errors (MAEs) of 5.25 ms and 4.56 ms on two datasets.

Tightly Coupled SLAM Algorithm Based on Similarity Detection Using LiDAR-IMU Sensor Fusion for Autonomous Navigation

In recent years, the rise of unmanned technology has made Simultaneous Localization and Mapping (SLAM) algorithms a focal point of research in the field of robotics. SLAM algorithms are primarily categorized into visual SLAM and laser SLAM, based on the type of external sensors employed. Laser SLAM algorithms have become essential in robotics and autonomous driving due to their insensitivity to lighting conditions, precise distance measurements, and ease of generating navigation maps. Throughout the development of SLAM technology, numerous effective algorithms have been introduced. However, existing algorithms still encounter challenges, such as localization errors and suboptimal utilization of sensor data. To address these issues, this paper proposes a tightly coupled SLAM algorithm based on similarity detection. The algorithm integrates Inertial Measurement Unit (IMU) and LiDAR odometry modules, employs a tightly coupled processing approach for sensor data, and utilizes curvature feature optimization extraction methods to enhance the accuracy and robustness of inter-frame matching. Additionally, the algorithm incorporates a local keyframe sliding window method and introduces a similarity detection mechanism, which reduces the real-time computational load and improves efficiency. Experimental results demonstrate that the algorithm achieves superior performance, with reduced positioning errors and enhanced global consistency, in tests conducted on the KITTI dataset. The accuracy of the real trajectory data compared to the ground truth is evaluated using metrics such as ATE (absolute trajectory error) and RMSE (root mean square error).

Evaluation of lower-body gait kinematics on outdoor surfaces using wearable sensors.

The effects of outdoor surfaces on gait are unclear due to difficulties associated with motion tracking outside laboratories. Today, inertial measurement unit (IMU) systems can be deployed to understand the biomechanical adaptations required to navigate real-world environments successfully. This study used IMUs devices to identify lower-limb kinematic adaptations while walking on outdoor surfaces. We hypothesize that gait adaptations between surface types will present as differences in lower-limb joint angles. Thirty able-bodied adults performed walking trials with IMUs on the lower back, thighs, and shanks. Outdoor walking surfaces were flat and even (flateven) (0° grade cement), cobblestone, grass, slope up, slope down, stairs up, and stairs down. A complementary-based sensor fusion algorithm was used to compute hip and knee joint flexion-extension angles, and data were normalized to 100% of the gait cycle based on foot-strike events. Flateven walking was compared against all other surfaces. Two-sample one-dimensional statistical parametric mapping (1d-SPM) t-tests were used to identify differences between angles (α≤0.05). Significant differences in joint angles were identified when grass, slope up, slope down, stairs up, and stairs down walking were compared with flateven (p≤0.005). Moreover, differences were found between slope and stair conditions (p≤0.004). No significant differences were noted between flateven and cobblestone. This study demonstrates that gait adaptations driven by differences in surface types can be observed using IMU sensors in an outdoor setting.

Tool Wear Monitoring in Hard Turning Using Sensor Fusion: An analytical approach

At present, the life expectancy of tool has become a vital aspect in the manufacturing industries, especially where materials with high hardness have more importance. Nowadays hard steel has been widely used for manufacturing of commercial parts in military aircrafts, car systems and hydraulic tools, etc. Manufacturing industries, mainly concentrates on mass production of the products with precision and accuracy. In such cases the continuous machining may weaken the tool causing tool wear which ultimately affects the quality with production rate. To avoid such an unwanted scenario, different tool wear prediction techniques have been introduced which uses cutting force signals or average chip-tool interface temperature or surface roughness signals, etc. According to the literature, different techniques are available for the pre-judgment of the tool wear that shows variation in the accuracy of prediction. Sensor fusion technique can be employed to overcome this problem by combining data from different sensors intelligently to improve the process. Sensor fusion uses this combined data to correct deficiencies of individual sensors and predict the tool wear accurately which compensate for the sudden breakage of the tool in real life applications. In this paper, a new approach of tool wear prediction has been introduced to correlate different available sensor by using sensor fusion technique. Also, a mathematical approach has been derived for the sensor fusion purpose. The experimentation has been carried out using coated carbide inserts on 55 HRC hardened steel.

Data level fusion of acoustic emission sensors using deep learning

The acoustic emission (AE) technique is commonly utilized for identifying source mechanisms and material damage. In applications requiring numerous sensors and limited detection areas, achieving significant cost savings, weight reduction, and miniaturization of AE sensors is crucial. This prevents excessive weight burdens on structures while minimizing interference with structural integrity. Thin Piezoelectric Wafer Active Sensors (PWAS), compared to conventional commercially available sensors, offer a miniature, lightweight, and affordable alternative. The low signal-to-noise ratio (SNR) of PWAS sensors and their limited effectiveness in monitoring thick structures result in the decreased reliability of a single classical PWAS sensor for damage detection. This research aims to enhance the functionality of PWAS in AE applications by employing multiple thin PWAS and performing a data-level fusion of their outputs. To achieve this, as a first step, the selection of the optimal PWAS is performed and a configuration is designed for multiple sensors. Pencil break lead (PBL) tests were performed to investigate the compatibility between selected PWAS and traditional WSα and R15α sensors. The responses of all sensors from different AE sources were compared in both the time and frequency domains. After that, convolutional neural networks (CNNs) combined with principal component analysis (PCA) are proposed for signal processing and data fusion. The signals generated by the PBL tests were used for network training and evaluation. This approach, developed by the authors, fuses the signals from multiple PWAS and reconstructs the signals obtained from conventional bulky AE sensors for damage detection. Three CNNs with different architectures were built and tested to optimize the network. It is found that the proposed methodology can effectively reconstruct and identify the PBL signals with high precision. The results demonstrate the feasibility of using a deep-learning-based method for AE monitoring using PWAS for real engineering structures.

Advanced AI-Driven Emotion Recognition Systems: Applications and Technical Challenges

Focusing on face and landmark analysis, the technical analysis investigates AI-based emotion identification systems' design, application, and difficulties. This thorough article explores the use of these systems in the manufacturing, scientific research, and healthcare industries, emphasizing both creative solutions and technological challenges. The article explores fundamental elements such as real-time emotion analysis frameworks, facial landmark identification systems, convolutional neural networks, and deep learning techniques. It shows how these systems are transforming human-computer interaction while addressing important issues in data protection and ethical implementation through a thorough examination of technical aspects, domain-specific applications, difficulties, and privacy considerations. To shed light on how emotion recognition technology is developing, the study also looks at potential future development paths, such as improvements in scalability, cross-platform integration, and sensor fusion technologies.

Development of a Multi-Source Satellite Fusion Method for XCH4 Product Generation in Oil and Gas Production Areas

Methane (CH4) is the second-largest greenhouse gas contributing to global climate warming. As of 2022, methane emissions from the oil and gas industry amounted to 3.586 million tons, representing 13.24% of total methane emissions and ranking second among all methane emission sources. To effectively control methane emissions in oilfield regions, this study proposes a multi-source remote sensing data fusion method based on the concept of data fusion, targeting high-emission areas such as oil and gas fields. The aim is to construct an XCH4 remote sensing dataset that meets the requirements for high resolution, wide coverage, and high accuracy. Initially, XCH4 data products from the GOSAT satellite and the TROPOMI sensor are matched both spatially and temporally. Subsequently, variables such as longitude, latitude, aerosol optical depth, surface albedo, digital elevation model (DEM), and month are incorporated. Using a local random forest (LRF) model for fusion, the resulting product combines the high accuracy of GOSAT data with the wide coverage of TROPOMI data. On this basis, ΔXCH4 is derived using GF-5. Combined with the GFEI prior emission inventory, the high-precision fusion dataset output by the LRF model is redistributed grid by grid in oilfield areas, producing a 1 km resolution XCH4 grid product, thereby constructing a high-precision, high-resolution dataset for oilfield regions. Finally, the challenges that emerged from the study were discussed and summarized, and it was envisioned that, in the future, with the advancement of satellite technology and algorithms, it would be possible to obtain more accurate and high-resolution datasets of methane concentration and apply such datasets to a wide range of fields, with the expectation that significant contributions could be made to reducing methane emissions and combating climate change.

Worst-Case Optimal Fault-Tolerant, Attack-Resilient Fusion Functions

From practical considerations, crisp sensor measurements are associated with tolerance value. Thus, the sensor measurements lead to intervals. Researchers proposed M, F, Ω, and N functions for the fusion of interval-valued sensor measurements. We proved that the F-function provides the worst-case optimal interval estimate given the number of faulty sensors. We generalize the F-function to multi-dimensional intervals. We also propose new results on the attack resilience of fusion functions. Thus, this research paper provides new insights into robust sensor fusion.

Review on Sensors and Path Planning Algorithms of Automatic Driving

automatic driving developed rapidly in recent years. Environment perception and path planning are two key functions in automatic driving, and sensors and algorithms are the basis for realizing these two functions. Through literature review and comparative analysis, this paper discusses sensors and path planning algorithms in automatic driving. An analysis is conducted on the principles, benefits, limitations, and applications of various sensor types in automatic driving. The RRT algorithm and A* algorithm are also discussed, as well as their advantages and constraints. Finally, this paper proposes suggestions such as deep learning, sensor fusion and cost reduction to update automatic driving. This paper provides a reference for the further development of automatic driving.

Multi-Sensor Fusion and Deep Learning: A New Frontier in Stabilization Algorithm Optimization

The rapid development of automation technology has highlighted the importance of stabilization algorithms in ensuring the stability and functionality of smart devices. These algorithms are critical for maintaining balance and precision, especially in applications such as unmanned aerial vehicles (UAVs), handheld devices, and autonomous systems. This research focuses on the application of stabilization algorithms across various domains, analyzing their principles, advantages, limitations, and potential optimization strategies. The study employs a literature review to investigate the integration of multi-sensor fusion and deep learning techniques to improve system stability. It shows that multi-sensor fusion significantly enhances system robustness by mitigating external disturbances, while deep learning improves autonomous decision-making in dynamic environments, particularly in UAV path planning and task execution. The findings confirm the crucial role of stabilization algorithms in modern systems and provide insights into future optimization directions. By combining deep learning with sensor fusion, future systems are expected to achieve greater stability and autonomy, contributing to the advancement of UAVs and other intelligent systems.

A Review of Multi-Sensor Fusion Techniques for Indoor Mobile Robot Navigation

The application of multi-sensor fusion technology in indoor mobile robot navigation and localization has been increasingly gaining attention, especially in complex indoor environments where achieving high-precision autonomous navigation poses a significant challenge. This review summarizes various sensor fusion methods, including the combination of Light Detection and Ranging (LiDAR), Inertial Measurement Unit (IMU), Ultra-Wideband (UWB), and others, with a focus on discussing the progress of fusion algorithms such as Extended Kalman Filter (EKF) and Adaptive Monte Carlo Localization (AMCL) in improving navigation accuracy and stability. In addition, this paper explores the application of visual Simultaneous Localization and Mapping (SLAM) methods incorporating deep learning in indoor robot navigation. Finally, the main challenges currently faced by multi-sensor fusion technology in robot autonomous navigation are analyzed, and future research directions are proposed, aiming to provide valuable references for researchers in the field.

Deep learning based smart traffic management using video analytics and IoT sensor fusion

Deep learning based smart traffic management using video analytics and IoT sensor fusion

Advancements in Sensor Fusion for Underwater SLAM: A Review on Enhanced Navigation and Environmental Perception

Underwater simultaneous localization and mapping (SLAM) has significant challenges due to the complexities of underwater environments, marked by limited visibility, variable conditions, and restricted global positioning system (GPS) availability. This study provides a comprehensive analysis of sensor fusion techniques in underwater SLAM, highlighting the amalgamation of proprioceptive and exteroceptive sensors to improve UUV navigational accuracy and system resilience. Essential sensor applications, including inertial measurement units (IMUs), Doppler velocity logs (DVLs), cameras, sonar, and LiDAR (light detection and ranging), are examined for their contributions to navigation and perception. Fusion methodologies, such as Kalman filters, particle filters, and graph-based SLAM, are evaluated for their benefits, limitations, and computational demands. Additionally, innovative technologies like quantum sensors and AI-driven filtering techniques are examined for their potential to enhance SLAM precision and adaptability. Case studies demonstrate practical applications, analyzing the compromises between accuracy, computational requirements, and adaptability to environmental changes. This paper proceeds to emphasize future directions, stressing the need for advanced filtering and machine learning to address sensor drift, noise, and environmental unpredictability, hence improving autonomous underwater navigation through reliable sensor fusion.

Multisource sensors navigation methodology applied to ADAS testing platform vehicle

This study addresses sensor signal loss in Advanced Driver Assistance Systems (ADAS) testing when switching between open and tunnel test sites. Current methods require pausing and switching sensor devices, disrupting the testing process. This study proposes a multisource sensor integrated navigation algorithm based on the Interacting Multiple Model (IMM) framework, using GNSS/INS error state Kalman filter (ESKF) positioning in open areas and UWB/INS ESKF positioning in GNSS failure scenarios, like tunnels. An Interacting Multiple Model based on Adaptive Transition Probability Matrix (ATPM-IMM) algorithm is introduced, enabling adaptive Markov jump probability for target tracking. The method ensures positioning accuracy using low-cost UWB and IMU and achieves seamless sensor source switching. Compared with traditional GNSS/INS and UWB/INS methods, the proposed approach improves accuracy by 31.17% and 41.23%, respectively, providing a reference for future multisource sensor fusion positioning research.

Adaptive output feedback fault-tolerant control for a class of nonlinear systems based on a sensor fusion mechanism

This paper investigates an output feedback adaptive fault-tolerant tracking control for a class of nonlinear systems with system nonlinearities, sensor failures and external disturbances, in which sensor redundancy is employed to enhance measurement reliability. A sensor fusion mechanism, together with a novel history-based weighted average algorithm is first designed to fuse all sensor outputs. Then, an adaptive controller based on the sensor fusion output, a dynamic gain and a state observer is constructed to handle all the uncertainties caused by system nonlinearities, external disturbances, sensor failures and fusion mechanism. It is shown that by using the proposed scheme, the closed-loop system is stable, the sensor fusion mechanism can eliminate the effects of faulty sensors, and the real tracking error can be driven into a small compact set mainly affected by the fusion error. Experimental results are accomplished to validate the proposed scheme.

Dual-Frequency Multi-Constellation Global Navigation Satellite System/Inertial Measurements Unit Tight Hybridization for Urban Air Mobility Applications

A global navigation satellite system (GNSS) for remotely piloted aircraft systems (RPASs) positioning is essential, thanks to the worldwide availability and continuity of this technology in the provision of positioning services. This makes the GNSS technology a critical element as malfunctions impacting on the determination of the position, velocity and timing (PVT) solution could determine safety issues. Such an aspect is particularly challenging in urban air mobility (UAM) scenarios, where low satellite visibility, multipath, radio frequency interference and cyber threats can dangerously affect the PVT solution. So, to meet integrity requirements, GNSS receiver measurements are augmented/fused with other aircraft sensors that can supply position and/or velocity information on the aircraft without relying on any other satellite and/or ground infrastructures. In this framework, in this paper, the algorithms of a hybrid navigation unit (HNU) for UAM applications are detailed, implementing a tightly coupled sensor fusion between a dual-frequency multi-constellation GNSS receiver, an inertial measurements unit and the barometric altitude from an air data computer. The implemented navigation algorithm is integrated with autonomous fault detection and exclusion of GPS/Galileo/BeiDou satellites and the estimation of navigation solution integrity/accuracy (i.e., protection level and figures of merit). In-flight tests were performed to validate the HNU functionalities demonstrating its effectiveness in UAM scenarios even in the presence of cyber threats. In detail, the navigation solution, compared with a real-time kinematic GPS receiver used as the reference centimetre-level position sensor, demonstrated good accuracy, with position errors below 15 m horizontally and 10 m vertically under nominal conditions (i.e., urban scenarios characterized by satellite low visibility and multipath). It continued to provide a valid navigation solution even in the presence of off-nominal events, such as spoofing attacks. The cyber threats were correctly detected and excluded by the system through the indication of the valid/not valid satellite measurements. However, the results indicate a need for fine-tuning the EKF to improve the estimation of figures of merit and protection levels associated to the navigation solution during the cyber-attacks. In contrast, solution accuracy and integrity indicators are well estimated in nominal conditions.

Nondestructive Detection of Litchi Stem Borers Using Multi-Sensor Data Fusion

To enhance lychee quality assessment and address inconsistencies in post-harvest pest detection, this study presents a multi-source fusion approach combining hyperspectral imaging, X-ray imaging, and visible/near-infrared (Vis/NIR) spectroscopy. Traditional single-sensor methods are limited in detecting pest damage, particularly in lychees with complex skins, as they often fail to capture both external and internal fruit characteristics. By integrating multiple sensors, our approach overcomes these limitations, offering a more accurate and robust detection system. Significant differences were observed between pest-free and infested lychees. Pest-free lychees exhibited higher hardness, soluble sugars (11% higher in flesh, 7% higher in peel), vitamin C (50% higher in flesh, 2% higher in peel), polyphenols, anthocyanins, and ORAC values (26%, 9%, and 14% higher, respectively). The Vis/NIR data processed with SG+SNV+CARS yielded a partial least squares regression (PLSR) model with an R2 of 0.82, an RMSE of 0.18, and accuracy of 89.22%. The hyperspectral model, using SG+MSC+SPA, achieved an R2 of 0.69, an RMSE of 0.23, and 81.74% accuracy, while the X-ray method with support vector regression (SVR) reached an R2 of 0.69, an RMSE of 0.22, and 76.25% accuracy. Through feature-level fusion, Recursive Feature Elimination with Cross-Validation (RFECV), and dimensionality reduction using PCA, we optimized hyperparameters and developed a Random Forest model. This model achieved 92.39% accuracy in pest detection, outperforming the individual methods by 3.17%, 10.25%, and 16.14%, respectively. The multi-source fusion approach also improved the overall accuracy by 4.79%, highlighting the critical role of sensor fusion in enhancing pest detection and supporting the development of automated non-destructive systems for lychee stem borer detection.

Oil pipeline multiple leakage detection and localization based on sensor fusion

Oil pipeline simultaneous leak localization solely using pressure and flow rate measurements at its boundaries is inherently challenging. Localizing simultaneous leaks in a long-distance oil pipeline solely using pressure and flow rate measurements at its boundaries presents significant challenges. This difficulty is attributed to the presence of noisy measurements and the interdependence of two equations that incorporate variables associated with leaks, leading to inherent uncertainty. Additionally, existing leak detection methodologies predominantly utilize a single sensor, which contributes to a high incidence of false alarms, particularly in complex operational environments. Developing nonlinear equations based on momentum and continuity principles is proposed as a comprehensive solution for estimating multiple leak locations in oil pipelines. In this study, multiple leaks in long-distance oil pipelines are estimated using a two-stage decision-making scheme. Arrays of sensors are proposed for improving the reliability and accuracy of simultaneous leak estimations, replacing the traditional reliance on single sensors along pipelines. The Fisher fusion technique combines inaccurate measurements to improve estimation accuracy. This method integrates sensor measurements even when noise is present, demonstrating its effectiveness. Comprehensive simulationsare conductedusing OLGA multiphase flow simulation software to evaluate the robustness and practicality of the proposed approach under various real-time conditions found in long-distance oil pipelines. The system’s state convergence and precision in simultaneous leak estimations highlight the effectiveness of the proposed strategy in oil pipeline control. Across various scenarios, the leak detector showed a 28 % improvement for a 50 Km oil pipeline.

Development And Testing of Arduino Based Autonomous Environmental Monitoring System

This paper presents the development of an Autonomous Environmental Monitoring System (AEMS) designed to track various environmental parameters. With the growing demand for autonomous monitoring systems in today’s modern era, we propose an Arduino-based real-time hardware solution to measure parameters such as flame, temperature, humidity, and distance. The system leverages advanced and cost-effective electronic sensors and hardware to monitor these environmental factors. We developed individual sensor modules for each parameter and integrated them into a unified multi-sensor system through sensor fusion. The collected data is then displayed on a 16 × 2 LCD module. The result is a fully functional autonomous monitoring system, tested under laboratory environmental conditions using a compact, handheld device.