Multi-sensor Information Research Articles

Multi-Sensor Information Fusion Technology in Robots

Multi-sensor information fusion is a technology that collects and processes data from multiple sensors to obtain more comprehensive and accurate information. In the field of robotics, multi-sensor information fusion technology is frequently applied to robots since the use of a single sensor in robots is often limited, this will also improve the accuracy and robustness of robot environment perception. After researching and organizing multi-sensor information fusion technology, robotics technology and other related materials, this paper provides a comprehensive introduction to the application of multi-sensor information fusion technology in the field of robotics, describes the logical structure of multi-sensor information fusion technology in robotics, explains the algorithms of multi-sensor information fusion technology that are mainly applied to robots. Finally, this paper summarizes and exemplifies the main applications of multi-sensor information fusion technology in the field of robotics.

Research Status and Prospect of the Key Technologies for Environment Perception of Intelligent Excavators

With the urgent need of the industry and the continuous development of artificial intelligence, research into intelligent excavators has achieved certain progress. However, intelligent excavators often face strong vibrations, dense dust, and complex objectives. These have brought severe challenges to environmental perception, and are important research difficulties that must be overcome in realizing the practical engineering applications of intelligent excavators. Many researchers have studied these problems in reducing vibration and dust noise for light detection and ranging (LiDAR) scanners, multi-sensor information fusion, and the segmentation and recognition of 3D scenes. This paper reviews the research status of these key technologies and discusses their development trends.

Research on a multi-sensor information fusion method for wind turbine gearbox fault diagnosis based on the BLSCFN model

Research on a multi-sensor information fusion method for wind turbine gearbox fault diagnosis based on the BLSCFN model

Application of multi-sensor information fusion technology in fault early warning of smart grid equipment

Application of multi-sensor information fusion technology in fault early warning of smart grid equipment

Structural Health Monitoring and Failure Analysis of Large-Scale Hydro-Steel Structures, Based on Multi-Sensor Information Fusion

Large-scale hydro-steel structures (LS-HSSs) are vital to hydraulic engineering, supporting critical functions such as water resource management, flood control, power generation, and navigation. However, due to prolonged exposure to severe environmental conditions and complex operational loads, these structures progressively degrade, posing increased risks over time. The absence of effective structural health monitoring (SHM) systems exacerbates these risks, as undetected damage and wear can compromise safety. This paper presents an advanced SHM framework designed to enhance the real-time monitoring and safety evaluation of LS-HSSs. The framework integrates the finite element method (FEM), multi-sensor data fusion, and Internet of Things (IoT) technologies into a closed-loop system for real-time perception, analysis, decision-making, and optimization. The system was deployed and validated at the Luhun Reservoir spillway, where it demonstrated stable and reliable performance for real-time anomaly detection and decision-making. Monitoring results over time were consistent, with stress values remaining below allowable thresholds and meeting safety standards. Specifically, stress monitoring during radial gate operations (with a current water level of 1.4 m) indicated that the dynamic stress values induced by flow vibrations at various points increased by approximately 2 MPa, with no significant impact loads. Moreover, the vibration amplitude during gate operation was below 0.03 mm, confirming the absence of critical structural damage and deformation. These results underscore the SHM system’s capacity to enhance operational safety and maintenance efficiency, highlighting its potential for broader application across water conservancy infrastructure.

Analysis of trust and efficient take-over strategy with multi-sensor information fusion in conditionally automated driving

Abstract. In the recent development of Autonomous Vehicles (AVs), human-machine trust and take-over strategy are important research contents, which is related to whether the driving system can make the most of its performance and ensure driving safety. At the same time, the unique advantages of Multi-Sensor Information Fusion (MSIF) also make it more and more applied to the auxiliary driving system. At present, research on MSIF applications in automated driving mainly focuses on external environment perception and map construction. However, there is little research on the application of MSIF in vehicle Human-Machine Interaction (HMI). Therefore, this paper analyzes the application of MSIF in trust detection and take-over HMI, proposes the advantages of applying MSIF to the HMI of AVs, and points out that the HMI design of intelligent vehicles in the future can enhance the application of MSIF. Dynamic monitoring of human-machine trust and dynamic adjustment of HMI strategy of the intelligent driving system are realized to achieve a better effect and further guarantee driving safety.

DeployFusion: A Deployable Monocular 3D Object Detection with Multi-Sensor Information Fusion in BEV for Edge Devices.

To address the challenges of suboptimal remote detection and significant computational burden in existing multi-sensor information fusion 3D object detection methods, a novel approach based on Bird's-Eye View (BEV) is proposed. This method utilizes an enhanced lightweight EdgeNeXt feature extraction network, incorporating residual branches to address network degradation caused by the excessive depth of STDA encoding blocks. Meantime, deformable convolution is used to expand the receptive field and reduce computational complexity. The feature fusion module constructs a two-stage fusion network to optimize the fusion and alignment of multi-sensor features. This network aligns image features to supplement environmental information with point cloud features, thereby obtaining the final BEV features. Additionally, a Transformer decoder that emphasizes global spatial cues is employed to process the BEV feature sequence, enabling precise detection of distant small objects. Experimental results demonstrate that this method surpasses the baseline network, with improvements of 4.5% in the NuScenes detection score and 5.5% in average precision for detection objects. Finally, the model is converted and accelerated using TensorRT tools for deployment on mobile devices, achieving an inference time of 138 ms per frame on the Jetson Orin NX embedded platform, thus enabling real-time 3D object detection.

Research on the role of multi-sensor system information fusion in improving hardware control accuracy of intelligent system

Abstract Multi-sensor management and control technology generally constructs a reasonable objective function to solve the optimal control command set to control a limited number of sensors to obtain higher quality measurement information, thus obtaining better target tracking performance. In the process of multi-sensor information fusion, there is not only the problem of information redundancy but also obvious time delay. A sensor fusion algorithm combined with global optimization algorithm is innovatively proposed. According to the key frames saved in the previous steps, feature points in local maps, sensor information, and loop information, a global optimization algorithm based on graph optimization model is constructed to optimize the position and pose of intelligent hardware system and the position of spatial feature points. Moreover, this work studies and experiments on multi-sensor fusion simultaneous localization and mapping (SLAM) comprehensively and systematically, and the experimental results show that the algorithm proposed in this work is superior to common open-source SLAM algorithm in positioning accuracy and mapping effect under special circumstances. Therefore, the method proposed in this work can be applied to intelligent driving of vehicles, vision-assisted movement of robots and intelligent control of unmanned aerial vehicles, thus effectively improving the hardware control accuracy of intelligent systems.

A vine-copulas based multi-sensor fusion structural damage monitoring method and its application in dam engineering

As a civil engineer that is susceptible to the mechanism of slow and continuous deterioration, the dam must detect damage according to the static data of the arranged sensors, which is an important issue in structural health monitoring. Due to the characteristics of huge solid structures, damage with a slight degree and small range usually does not cause significant changes in the monitoring data of a single sensor. Therefore, it is difficult to effectively detect local damage by traditional monitoring methods that establish operating warning values for a single sensor. Currently, there have been some approaches to fuse sensor information that consider correlations between multiple sensors as well as the overall operational behavior of the dam. However, their objectives are focused on improving the accuracy of response variable prediction models without addressing the local damage identification aspect. In this study, we apply the Vine-Copula model for the first time in a multi-sensor information fusion dam damage detection method based on static monitoring data. In this way, it improves the sensitivity of local damage identification for huge solid structures. A prediction model is fitted with the measuring data of each sensor to obtain the residual series. Then the joint probability distribution of the multidimensional residuals is acquired by constructing the Vine-Copula model. Finally, the joint cumulative distribution function value is used as an alarm limit to determine whether the structure has an abnormal damage condition. The effectiveness and accuracy of the proposed method were demonstrated in both a numerical arch dam simulating local damage and an actual rockfill dam engineering with cracks at the crest.

SRSGCN: A novel multi-sensor fault diagnosis method for hydraulic axial piston pump with limited data

Deep learning has immense potential in ensuring the safe operation of hydraulic axial piston pumps (HAPP). However, the complex operating environment and high cost of labeling have resulted in a scarcity of labeled fault samples. This paper proposes a novel method called Siamese Random Spatiotemporal Graph Convolutional Network (SRSGCN). Firstly, based on graph convolutional networks, a Random Spatiotemporal Graph (RSG) is designed to aggregate multi-sensor information at different time stamps, fully exploiting the spatiotemporal features of the original data. Secondly, the Siamese Neural Network (SNN) is improved by retaining the twin subnetwork structure and removing the similarity output part. While preserving feature extraction capabilities, it is endowed with classification ability. Based on its strong feature mining capability, SRSGCN can fully utilize the scarce sample information to improve diagnostic accuracy. Finally, a case study was conducted on our HAPP experimental platform. The experiments show that compared with other existing methods, this method has higher diagnostic accuracy and can effectively solve the problem of HAPP fault diagnosis under limited data conditions.

Neural architecture search for multi-sensor information fusion-based intelligent fault diagnosis

The multi-sensor information fusion-based intelligent fault diagnosis (MIF-IFD) method using deep learning aims to provide more accurate and reliable ways of identifying equipment health status than single-sensor methods. However, designing feature extraction modules in deep networks and determining information fusion execution layers still requires expertise and extensive experimentation, imposing high development costs. This limitation impedes the development and application of MIF-IFD. This paper proposes an architecture to mitigate the development challenges of MIF-IFD. Firstly, the basic module of the network consists of searchable feature extraction cells and a mask self-attention mechanism to achieve the extraction and fusion of multi-sensor information. Secondly, the optimal feature extraction cells and fusion starting layers are searched by constructing a continuous search space to compose the optimal fusion diagnostic structure, respectively. Finally, validation and analysis are conducted using two experimental datasets. The results demonstrate that the proposed method not only ensures diagnostic accuracy but also significantly improves development efficiency.

Automobile Intelligent Vehicle-machine and Human-computer Interaction System based on Big Data

This paper measures the accelerator, brake pedal, clutch, transmission device and steering wheel under different driving conditions in real-time and accurately on the simulation experiment platform. The goal is to conduct human-machine-road system interaction in a virtual simulation of human-vehicle-road systems. Fitting test data establishes the mathematical model of traffic control parameters. In terms of hardware, the distributed architecture of upper and lower computers is utilized. The system communicates point-to-point with the host computer through the RS-232 serial port. The system adopts multi-thread technology and serial communication technology. The simulation system of driving operation is designed with the visual central controller. The system takes 89S52 as the core. The slave program is written in C language. Then, the system establishes a multi-target coordinated obstacle avoidance method based on a multi-sensor information vehicle cooperation collision avoidance method. The multi-vehicle cooperation obstacle avoidance problem is transformed into an optimal control problem under multiple constraints. Simulation analysis shows that the velocity and displacement obtained by the multi-robot collaborative collision avoidance method are in good agreement with the measured values. Compared with the time series algorithm, the output accuracy of the proposed collaborative collision avoidance algorithm is significantly reduced, and the changes in velocity and displacement in the time domain are more stable.

LUO-V2 and MSPerception: A multi-fin co-drive robotic fish and its multi-sensor-based experimental system

Experimental systems for analyzing robotic fish swimming play a crucial role in optimizing robotic fish, which prompted us to design an experimental system that combines information from multiple sensors to address the shortcomings of current experimental systems with a single measurement. Furthermore, we are interested in a multi-fin co-drive robotic fish. This paper presents the design of a robotic fish, LUO-V2, and an experimental system, MSPerception. The LUO-V2 represents a novel solution for developing multi-fin co-drive robotic fish. It also provides a controller for LUO-V2, which can synchronously control the multi-fin of the robotic fish. Furthermore, the MSPerception system has been designed based on the fusion of multi-sensor information, which can sense the average thrust, swimming speed, and flow field, which provides an effective tool for studying robotic fish swimming. MSPerception was employed to investigate the factors influencing the swimming performance of LUO-V2 and to compare it with the flow field around the caudal fin of a real fish. The experimental results demonstrated that the multi-fin co-drive significantly enhanced the robotic fish’s performance. This research can provide a theoretical foundation for developing multi-fin co-drive robotic fish and experimental systems.

Hyper-relational interaction modeling in multi-modal trajectory prediction for intelligent connected vehicles in smart cites

Trajectory prediction of surrounding traffic participants is vital for the driving safety of Intelligent Connected Vehicles (ICVs). It has been enabled with the help of the availability of multi-sensor information collected by ICVs. For accurately predicting the future movements of traffic agents, it is crucial to subtlety model the inter-agent interaction. However, existing works focus on the correlations between agents and the map information while neglecting the importance of directly modeling the impact of map elements on inter-agent interactions, the direct modeling of which is beneficial for the representation of agent behaviors. Against this background, we propose to model the hyper-relational interaction, which incorporates map elements into the inter-agent interaction. To tackle the hyper-relational interaction, we propose a novel Hyper-relational Multi-modal Trajectory Prediction (HyperMTP) approach. Specifically, a hyper-relational driving graph is first constructed and the hyper-relational interaction is represented as the hyperedge, directly connecting to various nodes (i.e., agents and map elements). Then a structure-aware embedding initialization technique is developed to obtain unbiased initial embeddings. Afterward, hypergraph dual-attention networks are designed to capture correlations between graph elements while retaining the hyper-relational structure. Finally, a heterogeneous Transformer is devised to further capture the correlations between agents’ states and their corresponding hyper-relational interactions. Experimental results show that HyperMTP consistently outperforms the best-performing baseline with an average improvement of 4.8% across two real-world datasets. Moreover, HyperMTP also boosts the interpretability of trajectory prediction by quantifying the impact of map elements on inter-agent interactions.

Research on Cloud-Edge-Device Collaborative Intelligent Monitoring System of Grinding Wheel Wear State for High-Speed Cylindrical Grinding of Bearing Rings

Aiming at the problems of grinding wheel wear during high-speed cylindrical grinding, communication delays, and slow response during data acquisition, processing, and system operation, an intelligent online monitoring technology frame for CNC manufacturing units is proposed, incorporating a real-time-perception grinding mechanism and a cloud-edge device. Based on the grinding data and grinding wheel wear mechanism, a monitoring model using multi-sensor information fusion is constructed to assess the grinding wheel wear state. In addition, edge data acquisition and online monitoring software have been developed to improve the speed of data transmission and processing. Finally, based on the proposed framework, a cloud-edge device collaborative intelligent monitoring system for assessing grinding wheel wear during high-speed cylindrical grinding of bearing rings is constructed. It improves the grinding quality and efficiency, reduces the grinding cost, and incorporates remote control functionality.

High-accuracy road surface condition detection through multi-sensor information fusion based on WOA-BP neural network

High-accuracy road surface condition detection is crucial for enhancing traffic safety. However, existing methods struggle to comprehensively cover various road surface conditions, and the detection accuracy still requires improvement. In this study, we propose a novel road surface condition detection system integrating the whale optimization algorithm-enhanced backpropagation neural network (WOA-BP) with multi-sensor data, which significantly improves the detection accuracy. An experimental system was established, and the model's performance was validated through multiple experiments across a temperature range of −30℃ to 50℃. The results demonstrate that, compared to traditional machine learning algorithms such as back propagation neural network (BPNN), support vector machine (SVM), and random forest (RF), the WOA-BP neural network model shows superior performance in terms of detection accuracy and model stability. Specifically, it achieves a maximum accuracy rate of 100 % and an average accuracy rate of 98.8 % for detecting dry, wet, icy, and snowy conditions. By employing the WOA to refine the initial weights and thresholds of the BPNN, the issues of susceptibility to local optima and slow convergence are addressed, thereby enhancing the model's classification performance and robustness. Furthermore, the collaboration between the capacitive measurement-based road condition sensor, temperature sensor, and microwave measurement-based water film thickness sensor further enhances detection accuracy. This research delves into high-accuracy road condition detection technology utilizing the WOA-BP neural network, offering valuable insights for improving traffic safety and developing intelligent transportation systems.

Application of artificial intelligence sensor and visual image technology in the analysis of hydrophilic space landscape characteristics

With the continuous development of intelligent sensor technology and artificial intelligence, the application of artificial intelligence feature recognition at the level of intelligent sensor information acquisition to urban landscape analysis has become a current research trend and hot spot, but also brings us new research directions and challenges. In this paper, the multi-sensor collaborative filtering algorithm is optimized, and a method based on evidence theory is proposed to deal with the fuzzy and unclear information generated in the process of multi-sensor collaboration. Then, the analysis and recognition of multi-sensor fusion information using AI technology are analyzed in detail. This paper discusses the problems in multi-sensor information fusion and related solutions. Combined with the key nodes of 3D object AI feature recognition, multi-sensor collaborative Dempster Shafer evidence theory and 3D convolutional neural network waterfront space landscape feature recognition sub-model are constructed, and the waterfront space landscape recognition analysis model is tested and analyzed. The results show that the multi-sensor information collection Kalman filtering fusion algorithm effectively realizes the recognition of landscape feature information and the filtering of irrelevant interference information combined with an artificial intelligence algorithm to build a landscape feature recognition model. Among all kinds of landscape recognition, the recognition effect of beach inflow is the best, and the recognition accuracy, recall, and F1 value are all above 95. However, the identification effect of hydrophilic plank roads and underwater breakwater is relatively poor. The recognition accuracy of the hydrophilic platform is the lowest at 83.16 % among the eight landscape types, and the recall rate of underwater breakwater is the lowest at 79.96 % among the eight landscape types. In general, the multi-sensor Kalman filtering algorithm information fusion model designed in this paper has higher recognition accuracy and better classification performance for the collection and classification of the entire landscape dataset. The work of this paper provides a new idea and direction for the design and application of waterfront landscape identification and analysis systems based on intelligent sensors and artificial intelligence technology and lays a certain foundation for the intelligent design of urban waterfront landscapes.

Blood pressure prediction based on multi-sensor information fusion of electrocardiogram, photoplethysmography, and pressure pulse waveform

Cardiovascular diseases are now the leading cause of death that endangers people’s health. Thus, a precise and reliable blood pressure (BP) prediction method is essential. This paper proposes a noninvasive BP prediction method with the multi-feature fusion of electrocardiogram (ECG), photoplethysmography (PPG), and pressure pulse waveform (PPW). A multi-sensor information acquisition platform was developed to collect cardiovascular-related signals. Besides, the algorithms were designed to clean, preprocess, and extract features from sample data. Furthermore, the BP prediction model was constructed by using feature selection and feature fusion based on Random Forest Regression (RFR). Finally, the importance of features used for blood pressure prediction was analyzed, and the results of RFR-based blood pressure prediction were compared with those of other machine learning algorithms. The mean absolute errors of systolic and diastolic blood pressure prediction reached 0.90 mmHg and 2.47 mmHg, respectively. The results of the BP prediction model based on multi-sensor information fusion meet both the AAMI and BHS standards.

Multi-rate measurement fusion for power system state estimation

Abstract In recent years, the estimation of power systems has become increasingly critical. The introduction of Synchronized PMU has significantly enhanced the dynamic estimation capabilities of power systems. However, given the widespread deployment and usage of SCADA systems in the power grid, it is challenging to entirely replace SCADA systems with PMU systems in a short period. Therefore, utilizing a hybrid approach that combines the two systems for power state estimation is of utmost importance. This article achieves the integration of multi-rate and multi-sensor information in the power system by employing predictive interpolation on SCADA to match the sampling rate of PMU. The performance of power grid state estimation surpasses that of a single SCADA deployment.

An adaptive transfer fault detection method for rotary machine with multi-sensor information fusion

An adaptive transfer fault detection method for rotary machine with multi-sensor information fusion