Multi-rate Sensor Fusion Research Articles

A multi-rate sensor fusion and multi-task learning network for concurrent fault diagnosis of hydraulic systems

Hydraulic systems are widely used in key modern industrial fields such as mechanical manufacturing, aerospace, and heavy machinery, and their efficient and reliable operation is crucial to ensuring production safety and efficiency. However, hydraulic systems often experience concurrent faults, such as pump failures, valve blockages, pipeline leaks, and fluid contamination, which pose significant challenges to the fault diagnosis in hydraulic systems. This paper introduces a multi-task learning network that deconstructs the challenge of concurrent fault diagnosis into specific sub-tasks, enabling the simultaneous identification and classification of multiple hydraulic components' faults. Automatic channel filtering is designed to screen out sensitive channels of each component from multi-rate sensors. A dual-flow model is used to feature extraction, which can simultaneously extract the local spatial features and global semantic information. Then, four classification models are designed to identify the extracted shared features. An uncertainty weight loss is also proposed to balance the loss of different tasks. The experimental results show that our model significantly outperforms traditional methods and other popular multi-output methods in diagnosing concurrent faults.

Vehicle Localization Kalman Filtering for Traffic Light Advisor Application in Urban Scenarios.

The recent advancements in Intelligent Transportation Systems (ITS) have revealed significant potential for enhancing traffic management through Advanced Driver Assist Systems (ADASs), with benefits for both safety and environment. This research paper proposes a vehicle localization technique based on Kalman filtering, as accurate positioning of the ego-vehicle is essential for the proper functioning of the Traffic Light Advisor (TLA) system. The aim of the TLA is to calculate the most suitable speed to safely reach and pass the first traffic light in front of the vehicle and subsequently keep that velocity constant to overcome the following traffic light, thus allowing safer and more efficient driving practices, thereby reducing safety risks, and minimizing energy consumption. To overcome Global Positioning Systems (GPS) limitations encountered in urban scenarios, a multi-rate sensor fusion approach based on the Kalman filter with map matching and a simple kinematic one-dimensional model is proposed. The experimental results demonstrate an estimation error below 0.5 m on urban roads with GPS signal loss areas, making it suitable for TLA application. The experimental validation of the Traffic Light Advisor system confirmed the expected benefits with a 40% decrease in energy consumption compared to unassisted driving.

Multirate Sensor Fusion in the Presence of Irregular Measurements and Time-Varying Time Delays Using Synchronized, Neural, Extended Kalman Filters

Sensor fusion plays a critical role in improving estimation accuracy of process quality variables. In this article, the dual, neural, extended Kalman filter (DNEKF) and the state model compensation neural, extended Kalman filter (SNEKF) are synthesized to compensate for modeling errors in the extended Kalman filter (EKF)-based multirate sensor fusion. Specifically, fusion is performed in the presence of irregularly sampled, slow-rate measurements with time-varying time delays. The proposed algorithm estimates the state and neural network parameters simultaneously through state vector augmentation. The estimated parameters of the state model compensation neural network (SNN) are shared between the DNEKF and SNEKF. It is demonstrated through two numerical examples that the proposed algorithm effectively reduces the estimation error under different conditions. In addition, it successfully improves the critical industrial quality variable estimation accuracy from the fast-rate soft sensor for over 20%, in terms of the mean squared error, demonstrating its advantages.

Discrete-Time Sliding Mode Control Coupled with Asynchronous Sensor Fusion for Rigid-Link Flexible-Joint Manipulators

This paper proposes a novel discrete-time terminal sliding mode controller (DTSMC) coupled with an asynchronous multirate sensor fusion estimator for rigid-link flexible-joint (RLFJ) manipulator tracking control. A camera is employed as external sensors to observe the RLFJ manipulator’s state which cannot be directly obtained from the encoders since gear mechanisms or flexible joints exist. The extended Kalman filter- (EKF-) based asynchronous multirate sensor fusion method deals with the slow sampling rate and the latency of camera by using motor encoders to cover the missing information between two visual samples. In the proposed control scheme, a novel sliding mode surface is presented by taking advantage of both the estimation error and tracking error. It is proved that the proposed controller achieves convergence results for tracking control in the theoretical derivation. Simulation and experimental studies are included to validate the effectiveness of the proposed approach.

Dual Neural Extended Kalman Filtering Approach for Multirate Sensor Data Fusion

The dual neural extended Kalman filter (DNEKF) is proposed in this article to compensate for model inaccuracies and violations of noise assumption in the EKF-based multirate sensor fusion. The DNEKF employs two neural networks to improve process state and output predictions through simultaneous state and parameter estimations using state vector augmentation. Taking advantage of frequent but less accurate measurements and infrequent but more accurate measurements provides a multirate parameter update strategy for neural networks training in the proposed approach. Two numerical examples and one industrial application demonstrate the effectiveness of the proposed method to compensate for inadequate process knowledge to improve multirate sensor fusion performance.

Gaussian Process Online Learning With a Sparse Data Stream

Gaussian processes (GPs) have been exploited for various applications even including online learning. To learn time-varying hyperparameters from an information-limited sparse data stream, we consider the infinite-horizon Gaussian process (IHGP) with a low computational complexity. For example, the IHGP framework could provide efficient GP online learning with a sparse data stream in mobile devices. However, we show that the originally proposed IHGP has difficulty in learning time-varying hyperparameters online from the sparse data stream due to the numerically approximated gradient of the marginal likelihood function. In this paper, we show how to extend the IHGP in order to learn time-varying hyperparameters using a sparse and non-stationary data stream. In particular, our solution approach offers the exact gradient as the solution of a Lyapunov equation. Therefore, our approach achieves better performance with a sparse data stream while still keeping the computational complexity low. Finally, we present the comparison results to demonstrate that our approach outperforms the originally proposed IHGP on practical applications with sparse data streams. To demonstrate the effectiveness of our approach, we consider a multi-rate sensor fusion or an interpolation problem where slow vision systems need to be combined with other fast sensory units for feedback control in the field of autonomous driving. In particular, we apply our approach to vehicle lateral position error estimation together with a deep learning model for autonomous driving using non-stationary lateral position error signals in a model-free and data-driven fashion.

Embedded Identification of Surface Based on Multirate Sensor Fusion With Deep Neural Network

In this letter, we propose a multivariate time-series classification system that fuses multirate sensor measurements within the latent space of a deep neural network. In our network, the system identifies the surface category based on audio and inertial measurements generated from the surface impact, each of which has a different sampling rate and resolution in nature. We investigate the feasibility of categorizing ten different everyday surfaces using a proposed convolutional neural network, which is trained in an end-to-end manner. To validate our approach, we developed an embedded system and collected 60 000 data samples under a variety of conditions. The experimental results obtained exhibit a test accuracy for a blind test dataset of 93%, taking less than 300 ms for end-to-end classification in an embedded machine environment. We conclude this letter with a discussion of the results and future direction of research.

Increasing Time Resolution of State Estimation Using Kalman Filtering

Рассматриваются вопросы получения оценки состояния процесса с использованием аппарата калмановской фильтрации при наличии измерителя, выдающего показания с временным разрешением меньше требуемого. Выделены два пути решения проблемы: повышение частоты следования отсчетов выходных данных фильтра при использовании данных одного датчика и введение в систему второго, удовлетворяющего требованию по временному разрешению выдаваемых данных, но недостаточно точного, с последующим комплексированием показаний двух датчиков. Проведен анализ известных методов повышения разрешения оценок при использовании одного измерителя, предложены модификации. С использованием этих результатов предлагаются варианты алгоритмов комплексирования, основанные на известных ранее разработанных для объединения показаний с одинаковыми частотами следования и адаптированные под рассматриваемую задачу, в том числе метод объединения измерений на входе фильтра Калмана, метод объединения векторов состояния и его последующие модификации. Также приводятся известные из литературы алгоритмы комплексирования, изначально разработанные для объединения показаний с различными частотами следования. Проводится сравнительный анализ результатов работы всех алгоритмов на примере оценивания процесса по измерениям с различными уровнями шумов, различным соотношением интервалов следования показаний датчиков, исследуется их поведение при различных частотных свойствах фильтрующих контуров. Приводятся соображения по выбору оптимального для конкретного набора значений метода. Проведено сравнение ошибки оценивания с получаемой при использовании одного измерителя, сделаны выводы относительно целесообразности использования каждого из двух обозначенных в начале путей. The problems of state estimation using the Kalman filtering in the presence of a sensor with a less time resolution of measurements compared to required one are considered. Two ways of solving the problem are highlighted: the output sampling rate lifting using a single sensor and introducing into the system a second one that satisfies the requirement for time resolution of the output data, but is not accurate enough, followed by a fusion of these readings. The analysis of known methods for the state estimates lifting using a single sensor is completed, some modifications are proposed. Using these results, some versions of fusion algorithms based on the known ones developed for combining measurements with the same sampling rates and adapted to considered problem, including measurement fusion before Kalman filter, state vector fusion and its succeeding modifications, are proposed. Well-known published fusion algorithms originally designed for multi-rate sensor fusion are also presented. A comparison of the results of all the algorithms is performed by state estimation using the measurements with various noise and various ratios of the intervals between samples, response to changing the band-pass properties of filter loop is studied. Some advances to choose the optimal method for a special set of values are proposed. A comparison with single sensor technique is completed by evaluation the estimation error, conclusion remarks on advisability of using each of two described techniques are given.

A multi-rate sensor fusion approach using information filters for estimating aero-engine performance degradation

Gas-path performance estimation plays an important role in aero-engine health management, and Kalman Filter (KF) is a well-known technique to estimate performance degradation. In previous studies, it is assumed that different kinds of sensors are with the same sampling rate, and they are used for state estimation by the KF simultaneously. However, it is hard to achieve state estimation using various kinds of sensor measurements at the same sampling rate due to a complex network and physical characteristic differences between sensors, especially in an advanced multi-sensor architecture. For this purpose, a multi-rate sensor fusion using the information filtering approach is proposed based on the square-root cubature rule, which is called Multi-rate Square-root Cubature Information Filter (MSCIF) to track engine performance degradation. Soft measurement synchronization of the MSCIF is designed to provide a sensor fusion condition for multiple sampling rates of measurement, and a fault sensor is isolated by maximum likelihood validation before state estimation. The contribution of this paper is to supply a novel multi-rate information filter approach for sensor fault tolerant health estimation of an aero-engine in a multi-sensor system. Tests are conducted for aero-engine performance degradation estimation with multiple sampling rates of sensor measurement on both digital simulation and semi-physical experiment. Experimental results illustrate the superiority of the proposed algorithm in terms of degradation estimation accuracy and robustness to sensor failure in a multi-sensor system.

The Fractional Kalman Filter-Based Asynchronous Multirate Sensor Information Fusion

A fractional Kalman filter-based multirate sensor fusion algorithm is presented to fuse the asynchronous measurements of the multirate sensors. Based on the characteristics of multirate and delay measurement, the state is reestimated at the time when the delayed measurement occurs by using weighted fractional Kalman filter, and then the state estimation is updated at the current time when the delayed measurement arrives following the similar pattern of Kalman filter. The simulation examples are given to illustrate the effectiveness of the proposed fusion method.

Centralized security-guaranteed filtering in multirate-sensor fusion under deception attacks

In this paper, the centralized security-guaranteed filtering problem is studied for linear time-invariant stochastic systems with multirate-sensor fusion under deception attacks. The underlying system includes a number of sensor nodes with a centralized filter, where each sensor is allowed to be sampled at different rate. A new measurement output model is proposed to characterize both the multiple rates and the deception attacks. By exploiting the lifting technique, the multi-rate sensor system is cast into a single-rate discrete-time system. With a new concept of security level, the aim of this paper is to design a filter such that the filtering error dynamics achieves the prescribed level of the security under deception attacks. By using the stochastic analysis techniques, sufficient conditions are first derived such that the filtering error system is guaranteed to have the desired security level, and then the filter gain is parameterized by using the semi-definite programme method with certain nonlinear constraints. Finally, a numerical simulation example is provided to demonstrate the feasibility of the proposed filtering scheme.

Model-reduced fault detection for multi-rate sensor fusion with unknown inputs

In multi-sensor fusion, it is hard to guarantee that all sensors work at the single sampling rate, especially in the distributive and/or heterogeneous case, and fault detection (FD) in multi-rate sensor fusion may face the existence of unknown inputs (UIs) in complex environment. Meanwhile, model reduction often refers to propose a possible lower-dimensional model to replace the original model without adding significant error in practical applications. By the fact that FD in dynamic systems should only focus on the fault-related controllability and observability characteristics, it is a good idea to obtain the fault-related controllable and observable subsystem via system decomposition (i.e., model reduction) for FD. Such a kind of model reduction not only guarantee the FD performance, but also reduce the system dimensions. To this end, we propose the model-reduced fault detection (MRFD) problem for multi-rate sensor fusion subject to UIs and faults imposed on the actuator and sensors. Our aim is to design a fast and computation-effective FD scheme based on the reduced model. We use the singular decomposition for UI decoupling, and then obtain the fault-related subsystem via controllability and observability decomposition. And then the multi-rate observer (MRO) with causality constraints is designed. Different from the traditional observer used for FD, the proposed MRO outputs the fault-related partial state estimate as soon as any a sensor measurement is received, resulting in fast and computation-effective FD. Furthermore, conditions for the existence of a stable MRO, fault-to-state controllability, and fault detectability are explored. A simulation example for simplified longitudinal flight control system and method comparison with the existing multi-rate FD algorithms show the effectiveness of the proposed MRFD method.

The joint optimal filtering and fault detection for multi-rate sensor fusion under unknown inputs

In multi-sensor fusion, it is hard to guarantee that all sensors have an identical sampling rate, especially in the distributive and/or heterogeneous case. Meanwhile, stochastic noise, unknown inputs (UIs), and faults may coexist in complex environment. To this end, we propose the problem of joint optimal filtering and fault detection (FD) for multi-rate sensor fusion subject to UIs, stochastic noise with known covariance, and faults imposed on the actuator and sensors. Furthermore, the new scheme of optimal multi-rate observer (MRO) is presented and applied to detect faults. The observer parameters are determined optimally in pursuit of the UI decoupling and maximizing noise attenuation under the causality constraint due to multi-rate nature. Finally, the output estimation error of the MRO is used as a residual signal for FD via a hypothesis test in which the threshold is adaptively designed according to the MRO parameters. One numerical example is given to show the effectiveness of our proposed method.

Transformations of Gaussian Process priors for user matching

We describe the use of transformations of Gaussian Process (GP) priors to improve the context sensing capability of a system composed of a Kinect sensor and mobile inertial sensors. The Bayesian nonparametric model provides a principled mechanism for incorporating the low-sampling-rate position measurements and the high-sampling-rate derivatives in multi-rate sensor fusion which takes account of the uncertainty of each sensor type. The complementary properties of these sensors enable the GP model to calculate the likelihood of the observed Kinect skeletons and inertial data to identify individual users.We conducted three experiments to test the performance of the proposed GP model: (1) subtle hand movements, (2) walking with a mobile device in the trouser pocket, and (3) walking with a mobile device held in the hand. We compared the GP with the direct acceleration comparison method. Experimental results show that the GP approach can achieve successful matches (with mean accuracy µ 90 % ) in all 3 contexts, including when there are only subtle hand movements, where the acceleration comparison method performs poorly ( µ < 20 % ) . HighlightsWe explore the complementary properties of Kinect sensor and inertial sensors.We describe a GP model to improve the context sensing in proxemic interactions.The model incorporates positions and accelerations in multi-rate sensor fusion.We identify users by matching Kinect skeletons with the sensed data from devices.User matching and identification through people's everyday movements is feasible.

Fault detection for multi‐rate sensor fusion under multiple uncertainties

In multi-sensor fusion, it is hard to guarantee that all sensors have an identical sampling rate, especially in the distributive and/or heterogeneous case. Meanwhile, system modelling may face the coexistence of multiple uncertainties including stochastic noise, unknown input (UI) and faults in complex environment. To this end, the authors propose the problem of fault detection for multi-rate sensor fusion systems subject to UI, stochastic noise with known covariance, and faults imposed on the actuator and sensors. Furthermore, the new form of multi-rate observer (MRO) is presented and lifted to the single-rate one with causality constraint for parameter design. Observer parameters are determined optimally in pursuit of the UI decoupling and maximum noise attenuation under the causality constraint. Differing from the traditional observer, the proposed MRO is time varying, that is, its parameters need recursive computation and hence has better adaptability to the effect of uncertainties. Finally, a multi-rate residual generator is constructed via a hypothesis test in which the threshold is adaptively designed. A numerical example is given to show the effectiveness of their proposed method.

다중주기 칼만 필터를 이용한 비동기 센서 융합

We propose a multi-rate sensor fusion of vision and radar using Kalman filter to solve problems of asynchronized and multi-rate sampling periods in object vehicle tracking. A model based prediction of object vehicles is performed with a decentralized multi-rate Kalman filter for each sensor (vision and radar sensors.) To obtain the improvement in the performance of position prediction, different weighting is applied to each sensor’s predicted object position from the multi-rate Kalman filter. The proposed method can provide estimated position of the object vehicles at every sampling time of ECU. The Mahalanobis distance is used to make correspondence among the measured and predicted objects. Through the experimental results, we validate that the post-processed fusion data give us improved tracking performance. The proposed method obtained two times improvement in the object tracking performance compared to single sensor method (camera or radar sensor) in the view point of roots mean square error.

Linear‐minimum‐mean‐square‐error observer for multi‐rate sensor fusion with missing measurements

This note presents the problem of designing the linear-minimum-mean-square-error observer for a class of multi-rate sensor fusion systems with missing measurements. Under the casuality constraint because of the multi-rate nature, the covariances of the equivalent noises in the estimation error system are obtained via multi-rate recursive computation. Through minimising the traces of the covariances of the estimation errors, the optimal observer is obtained. Fortunately, all the observer parameters can be calculated off-line. A numerical example is given to show the effectiveness of the proposed observer.

Multi-rate sensor fusion-based adaptive discrete finite-time synergetic control for flexible-joint mechanical systems

This paper proposes an adaptive discrete finite-time synergetic control (ADFTSC) scheme based on a multi-rate sensor fusion estimator for flexible-joint mechanical systems in the presence of unmeasured states and dynamic uncertainties. Multi-rate sensors are employed to observe the system states which cannot be directly obtained by encoders due to the existence of joint flexibilities. By using an extended Kalman filter (EKF), the finite-time synergetic controller is designed based on a sensor fusion estimator which estimates states and parameters of the mechanical system with multi-rate measurements. The proposed controller can guarantee the finite-time convergence of tracking errors by the theoretical derivation. Simulation and experimental studies are included to validate the effectiveness of the proposed approach.

비전 센서와 자이로 센서의 융합을 통한 보행 로봇의 자세 추정

Tilt sensor is required to control the attitude of the biped robot when it walks on an uneven terrain. Vision sensor, which is used for recognizing human or detecting obstacles, can be used as a tilt angle sensor by comparing current image and reference image. However, vision sensor alone has a lot of technological limitations to control biped robot such as low sampling frequency and estimation time delay. In order to verify limitations of vision sensor, experimental setup of an inverted pendulum, which represents pitch motion of the walking or running robot, is used and it is proved that only vision sensor cannot control an inverted pendulum mainly because of the time delay. In this paper, to overcome limitations of vision sensor, Kalman filter for the multi-rate sensor fusion algorithm is applied with low-quality gyro sensor. It solves limitations of the vision sensor as well as eliminates drift of gyro sensor. Through the experiment of an inverted pendulum control, it is found that the tilt estimation performance of fusion sensor is greatly improved enough to control the attitude of an inverted pendulum.