Multi-sensor System Research Articles

Condition Monitoring and Fault Detection of Wind Turbine Driveline With the Implementation of Deep Residual Long Short-Term Memory Network

In view of the wide application of wind power, it is critically essential to develop solutions to the high fault rate and the long mean time to repair (MTTR). Multi-sensor fusion methods have been widely applied in condition monitoring and anomaly detection of industrial installation. Aimed at the wind turbine driveline, a new method utilizing the deep residual LSTM network with Attention model (ResLSTM-AM) is proposed in this paper. Firstly, data from multi-sensor fusion systems like supervisory control and data acquisition (SCADA) is preprocessed by quartile data cleaning and corresponding selection using the calculation of maximal information coefficient (MIC) to enhance its reliability. Secondly, we establish the neural network by the deep residual network (ResNet), long short-term memory network (LSTM), and attention mechanism (AM) for time series forecasting of wind turbines. The newly added residual connecting architecture in LSTM layers ensures a higher learning rate in feature extraction and provides a more flexible data flow both in forward and backward signals. Additionally, the attention mechanism after a LSTM layer amplifies the influence of vital data. The model gets validated based on historical SCADA data from actual wind turbines containing healthy conditions and anomaly conditions, showing ResNet contributes more. Based on the root-mean-square error (RMSE), the alarming threshold is calculated by an exponential weighted moving average (EWMA). For two chosen variables, the proposed method is confirmed to be efficient and reliable with the accuracy of 0.9945 and 0.9880 in the results of trials and comparisons with other existing models.

A State Estimation Method for Multisensor Uncertain Systems Based on Sequential Fusion and Zonotope

Most actual multi-sensor systems are usually affected by unexpected distractions such that the probability distributions of noises may become unforeseeable. For the classical state estimation methods, if the probability distributions of noises are not accurate, it will cause large estimation errors or even filtering performance divergence. Therefore, a fast and accurate measurement-sequence-fusion-zonotope (MSF-Z) state estimation method is proposed, which is the first attempt for multi-sensor sequential fusion to use only the boundary knowledge without using probability distributions of noises. The method firstly gives a family of zonotopes containing the intersection of the predicted state set and the measurement polytope, where all measurements are fused one by one according to their time sequence arriving at the fusion center, which improves the real-time performance, meanwhile, the dimension of the output matrix is reduced, which reduces the computational complexity. Then, find a zonotope with the smallest volume as the external description of the intersection. Next, obtain the compact set of the estimated state using the reduction operator to improve accuracy, which effectively solves the problem of multi-sensor state estimation when the probability distributions of noises are unknown. Finally, the simulation of a turbofan engine control system verifies that the proposed method has high accuracy and good time performance.

Globally optimal distributed and sequential state fusion filters for multi-sensor systems with correlated noises

The problem concerning the multi-sensor fusion filtering for systems with correlated system and measurement noises is studied in this paper. Each sensor produces a local predictor by using its own measurements and then sends it to a fusion center. In the fusion center, by applying the projection theory and difference technology, the globally optimal distributed state fusion filter without feedback and sequential state fusion filter with recursive estimators of noises are presented in the sense of linear minimum variance by using all received local predictors, respectively. Under the condition that local prediction gain matrices are of full column rank, the proposed distributed and sequential state fusion filters can achieve the same estimation accuracy as the centralized fusion filter, i.e., they both have global optimality. The optimality is strictly proved. The corresponding steady-state fusion filters are also explored. A sufficient condition for the convergence of the proposed fusion filters is given. A target tracking example verifies the effectiveness of the proposed algorithms.

Potentiometric Sensor Arrays Based on Hybrid PFSA/CNTs Membranes for the Analysis of UV-Degraded Drugs.

The degradation of drugs is a substantial problem since it affects the safety and effectiveness of pharmaceutical products, as well as their influence on the environment. A novel system of three potentiometric cross-sensitive sensors (using the Donnan potential (DP) as an analytical signal) and a reference electrode was developed for the analysis of UV-degraded sulfacetamide drugs. The membranes for DP-sensors were prepared by a casting procedure from a dispersion of perfluorosulfonic acid (PFSA) polymer, containing carbon nanotubes (CNTs), whose surface was preliminarily modified with carboxyl, sulfonic acid, or (3-aminopropyl)trimethoxysilanol groups. A correlation between the sorption and transport properties of the hybrid membranes and cross-sensitivity of the DP-sensor to sulfacetamide, its degradation product, and inorganic ions was revealed. The analysis of the UV-degraded sulfacetamide drugs using the multisensory system based on hybrid membranes with optimized properties did not require a pre-separation of the components. The limits of detection of sulfacetamide, sulfanilamide, and sodium were 1.8 × 10-7, 5.8 × 10-7, and 1.8 × 10-7 M. The relative errors of the determination of the components of the UV-degraded sulfacetamide drugs were 2-3% (at 6-8% relative standard deviation). PFSA/CNT hybrid materials provided the stable work of the sensors for at least one year.

Electrical properties and thermally induced aging processes in eco-friendly Cu0.1Ni0.1Co1.6Mn1.2O4 and Cu0.1Ni0.8Co0.2Mn1.9O4-based thick-film structures for multisensor systems

The electrical properties and kinetic dependences of thermally induced drift of electrical resistance in eco-friendly single- and multilayer temperature-sensitive thick-film structures based on Cu0.1Ni0.1Co1.6Mn1.2O4 (with p+-type electrical conductivity) and Cu0.1Ni0.8Co0.2Mn1.9O4 (with p-type of electrical conductivity) ceramics for multisensor nanosystems. It has been established that in the p- and p+-type thick films the processes of burning out the residues of the organic bond between grain spinels and simultaneous penetration of metallic silver into the space are activated. Such processes are described by a compressed exponential-power relaxation function. Thick-film p+-p structures are characterized by own aging processes, which manifest themselves in an increase in electrical resistance. Kinetics of their thermally-induced aging is adequately described by an extended exponential-power function.

Machine learning-based soft-sensor development for road quality classification

Vibrations in road vehicles cause several harmful effects, health problems can occur for the passengers, and mechanical damage can occur to the vehicle components. Given the health, safety, and financial issues that arise, keeping the road network in good condition and detecting road defects as early as possible requires an extensive monitoring system. Related to this, our study presents the development of hardware and software for a low-cost, multi-sensor road quality monitoring system for passenger vehicles. The developed monitoring system can classify road sections according to their quality parameters into four classes. In order to detect vibrations in the vehicle, accelerometers and gyroscope sensors are installed at several points. Then, a machine learning-based soft-sensor development is introduced. Besides noise filtering, each data point is resampled by spatial frequency to reduce the velocity dependence. Subsequently, a decision tree-based classification model is trained using features from the power spectrum and principal component analysis. The classification algorithm is validated and tested with measurement data in a real-world environment. In addition to reviewing the accuracy of the model, we examine the correlation of the data measured in the cabin and on the suspension to see how much additional information is provided by the sensor on the axle.

SEAMLESS REALTIME LANE LEVEL VEHICULAR NAVIGATION IN GNSS CHALLENGING ENVIRONMENTS USING A RTK GNSS/IMU/VINS INTEGRATION SCHEME

Abstract. Outdoor positioning requires a reliable solution that can work in environments where satellite signals are often blocked or degraded. Global Navigation Satellite System (GNSS) is a common choice, but it may not provide accurate results for land vehicles. To address this challenge, this research proposes a multi-sensor integrated system for vehicle navigation that combines GNSS with other sensors. The system uses Extended Kalman Filter (EKF) to fuse the data from different sources and improve the navigation performance. The algorithm targets to provide seamless navigation for urban environments as well as various indoor environments fields with INS/GNSS/VIO aiding integrated solutions. The experimental vehicle of this research is equipped with a tactical-grade inertial sensing measurement unit (IMU) as the test system, a self-designed and assembled visual platform, which includes a camera with a time synchronization protocol and a low-cost IMU. Also, both indoor experimental fields and outdoor urban scenarios with different high challenging were tested to verify the developed algorithm. To evaluate the performance of the proposed real-time navigation system, we use a high-accuracy navigation-grade system as a reference, which provides a stable and reliable trajectory. The result indicates that using the GNSS RTK solution with VIO aiding integration scheme reduced the RMS errors in long outage (450 sec, 1812 m) by 87.4% and 79.9% in position and velocity error, respectively. In urban scenario, the along-track/cross-track maximum errors can achieve 1.4 m / 1.5 m. Overall, these contribute to the development of real-time navigation systems for self-driving vehicle in the future.

A novel low-cost multi-sensor solution for pavement distress segmentation and characterization at night

The pavement distress measurement is a crucial aspect in guaranteeing the safety of transportation infrastructure. In this regard, we introduce a novel and cost-effective multi-sensor approach for pavement segmentation during low-light night conditions. By utilizing the low-cost Azure Kinect multi-sensor system, we generated a multi-sensor dataset that encompasses aligned IR, RGB, and depth images. Then we carried out the data annotation process on the RGB images. A total of 11,343 manual annotations were meticulously made on 791 images, which were randomly selected from a collection of 96,891 frames. Subsequently, four different deep learning-based image segmentation models were analyzed both quantitatively and qualitatively. The results indicated that the segmentation performance on the IR dataset outperformed that of the RGB dataset. The model with the highest mIoU (mean Intersection over Union) of 0.7169 was ConvNext when trained on the IR dataset. Furthermore, we proposed the use of relative height for evaluating the severity of pavement distress. On the aligned depth map, the relative height was calculated using the depth data from the corresponding pavement distress area. Additionally, a quantitative comparison between manual annotations and the results obtained through deep learning revealed that the latter was more effective in identifying more severe forms of pavement distress. Through this study, we established the feasibility of collecting pavement distress data during nighttime using a low-cost multi-sensor system.

Mission performance analysis of phased-mission systems with cross-phase competing failures

Phased-mission systems (PMSs) with sophisticated failure modes could behave diverse mission performance levels. The non-negligible competing failures, resulting from complicated components’ failure patterns, have posed a great challenge for the mission performance analysis of PMSs. Even though many methods have been developed to facilitate reliability assessment of PMSs with competing failures in the past few decades, the mission performance analysis of PMSs with competing failures, varying from phase to phase, is still in difficulty. This feature is defined as the cross-phase competing failure, characterized by a two-by-two crossover combination of three or more failure modes in distinct mission phases. In this study, mission performance analysis of PMSs with cross-phase competing failures is conducted. By the divide-and-conquer strategy, the conditional probability of paths, resulting from the phase-dependent effects of components, is computed via an iterative method. Based on the phase equivalent binary decision diagrams, the mission performance analysis of PMSs with cross-phase competing failures is conducted. The probability distribution of mission performance levels, together with the system reliability, is assessed phase-by-phase based on the lifetime distributions of components and mission durations. A distributed multi-sensor multi-target tracking system, along with a set of comparative studies, is given to demonstrate the effectiveness and applicability of the proposed method. The results show that the proposed approach to mission performance analysis of PMSs is more computationally efficient than the existing combinational models.

Charting infants’ motor development at home using a wearable system: validation and comparison to physical growth charts

Early neurodevelopmental care and research are in urgent need of practical methods for quantitative assessment of early motor development. Here, performance of a wearable system in early motor assessment was validated and compared to developmental tracking of physical growth charts. Altogether 1358h of spontaneous movement during 226 recording sessions in 116 infants (age 4-19 months) were analysed using a multisensor wearable system. A deep learning-based automatic pipeline quantified categories of infants' postures and movements at a time scale of seconds. Results from an archived cohort (dataset 1, N=55 infants) recorded under partial supervision were compared to a validation cohort (dataset 2, N=61) recorded at infants' homes by the parents. Aggregated recording-level measures including developmental age prediction (DAP) were used for comparison between cohorts. The motor growth was also compared with respective DAP estimates based on physical growth data (length, weight, and head circumference) obtained from a large cohort (N=17,838 infants; age 4-18 months). Age-specific distributions of posture and movement categories were highly similar between infant cohorts. The DAP scores correlated tightly with age, explaining 97-99% (94-99% CI 95) of the variance at the group average level, and 80-82% (72-88%) of the variance in the individual recordings. Both the average motor and the physical growth measures showed a very strong fit to their respective developmental models (R2=0.99). However, single measurements showed more modality-dependent variation that was lowest for motor (σ=1.4 [1.3-1.5 CI 95] months), length (σ=1.5 months), and combined physical (σ=1.5 months) measurements, and it was clearly higher for the weight (σ=1.9 months) and head circumference (σ=1.9 months) measurements. Longitudinal tracking showed clear individual trajectories, and its accuracy was comparable between motor and physical measures with longer measurement intervals. A quantified, transparent and explainable assessment of infants' motor performance is possible with a fully automated analysis pipeline, and the results replicate across independent cohorts from out-of-hospital recordings. A holistic assessment of motor development provides an accuracy that is comparable with the conventional physical growth measures. A quantitative measure of infants' motor development may directly support individual diagnostics and care, as well as facilitate clinical research as an outcome measure in early intervention trials. This work was supported by the Finnish Academy (314602, 335788, 335872, 332017, 343498), Finnish Pediatric Foundation (Lastentautiensäätiö), Aivosäätiö, Sigrid Jusélius Foundation, and HUS Children's Hospital/HUS diagnostic center research funds.

Asynchronous Anti-Bias Track-to-Track Association Algorithm Based on Nearest Neighbor Interval Average Distance for Multi-Sensor Tracking Systems

Due to sensor characteristics, geographical environment, electromagnetic interference, electromagnetic silence, information countermeasures, and other reasons, there may be significant system errors in sensors in multi-sensor tracking systems, resulting in poor track-to-track association (TTTA) effect of the system. In order to solve the problem of TTTA under large system errors, this paper proposes an asynchronous anti-bias TTTA algorithm that utilizes the average distance between the nearest neighbor intervals between tracks. This algorithm proposes a systematic error interval processing method to track coordinates, and then defines the nearest neighbor interval average distance between interval coordinate datasets and interval coordinate points, and then uses grey theory to calculate the correlation degree between tracks. Finally, the Jonker–Volgenant algorithm is combined to use the canonical allocation method for TTTA judgment. The algorithm requires less prior information and does not require error registration. The simulation results show that the algorithm can ensure a high average correct association rate (over 98%) of asynchronous unequal rate tracks under large system errors, and achieve stable association, with good association and anti-bias performance. Compared with other algorithms, the algorithm maintains good performance for different target numbers and processing cycles, and has good superiority and robustness.

Assessing the Value of Multimodal Interfaces: A Study on Human-Machine Interaction in Weld Inspection Workstations.

Multimodal user interfaces promise natural and intuitive human-machine interactions. However, is the extra effort for the development of a complex multisensor system justified, or can users also be satisfied with only one input modality? This study investigates interactions in an industrial weld inspection workstation. Three unimodal interfaces, including spatial interaction with buttons augmented on a workpiece or a worktable, and speech commands, were tested individually and in a multimodal combination. Within the unimodal conditions, users preferred the augmented worktable, but overall, the interindividual usage of all input technologies in the multimodal condition was ranked best. Our findings indicate that the implementation and the use of multiple input modalities is valuable and that it is difficult to predict the usability of individual input modalities for complex systems.

Set-membership multi-sensor secure fusion estimation against two-channel malicious attacks

In this study, the set-membership secure fusion estimation problem against malicious attacks is addressed. In practice, adversaries are able to tamper with the measurement outputs and local estimates from different communication channels. By rewriting the attacked information into two parts that are related and unrelated to the initial value, a novel secure estimation model is developed to describe the multi-sensor fusion systems that are subject to two-channel bounded attacks. An existence condition of the set-membership secure fusion estimator is presented to guarantee that the secure fusion estimation errors are always confined to expected ellipsoids. Subsequently, the secure fusion weight matrices are determined by solving a constructed convex optimization problem constrained by a linear matrix inequality condition. Lastly, the rationality of the designed set-membership secure fusion estimation algorithm is verified using two simulation examples.

Juice Quality Evaluation with Multisensor Systems—A Review

E-nose and e-tongue are advanced technologies that allow for the fast and precise analysis of smells and flavours using special sensors. Both technologies are widely used, especially in the food industry, where they are implemented, e.g., for identifying ingredients and product quality, detecting contamination, and assessing their stability and shelf life. Therefore, the aim of this article is to provide a comprehensive review of the application of e-nose and e-tongue in various industries, focusing in particular on the use of these technologies in the fruit and vegetable juice industry. For this purpose, an analysis of research carried out worldwide over the last five years, concerning the possibility of using the considered multisensory systems to test the quality and taste and aroma profiles of juices is included. In addition, the review contains a brief characterization of these innovative devices through information such as their origin, mode of operation, types, advantages and disadvantages, challenges and perspectives, as well as the possibility of their applications in other industries besides the juice industry.

Sequential Fusion Filter for State Estimation of Nonlinear Multi-Sensor Systems with Cross-Correlated Noise and Packet Dropout Compensation

This paper is concerned with the problem of state estimation for nonlinear multi-sensor systems with cross-correlated noise and packet loss compensation. In this case, the cross-correlated noise is modeled by the synchronous correlation of the observation noise of each sensor, and the observation noise of each sensor is correlated with the process noise at the previous moment. Meanwhile, in the process of state estimation, since the measurement data may be transmitted in an unreliable network, data packet dropout will inevitably occur, leading to a reduction in estimation accuracy. To address this undesirable situation, this paper proposes a state estimation method for nonlinear multi-sensor systems with cross-correlated noise and packet dropout compensation based on a sequential fusion framework. Firstly, a prediction compensation mechanism and a strategy based on observation noise estimation are used to update the measurement data while avoiding the noise decorrelation step. Secondly, a design step for a sequential fusion state estimation filter is derived based on an innovation analysis method. Then, a numerical implementation of the sequential fusion state estimator is given based on the third-degree spherical-radial cubature rule. Finally, the univariate nonstationary growth model (UNGM) is combined with simulation to verify the effectiveness and feasibility of the proposed algorithm.

Sequential fusion estimation for Markov jump systems with heavy-tailed noises

We study a sequential fusion estimation problem for Markov jump multi-sensor systems with heavy-tailed noises. By modelling the noises as Student's t distributions, a sequential fusion estimation algorithm is designed by utilising the interacting multiple model method and Bayes' rule. To improve the robustness against measurement outliers caused by measurement heavy-tailed noise, an F-distribution detection strategy is designed to detect and reject the measurement outliers. Simulation results demonstrate that the designed sequential fusion estimation algorithm can effectively fuse the measurements from multiple sensors, and the accuracy of the designed algorithm is superior to the existing interacting multiple model Student's t batch fusion algorithm and single model adaptive Student's t batch fusion algorithm when there exist model switching and disturbances with heavy-tailed property.

Investigation of a wearable piezoelectric-IMU multi-modal sensing system for real-time muscle force estimation

Piezoelectric based ultrasonic transducer shows a promising application prospect in the wearable muscle force estimator by detecting the morphological-biochemical peculiarity of human motion. However, due to the nonlinearity of muscle contraction, muscle force estimation in a dynamic motion, such as leg lifting, is still a challenge. In this study, a wearable multi-sensory system was developed for muscle force estimation in the isometric contraction assessment and during the dynamic training. A customized wearable ultrasound system was adopted for real-time deformation measurement of muscle, and an inertial measurement unit sensor was utilized to detect the joint angle. Thus, the muscle force can be predicted by identifying the muscle deformation as well as considering the muscle thickness change caused by the joint angel variation. The robustness and efficiency of the system was investigated by evaluating the muscle force of the rectus femoris during the isometric contraction assessment and the knee’s dynamic exercise. The accuracy of muscle force prediction is over 90%. During the knee’s dynamic exercise, the predicted force output of the lower-limb agreed well with the measured value, demonstrating the promising application of the system in dynamic muscle force estimation. This approach can provide real-time muscle force information for the patients to improve the rehabilitative training effect when using an exoskeletal rehabilitation robot as well as evaluate their recovery situation.

Distributed variational Bayesian adaptive filtering for randomly delayed measurements and unknown noise statistics in multi-sensor networked systems

To deal with randomly delayed measurements and unknown noise statistics in multi-sensor networked systems, a distributed variational Bayesian adaptive cubature Kalman filtering algorithm (DVB_ACKF) is presented in this paper. To compensate for the effects induced by randomly delayed measurements, a refined measurement reorganization method is designed via a two-stage estimation strategy. Specifically, we first estimate the time delay step. The measurements are then rearranged based on the estimated results from the first stage. Then, a novel noise estimator with the sliding-window and variational Bayesian method is developed to estimate the mean and covariance of the unknown noise. The network topology attribute and measurement accuracy have been combined to eliminate the distributed fusion error. The tuning parameter and forgetting factor as well as their bounds are analyzed. Furthermore, numerical simulations are carried out to verify the superior performance of the developed algorithm.

An ROI Optimization Method Based on Dynamic Estimation Adjustment Model

An important research direction in the field of traffic light recognition of autonomous systems is to accurately obtain the region of interest (ROI) of the image through the multi-sensor assisted method. Dynamic evaluation of the performance of the multi-sensor (GNSS, IMU, and odometer) fusion positioning system to obtain the optimum size of the ROI is essential for further improvement of recognition accuracy. In this paper, we propose a dynamic estimation adjustment (DEA) model construction method to optimize the ROI. First, according to the residual variance of the integrated navigation system and the vehicle velocity, we divide the innovation into an approximate Gaussian fitting region (AGFR) and a Gaussian convergence region (GCR) and estimate them using variational Bayesian gated recurrent unit (VBGRU) networks and a Gaussian mixture model (GMM), respectively, to obtain the GNSS measurement uncertainty. Then, the relationship between the GNSS measurement uncertainty and the multi-sensor aided ROI acquisition error is deduced and analyzed in detail. Further, we build a dynamic estimation adjustment model to convert the innovation of the multi-sensor integrated navigation system into the optimal ROI size of the traffic lights online. Finally, we use the YOLOv4 model to detect and recognize the traffic lights in the ROI. Based on laboratory simulation and real road tests, we verify the performance of the DEA model. The experimental results show that the proposed algorithm is more suitable for the application of autonomous vehicles in complex urban road scenarios than the existing achievements.

An improved LSTM-based model for identifying high working intensity load segments of the tractor load spectrum

In order to solve the problems of serious redundancy of existing load spectrum data, unclear identification of load segments corresponding to high working intensity, and failure of load spectrum interception to meet the requirements of heavy load conditions, this paper proposes an MHA-ConvLSTM (Multi-head attention Convolutional LSTM) network model for identifying high working intensity load segments of the tractor load spectrum. The deep learning model integrates a multi-head attention mechanism and ConvLSTM network as the core, keeping the temporal order of continuously changing load sequences as the basic principle, deeply mining the local features contained within the small range load of dynamically changing load, and strengthening the matching relationship of the intrinsic features in long distance and large data volume load. This research selects rotary tillage as the verification condition, builds the multi-sensor test system to carry out the working load test, and takes the tractor rotary tillage load spectrum data as the validation object. The analysis shows that the accuracy and F1-score of the MHA-ConvLSTM model reach 97.69% and 97.83%, respectively, and the operation time is only 0.5289 s, which is 15.82% faster than LSTM. In addition, the model in this paper was used to identify 399 load segments with high workload and high work intensity, and 388 load segments were successfully verified with an error rate of less than 3%. This paper provides a new technical solution for applying the agricultural equipment load spectrum.