Multi-sensor Data Research Articles

Particle accelerator power system early fault diagnosis based on deep learning and multi-sensor feature fusion

Particle accelerators play a crucial role in scientific research and industrial applications, and enhancing their reliability, ensuring stable operation, and reducing downtime caused by faults are essential for achieving research goals. This paper introduces a novel particle accelerator fault diagnosis method based on deep learning and multi-sensor feature fusion. The approach employs one-dimensional convolution to extract signals from multiple sensors and achieves comprehensive feature fusion of multi-sensor data, effectively overcoming the limitations of individual sensors. It combines 1d convolutional neural networks (1D-CNN) and long short-term memory networks (LSTM) to enhance spatiotemporal feature extraction. This approach efficiently extracts features from multiple sensors while concurrently reducing data length and training time. The study uses real signals from particle accelerators and aims to achieve early detection of faults by identifying abnormal signals preceding accelerator malfunctions. Comparative analysis with other machine learning models and the use of multiple evaluation metrics validate the effectiveness and generalizability of this method. The research provides a better performance fault diagnosis model with significant implications for reducing downtime, troubleshooting faults, and improving the reliability of particle accelerators.

Seasonal patterns and processes of migration in a long-distance migratory bird: energy or time minimization?

Optimal migration theory prescribes adaptive strategies of energy, time or mortality minimization. To test alternative hypotheses of energy- and time-minimization migration we used multisensory data loggers that record time-resolved flight activity and light for positioning by geolocation in a long-distance migratory shorebird, the little ringed plover, Charadrius dubius. We could reject the hypothesis of energy minimization based on a relationship between stopover duration and subsequent flight time as predicted for a time minimizer. We found seasonally diverging slopes between stopover and flight durations in relation to the progress (time) of migration, which follows a time-minimizing policy if resource gradients along the migration route increase in autumn and decrease in spring. Total flight duration did not differ significantly between autumn and spring migration, although spring migration was 6% shorter. Overall duration of autumn migration was longer than that in spring, mainly owing to a mid-migration stop in most birds, when they likely initiated moult. Overall migration speed was significantly different between autumn and spring. Migratory flights often occurred as runs of two to seven nocturnal flights on adjacent days, which may be countering a time-minimization strategy. Other factors may influence a preference for nocturnal migration, such as avoiding flight in turbulent conditions, heat stress and diurnal predators.

Digital outcome measures from smartwatch data relate to non-motor features of Parkinson’s disease

Monitoring of Parkinson’s disease (PD) has seen substantial improvement over recent years as digital sensors enable a passive and continuous collection of information in the home environment. However, the primary focus of this work has been motor symptoms, with little focus on the non-motor aspects of the disease. To address this, we combined longitudinal clinical non-motor assessment data and digital multi-sensor data from the Verily Study Watch for 149 participants from the Parkinson’s Progression Monitoring Initiative (PPMI) cohort with a diagnosis of PD. We show that digitally collected physical activity and sleep measures significantly relate to clinical non-motor assessments of cognitive, autonomic, and daily living impairment. However, the poor predictive performance we observed, highlights the need for better targeted digital outcome measures to enable monitoring of non-motor symptoms.

Fault diagnosis of axial piston pump based on multi-source subdomain adaptation and sensor data fusion

The axial piston pump is the core component in hydraulic systems. Its condition monitoring and fault diagnosis are crucial to ensure the safe and reliable operation of hydraulic systems. However, most of the existing fault diagnosis methods for axial piston pumps use the same working condition data. In actual operation, axial piston pump often experience varying loads, and the collected data is typically correlated but has different distributions. Therefore, a transfer learning method of multi-source subdomain adaptation and sensor fusion (MSASF) is proposed for fault diagnosis of axial piston pump. The proposed MSASF has three modules, a shared feature extraction module, a domain-specific feature extraction module and an output decision module. Firstly, the adaptive weighted fusion of multi-sensor data features is realized by the shared feature extraction module and the common features of multi-source heterogeneous data are extracted. Secondly, in the domain-specific feature extraction module, multi-branch network is used to extract features of each pair of source and target domains and the local maximum mean difference is utilized to align the sub-domain distribution of each pair of source and target domains. Finally, in the output decision module, the distribution distance between each pair of source domain and target domain is calculated using maximum mean discrepancy to obtain its weighted score. Combined with the classification output of each source domain, the final diagnosis decision is made. A dataset was constructed for the axial piston pump design fault experiment, and four sets of transfer tasks were designed to compare with those of seven classic methods. The experimental results showed that the proposed MSASF method exhibits a superior domain adaptation effect and fault diagnosis performance.

Analysis of the Motion Postures in Equestrian Sports Based on Multi-Sensor Data Fusion

Horse riding, in its essence, is both an art and a captivating sport. In these events, riders showcase their refined skills and deep bond with their horses, most notably through changes in physical positioning. This research aims to accurately capture and analyze the spatial postures of riders in equestrian sports, exploring the dynamic differences between professional and amateur riders. To achieve this, we employed a multi-sensor data fusion approach based on human kinetics theories, enhanced by an extended Kalman filter. This method, combined with an optical tracking system, enabled us to intricately compare and analyze the 3D postures of riders during walking and trotting phases. Our research captured these nuanced postural shifts using an inertial sensor network, yielding nine-axis motion data of riders during their performance. The accuracy of our fusion technique was validated using the optical tracking system. By analyzing the motion data, we discerned posture variations among the riders of differing expertise levels, even when executing the same riding technique. The insights from this research offer a quantifiable metric for refining equestrian training and contribute to understanding the complex dynamics of rider–horse interaction.

Improved fault detection and diagnosis using graph auto encoder and attention-based graph convolution networks

A powerful fault detection and diagnosis (FDD) system plays a pivotal role in achieving operational excellence by maximizing system performance, optimizing maintenance strategies, and ensuring the longevity and resilience of process plants. In the context of FDD for multivariate sensor data, this study presents an improved FDD approach using graph-based neural networks. This graph neural network uses an adjacency matrix developed by extracting the expert domain knowledge and topological information of the multi-sensor system. This additional graph representation of the system is incorporated along with multivariate sensor data to capture the spatial and temporal information in neural networks efficiently. In this regard, we propose and evaluate: 1) A Graph Auto Encoder (GAE) based fault detection strategy and 2) An Attention-based Spatial Temporal Graph Convolution Network (ASTGCN) based fault diagnosis methodology. By leveraging the additional knowledge in the form of graphs, the GAE captures the complex relationships and dependencies among sensors, enabling effective anomaly detection, which identifies abnormal patterns and deviations from normal behavior, thus indicating potential faults in the system. The ASTGCN incorporates attention mechanisms to selectively focus on relevant sensor nodes and capture their spatial and temporal dependencies for fault diagnosis. The effectiveness of the proposed FDD approach is demonstrated using the benchmark Tennessee Eastman Process (TEP) problem. The results show that the proposed approaches outperform traditional methods and highlight the importance of leveraging graph-based knowledge for FDD in complex systems.

Changes in Convective Precipitation Reflectivity over the CONUS Revealed by High-Resolution Radar Observations from 2015 to 2021

The change in extreme precipitation events in the conterminous United States (CONUS) has been of interest to the research communities in recent years for its intensification under environmental and climate change. Previous studies have not yet used sub-hourly precipitation observations to examine convective precipitation change over the CONUS. This study aims to fill the gap by examining convective precipitation, identified by radar reflectivity, in the CONUS using the state-of-the-art Multi-radar Multi-sensor data, operated at the NOAA/National Severe Storms Laboratory, with an unprecedentedly high spatial (1 km) and temporal (2 min) resolutions. These high-resolution data are expected to better capture the precipitation peak and the precipitation pattern. The results showed that in CONUS, precipitation reflectivity increased both in magnitude and the number of convective days from 2015 to 2021. For example, in 2019, 60% of areas showed an increase in the magnitude of precipitation, and the average number of convective days over CONUS has increased by 19%. Changes in precipitation also vary by season and region. This study highlights the need for continued monitoring and understanding of the evolving pattern of extreme precipitation in the CONUS, especially at sub-hourly frequency, as it exposes significant impacts on various sectors, including agriculture, infrastructure, and human health.

Physics-Informed Approximation of Internal Thermal History for Surface Deformation Predictions in Wire Arc Directed Energy Deposition

Abstract This work presents a physics-informed fusion methodology for deformation detection using multi-sensor thermal data. A challenge with additive manufacturing (AM) is that abnormalities commonly occur due to rapid changes in the thermal gradient. Different non-destructive in-situ thermal sensors capture parts of the thermal history but are limited by the visible temperature spectrum and sensor field of view of the fabrication process. Various sensors mitigate problems with the loss of thermal history information; however, it brings forth challenges with integrating different data streams and the need to interpolate the internal thermal history. This study develops a thermal data-informed heat flux methodology that fills the gap in fusing numerical temperature approximation with data-driven knowledge of the surface of additive manufactured components. First, this study fuses infrared (IR) thermal data complexities during the AM process with the Goldak double ellipsoidal heat flux to model the energy input into the component. Second, a thermal physics-informed model input (PIMI) is created with thermal data-informed heat flux to capture internal thermal history. Lastly, a regression convolutional neural network (CNN) captures the relationship between the three-dimensional thermal gradient and the resulting surface deformation. The rapid thermal gradient formation and identification of deformation is a key step toward using thermal history data and machine learning to improve quality control in AM. The proposed surface deformation detection model achieved an mean squared error of 1.14 mm and an R2 of 0.89 in the case study when fabricating thin-walled structures.

A Visual-Inertial Pressure Fusion-Based Underwater Simultaneous Localization and Mapping System.

Detecting objects, particularly naval mines, on the seafloor is a complex task. In naval mine countermeasures (MCM) operations, sidescan or synthetic aperture sonars have been used to search large areas. However, a single sensor cannot meet the requirements of high-precision autonomous navigation. Based on the ORB-SLAM3-VI framework, we propose ORB-SLAM3-VIP, which integrates a depth sensor, an IMU sensor and an optical sensor. This method integrates the measurements of depth sensors and an IMU sensor into the visual SLAM algorithm through tight coupling, and establishes a multi-sensor fusion SLAM model. Depth constraints are introduced into the process of initialization, scale fine-tuning, tracking and mapping to constrain the position of the sensor in the z-axis and improve the accuracy of pose estimation and map scale estimate. The test on seven sets of underwater multi-sensor sequence data in the AQUALOC dataset shows that, compared with ORB-SLAM3-VI, the ORB-SLAM3-VIP system proposed in this paper reduces the scale error in all sequences by up to 41.2%, and reduces the trajectory error by up to 41.2%. The square root has also been reduced by up to 41.6%.

From Classification to Clinical Insights

Passively collected behavioral health data from ubiquitous sensors could provide mental health professionals valuable insights into patient's daily lives, but such efforts are impeded by disparate metrics, lack of interoperability, and unclear correlations between the measured signals and an individual's mental health. To address these challenges, we pioneer the exploration of large language models (LLMs) to synthesize clinically relevant insights from multi-sensor data. We develop chain-of-thought prompting methods to generate LLM reasoning on how data pertaining to activity, sleep and social interaction relate to conditions such as depression and anxiety. We then prompt the LLM to perform binary classification, achieving accuracies of 61.1%, exceeding the state of the art. We find models like GPT-4 correctly reference numerical data 75% of the time. While we began our investigation by developing methods to use LLMs to output binary classifications for conditions like depression, we find instead that their greatest potential value to clinicians lies not in diagnostic classification, but rather in rigorous analysis of diverse self-tracking data to generate natural language summaries that synthesize multiple data streams and identify potential concerns. Clinicians envisioned using these insights in a variety of ways, principally for fostering collaborative investigation with patients to strengthen the therapeutic alliance and guide treatment. We describe this collaborative engagement, additional envisioned uses, and associated concerns that must be addressed before adoption in real-world contexts.

MST-GAT: A multi-perspective spatial-temporal graph attention network for multi-sensor equipment remaining useful life prediction

Graph neural networks (GNNs) have been applied as an emerging technique for remaining useful life (RUL) prediction due to their ability to effectively exploit the dependencies between multi-sensor data. Despite existing GNN-based RUL prediction methods have achieved promising results, they do not consider the spatial-temporal dependencies of multi-sensor data from different perspectives (global and local), making it difficult to fully utilize the useful information in multi-sensor data. Secondly, traditional point prediction method can only provide the possible future RUL prediction value, but cannot quantify the uncertainty of RUL prediction, making it difficult to provide more information for decision-making. To address the above challenges, we propose a multi-perspective spatial-temporal graph attention network (MST-GAT) for multi-sensor equipment RUL prediction. MST-GAT constructs a multi-perspective graph attention network (MP-GAT) and a temporal convolutional network (TCN) to capture complex spatial-temporal dependencies in multi-sensor data. Specifically, MP-GAT employs a multi-head self-attention mechanism and two multi-head explicit sparse self-attention mechanisms to explicitly model global and local spatial dependencies, and feeds multi-perspective spatial information into TCN to capture temporal dependencies. Furthermore, a joint optimization network is constructed for predicting RUL and its prediction interval to quantify uncertainty. Experiments on three benchmark datasets demonstrate that MST-GAT outperforms state-of-the-art baselines in RUL prediction and can accurately quantify uncertainty.

Combined onshore and offshore wide-scale seismic data acquisition and imaging for carbon capture and storage exploration in Havnsø, Denmark

Strong global actions for climate change include carbon capture and storage (CCS) as a feasible solution to reach carbon neutrality and raise opportunities for detailed subsurface investigations. An acquisition set-up designed for onshore-offshore zones was maximized for wide-scale high-resolution structural imaging and implemented to cover a domal structure of interest for CCS utilization close to the town of Havnsø (Denmark). The challenges of the combined acquisition and processing of land and marine multisensor data along a 42 km seismic profile are analyzed, the suggested solutions are applied, and the limitations are discussed. On the onshore side, a nodal array and a seismic landstreamer system were simultaneously used, whereas along the transition zone, a marine seismic streamer and ocean-bottom seismometers were added to record the seismic response generated by two seismic vibrator sources. The adopted sensing domains (velocity, acceleration, and pressure) were studied, and different processing steps were evaluated to enable their processing and subsequent data set merging. Results suggest, as the best approach, a separate prestack processing of the different data sets and the computation of new geometries and new surface-consistent residual static correction after their merging. The data acquired in the transition zone illuminate, for the first time, the subsurface geology of the region, delineating an expected domal closure. The final seismic section shows high continuity of the reflections with good resolution along the entire profile, identifying the main reservoir structure and the surrounding areas, which are important to ensure reservoir integrity and safe exploitation over longer time scales. Shallow and deep reflections are consistent with the stratigraphic column from a well log near the profile. The presented study shows a comprehensive workflow for processing such a multisensor data set in onshore and transition zone settings.

Multi-sensor Data Analysis for Aerial Image Semantic Segmentation and Vectorization

Abstract. One of the urgent and constantly in demand problems is updating maps. Maps, representing geo-information in vector form, have undoubted advantages in compactness and ”readability” compared to aerial photographs. The issue of maps actuality is critically important for rational urban planning, precision farming, the relevance of the cadastre and other geospatial applications. Various sources of data are used for maps updating, with aerial imagery being the main and rich source of information. Automatic processing of aerial photographs makes it possible to efficiently extract vector information, providing operational monitoring and accounting for changes that have appeared. The presented study addresses the problem of multi sensor information fusion in order to obtain accurate vector information. We use aerial images as a main data source and additionally the data of laser scanning and ground survey to increase performance of automatic image semantic segmentation and vectorization. The proposed framework is demonstrated on the task of forest monitoring.

Joint learning strategy of multi-scale multi-task convolutional neural network for aero-engine prognosis

Remaining useful life (RUL) prediction and health status (HS) assessment are two key tasks in aero-engine prognostics and health management (PHM) system. However, existing deep learning-based prognostic models perform RUL prediction and HS assessment tasks separately, without considering the correlation between these two tasks. Secondly, traditional deep learning can only extract single-scale features, which limits the ability to extract complex degradation features from high-dimensional condition monitoring data. Therefore, this work proposes a multi-scale and multi-task convolutional neural network for joint learning of aero-engine RUL prediction and HS assessment. Firstly, multi-sensor data with multiple cycles are converted into image samples to integrate more condition monitoring information that is beneficial to prognosis. Then, the multi-scale feature fusion block is designed as the shared network for multi-task, utilizing convolutional layers with filters of different sizes to enhance the ability to extract complex degradation features from high-dimensional condition monitoring data. And a multi-layer concatenation block is constructed to integrate multi-scale features at different levels to fully utilize the important information at different levels. On this basis, a multi-task joint learning block is constructed and a joint loss function is developed for joint learning of RUL prediction and HS assessment. Finally, experiments on two engine degradation datasets, CMAPSS and N-CMAPSS, demonstrate that the proposed network has excellent RUL prediction and HS assessment performance, and outperforms other state-of-the-art methods.

Multivariate sensor data analysis for fault detection toward feedback loop-based continuous learning in FFF 3D printer

Multivariate sensor data analysis for fault detection toward feedback loop-based continuous learning in FFF 3D printer

Research on fire detection method of complex space based on multi-sensor data fusion

Efficient and reliable fire detection methods are essential for addressing complex spatial characteristics of older residential communities, which often face poor fire safety conditions, various fire types, and challenges in rescue operations. In response to the problem of lacking comprehensive analysis of multidimensional data in traditional fire detection, which often leads to false alarms and missed alarms, this paper proposes a fire detection method based on multi-sensor data fusion. Various sensors are employed, including smoke sensors, temperature sensors, CO sensors, and flame sensors. These sensors collect multidimensional data, which are fused using improved adaptive filters and fuzzy Bayesian logical reasoning algorithms to establish a fire detection model based on multi-sensor data fusion. To additionally, false alarms and missed detections caused by sensor damage are further solved by exploring hidden spaces. Experimental results demonstrate that the proposed fire detection system based on multi-sensor data fusion effectively analyses fire occurrence and progression in older residential communities, and performs stability, accuracy and real-time property.

SERT: A transformer based model for multivariate temporal sensor data with missing values for environmental monitoring

Environmental monitoring is crucial to our understanding of climate change, biodiversity loss and pollution. The availability of large-scale spatio-temporal data from sources such as sensors and satellites allows us to develop sophisticated models for forecasting and understanding key drivers. However, the data collected from sensors often contain missing values due to faulty equipment or maintenance issues. The missing values rarely occur simultaneously leading to data that are multivariate misaligned sparse time series. We propose two models that are capable of performing multivariate spatio-temporal forecasting while handling missing data naturally without the need for imputation. The first model is a transformer-based model, which we name SERT (Spatio-temporal Encoder Representations from Transformers). The second is a simpler model named SST-ANN (Sparse Spatio-Temporal Artificial Neural Network) which is capable of providing interpretable results. We conduct extensive experiments on two different datasets for multivariate spatio-temporal forecasting and show that our models have competitive or superior performance to those at the state-of-the-art.

A novel multi-sensor hybrid fusion framework

Multi-sensor data fusion has emerged as a powerful approach to enhance the accuracy and robustness of diagnostic systems. However, effectively integrating multiple sensor data remains a challenge. To address this issue, this paper proposes a novel multi-sensor fusion framework. Firstly, a vibration signal weighted fusion rule based on Kullback–Leibler divergence-permutation entropy is introduced, which adaptively determines the weighting coefficients by considering the positional differences of different sensors. Secondly, a lightweight multi-scale convolutional neural network is designed for feature extraction and fusion of multi-sensor data. An ensemble classifier is employed for fault classification, and an improved hard voting strategy is proposed to achieve more reliable decision fusion. Finally, the superiority of the proposed method is validated using modular state detection data from the Kaggle database.

Multi-sensor data fusion-enabled lightweight convolutional double regularization contrast transformer for aerospace bearing small samples fault diagnosis

Aiming at the problems of low information utilization and lack of feature mining capability in multi-sensor fusion networks, this study presents a multi-sensor data fusion-enabled lightweight convolutional double regularization contrast transformer for aerospace bearing small samples fault diagnosis. Firstly, a metric termed integrated cliff entropy is devised to assign weights to vibration signals from diverse sensor channels. It aims to enhance the cyclic impulse characteristics within the fused signals, thereby facilitating more precise fault identification. Secondly, a lightweight Diwaveformer architecture is constructed as the backbone of contrast learning. It enables the global and local features of faulty signals to be comprehensively extracted with less computational effort. Finally, a double contrast loss is constructed to optimize the distribution of intra-class and inter-class features to improve the fault identification ability of the network with small samples. Additionally, a discard regularization method is designed to remove the projection head during the contrast learning process, further advancing the model lightweight. Our method achieved accuracies of 95.54% and 92.56% on two aerospace bearing datasets with extremely sparse training samples, which proved its superior performance.

SENTINEL GUARD: A FATIGUE DETECTION SYSTEM WITH MULTI-CHANNEL ALERT INTEGRATION

The fatigue detection system is a critical application aimed at enhancing safety in various domains by addressing the risks associated with drowsiness related incidents. Leveraging advanced technologies such as machine learning, computer vision, and real-time communication, the system offers a comprehensive solution for monitoring and alerting users to signs of fatigue in real-time. Through the integration of pre-trained models, such as the shape_predictor_68_face_landmarks.dat, the system accurately detects facial landmarks and tracks eye movements, enabling the identification of drowsiness indicators. Upon detection, the system generates alerts via email and WhatsApp, accompanied by audible alarm buzzers to prompt immediate user intervention. The inclusion of live location information in alerts provides additional context, facilitating timely assistance and support. Furthermore, the system's architecture allows for future enhancements, including the integration of multi-sensor data fusion techniques and collaboration with industry stakeholders to promote standardized protocols and widespread adoption. Overall, the fatigue detection system represents a significant step towards mitigating the risks of drowsinessrelated accidents and improving safety across various applications Key Words: Fatigue detection, Machine learning, CNN, Real time alerts, Facial Expression analysis, Open CV