Heterogeneous Monitoring Data Research Articles

HeMiRCA: Fine-Grained Root Cause Analysis for Microservices with Heterogeneous Data Sources

Microservices architecture improves software scalability, resilience, and agility but also poses significant challenges to system reliability due to their complexity and dynamic nature. Identifying and resolving anomalies promptly is crucial because they can quickly propagate to other microservices and cause severe damage to the system. Existing root-cause metric localization approaches rely on metrics or metrics-anomalies correlations but overlook other monitoring data sources ( e.g. , traces). We are the first to identify and leverage the anomaly-aware monotonic correlation between heterogeneous monitoring data, motivated by which we propose a novel framework, HeMiRCA , for hierarchical root cause analysis using Spearman correlation. HeMiRCA is based on the key observation that the microservice responsible for a particular type of fault exhibits a monotonic correlation between the trends of its associated metrics and the trace-based anomaly score of the system. HeMiRCA first calculates time-series anomaly scores using traces and then exploits the correlations between multivariate metrics and the scores to rank the suspicious metrics and microservices. HeMiRCA has been evaluated on two datasets collected from widely used microservice systems. The results show that HeMiRCA outperforms the state-of-the-art approaches by a large margin in identifying root causes at both service level and metric level, achieving a top-1 hit ratio of 82.7% and 74% on average, respectively.

Dynamic comprehensive quality assessment of post-harvest grape in different transportation chains using SAHP–CatBoost machine learning

Abstract Objectives Food quality assessment is critical for indicating the shelf-life and ensuring food safety or value. Due to high environmental sensitivity, the post-harvest quality of fresh fruit will undergo complex changes in the supply chain, with various dynamic quality-related features. It is difficult to efficiently and accurately extract comprehensive quality feature of post-harvest fruits from high-dimensional monitoring data with heterogeneous characteristics (numerical and categorical). Therefore, we proposed a dynamic comprehensive quality assessment method based on self-adaptive analytic hierarchy process (SAHP) integrated with the CatBoost model. Materials and Methods By adaptive weight optimization, the SAHP was utilized to analyze the multi-source quality information and obtain the quantized fusion value, as an output sample of CatBoost machine learning. Then, using heterogeneous monitoring data as input, the CatBoost model was directly trained through unbiased boosting with categorical features for dynamic assessment of overall quality status. Results Three quality index monitoring data sets for ‘Jufeng’ grape in different transportation chains (normal temperature, cold insulation, and cold chain) were individually constructed as the research samples. Furthermore, compared to other machine learning methods, the SAHP–CatBoost had more accurate results in comprehensive quality feature extraction. In actual transportation chains, the mean absolute error, mean absolute percentage error, and root mean squared error of dynamic comprehensive assessment were limited to 0.0044, 1.012%, and 0.0078, respectively. Conclusions The proposed method is efficient in handling heterogeneous monitoring data and extracting comprehensive quality information of post-harvest grape as a robust shelf-life indicator. It can reasonably guide post-harvest quality management to reduce food loss and improve economic benefits.

High-spatiotemporal-resolution PM2.5 forecasting by hybrid deep learning models with ensembled massive heterogeneous monitoring data

High-resolution real-time air quality forecasting can alert decision-makers and residents about forthcoming air pollution events and refine air quality management. The Environmental Protection Administration in Taiwan has deployed numerous low-cost air quality microsensors near industrial zones lately to facilitate local air quality monitoring. Nevertheless, the frequent occurrence of missing sensor data due to problems of mobile transmission, frontend/backend device malfunction, or other unforeseen issues would raise difficulty in making quick responses to air pollution incidents. This study proposed a hybrid deep learning model (AE-CNN-BP) collaborating an Autoencoder (AE), a Convolutional Neural Network (CNN), and a Back Propagation Neural Network (BPNN) to effectively extract crucial features from big data for making successive high-spatiotemporal-resolution forecasts of PM2.5 concentrations 4 h ahead. The proposed model was trained and tested in three industrial zones densely installed with microsensors in Kaohsiung City of Taiwan. A high pollution incident was selected to evaluate model performance. The results show that the proposed model could reliably produce nice high-spatiotemporal-resolution forecasts for 12 air quality monitoring stations and 485 microsensors, with Coefficient of Determination (R2) values and Root Mean Squared Error (RMSE) of 0.82 (0.76) and 11.05 (12.75) μg/m3 in the training (testing) stage, respectively. For the selected incident, the Mean Absolute Percentage Error (MAPE) values of the proposed model were 22.3% and 27.1% at T+1 and T+4, respectively. This study demonstrates that the proposed deep learning model based on ensemble datasets of sparsely distributed monitoring stations and densely deployed microsensors can offer reliable high-spatiotemporal-resolution air quality forecasts, benefiting environmental studies and informed policymaking by accounting for local-scale variations in PM2.5 concentrations.

A Nonparametric Bayesian Approach for Bridge Reliability Assessment Using Structural Health Monitoring Data

Integrating structural health monitoring (SHM) data into reliability assessment has increasingly been practiced in the condition evaluation of in-service bridges over the past decade. The selection of probability distribution models for load- and resistance-related random variables is a prerequisite for monitoring-based reliability assessment. However, the underlying probabilistic assumptions of the used models could be restrictive and unverifiable especially when dealing with real-world heterogeneous monitoring data, weakening the confidence on the estimated reliability index. This study aims to develop a nonparametric Bayesian model with the Dirichlet process prior for bridge reliability assessment, where the model order constraint can be released such that the complexity of the model adapts to the observed data. Reliability analysis via the nonparametric Bayesian model allows the aleatory uncertainty and the epistemic uncertainty arising from monitoring data to be concurrently accounted for in the formulated reliability index. A numerical example is presented to verify the effectiveness of the nonparametric Bayesian model for dealing with multimodal data. The feasibility of the proposed approach for reliability assessment is then demonstrated with one-year strain monitoring data acquired from a large-scale bridge instrumented with the SHM system.

Wind load identification of lattice towers using multi-source heterogeneous monitoring data

Wind load is one of the governing design loads for lattice tower structures. However, such information is difficult to be measured directly, and the traditional indirect identification methods highly depend on the accuracy of finite element model and are limited to fluctuating wind loads. To solve this problem, a novel wind load identification method from limited multi-source heterogeneous monitoring data is proposed. First, the complex model of a lattice tower is simplified into a reduced-order model based on the similarity principle of dynamic characteristics. And then, limited strain and acceleration data are fused to reconstruct wind-induced responses and modal parameters based on Kalman filter algorithm and mode-superposition method. Finally, the total wind loads, including both mean and fluctuating components, are obtained from identified modal parameters and modal responses. Numerical simulation of a lattice tower verified that the proposed method can reconstruct wind load and base reaction forces with high accuracy, and the anti-noise analysis shows that the proposed method has strong noise robustness. Moreover, the field test of a transmission tower during Typhoon Mangkhut was further carried out to validate the proposed method.

BIM and ontology-based knowledge management for dam safety monitoring

Dam Safety Monitoring Systems (DSMSs) are crucial in evaluating the operational state of dams. However, heterogeneous monitoring data distributed from multiple sources during the dam operation lacks a unified data integration method and prohibits knowledge extraction and intelligent analysis, which currently poses a labor-intensive task. To address this issue, a solution relying on Building Information Modeling (BIM) and domain ontologies is proposed. Specifically, a DSMS domain ontology (OntoDSMS) is developed by comprehensively collecting domain knowledge and extracting context information from the dam information model. Furthermore, a rule-based reasoner and SPARQL queries are implemented. The proposed approach facilitates the effective integration of dam safety monitoring information while reducing retrieval time effectively compared with traditional databases. A case is illustrated to demonstrate the feasibility and practicality of the proposed approach.

A Modified Generative Adversarial Network for Fault Diagnosis in High-Speed Train Components with Imbalanced and Heterogeneous Monitoring Data

Data-driven methods are widely considered for fault diagnosis in complex systems. However, in practice the between-class imbalance due to limited faulty samples may deteriorate their classification performance. To address this issue, synthetic minority methods for enhancing data have been proved to be effective in many applications. Generative Adversarial Networks (GANs), capable of automatic features extraction, can also be adopted for augmenting the faulty samples. However, the monitoring data of a complex system may include not only continuous signals, but also discrete/categorical signals. Since the current GAN methods still have some challenges in handling such heterogeneous monitoring data, a Mixed Dual Discriminator GAN (noted as M-D2GAN) is proposed in this work. In order to render the expanded fault samples more aligned with the real situation, and improve the accuracy and robustness of the fault diagnosis model, different types of variables are generated in different ways, including: floating-point, integer, categorical, and hierarchical. For effectively considering the class imbalance problem, proper modifications are made to the GAN model, where a normal class discriminator is added. A practical case study concerning the braking system of a high-speed train is carried out to verify the effectiveness of the proposed framework. Compared to the classic GAN, the proposed framework achieves better results with respect to F-measure and G-mean metrics.

Application of the tunnel fire scenario dynamic reconstruction based on Intelligent Disaster Prevention Platform in Shanghai

Effective evacuation and fire-fighting response require an accurate, prompt and dynamic reconstruction and visualization of tunnel fire scenarios. This paper aims to introduce the application of the Intelligent Disaster Prevention Platform integrating tunnel fire knowledge with the machine learning approach in Shanghai tunnels. The platform can provide key fire parameters consist of fire location, real-time heat release rate, temperature distribution and smoke movement, and reconstruct the tunnel fire scenario. The platform uses machine learning algorithms and tunnel fire knowledge base to conduct in-depth mining and analysis of multi-source heterogeneous monitoring data of tunnel disaster prevention equipment, including four modules: environmental parameters perception, intelligent fire warning, fire scenarios reconstruction and personnel dynamic evacuation, so as to provide scientific decision support for intelligent fire firefighting. Meanwhile, the platform has been demonstrated and applied in many urban underground roads in Shanghai, such as Hongmei South Road Tunnel, Yan’an East Road Tunnel and Dalian Road Tunnel, which significantly reduces the false alarm rate of tunnel fires and provides early warning in fire accidents. The relevant experience can provide guidance for underground infrastructure.

Construction completion forecasting on the basis of statistical modeling and heterogeneous monitoring data

Modern construction represents a complex production process whose effective regulation is based on information about the period of construction works, obtained by way of monitoring. Any delay in the execution of certain works frustrates scheduled project commissioning, which results in increased project management costs, forfeits and lost benefits. Existing monitoring systems need to be improved through the use of probabilistic scheduling geared towards the forecasting of completion dates of certain works and the construction process as a whole. Construction monitoring may be improved by means of taking management quality into account by means of distributing random work durations in the process of statistical modeling of functions. The introduction of six random duration distribution functions, reflecting management quality, is proposed for the improvement of statistical modeling, whereas the use of scheduled durations of works is expedient for identifying optimistic characteristics. The data, extracted from monitoring reports, needs to be used as pessimistic distributions. Monitoring reports contain heterogeneous data, and they may even have no information on particular types of works. Therefore, pessimistic durations based on missing data should be calculated using the time-space analogy method.

Multi-sensor heterogeneous data fusion method for rotor system diagnosis based on multi-mode residual network and discriminant correlation analysis

Deep learning algorithms have become a research hotpot in the field of fault diagnosis due to their powerful feature adaptive extraction capabilities. However, the monitoring data used as the input of deep learning typically includes only single-sensor data (e.g. vibration signals), not the multi-sensor unstructured data (e.g. infrared images) that can provide fault information from different angles, resulting in low diagnostic accuracy and weak generalization ability of the built model. To apply multi-sensor heterogeneous monitoring data fully, a novel fusion diagnosis method based on multi-mode residual network (M-ResNet) and discriminant correlation analysis (DCA) is proposed in this paper. Firstly, M-ResNet composed of one-dimensional ResNet (1D-ResNet) and two-dimensional ResNet (2D-ResNet) is constructed to extract richer and more comprehensive fault features. 1D-ResNet and 2D-ResNet are used to process structured vibration signals and unstructured infrared images respectively, so as to obtain more comprehensive fault information. Secondly, DCA is used to fuse the vibration signal features and infrared image features extracted by M-ResNet to maximize the intra-class correlation and eliminate inter-class correlation of features, thereby improving the accuracy of fault diagnosis. Finally, the fusion features are input into the Softmax classifier to achieve fault classification. The effectiveness of the proposed method is verified by experiments on the rotor system, where it achieves a remarkable classification rate of 99.79%. Compared with the similar methods, the proposed method exhibits outstanding performance, indicating the feasibility of using multi-sensor heterogeneous data for rotor system fault diagnosis.

Geological disaster monitoring and early warning system based on big data analysis

The existing geological disaster monitoring and early warning system has the problems of large monitoring errors and low accuracy. This paper designs a geological disaster monitoring and early warning system based on big data analysis. The geological disaster monitoring network module monitors the geological deformation and disaster inducing factors in real time through sensors and transmits them to the large-capacity data remote wireless transmission network module in the multi-dimensional heterogeneous monitoring data integration module through the Beidou satellite communication technology. This module uses a multi-sensor data fusion algorithm based on the D-S evidence theory and fuzzy mathematics to realize the integration of geological multi-dimensional heterogeneous monitoring data. Then, it uses cloud computing-based geological abnormal data mining method to mine abnormal geological information and transmit it to the early warning information release module. Geological disaster warning information will be released in the form of text messages and emails. Through verification, the system has high accuracy in monitoring and early warning of different types of geological disasters and provides a basis for timely and scientific deployment of geological disasters.

Hierarchical Edge Computing: A Novel Multi-Source Multi-Dimensional Data Anomaly Detection Scheme for Industrial Internet of Things

Every year, many people around the world die because of mining accidents. Industrial Internet of Things (IIoT) can be employed to sense public safety hazards and provide early warning of accidents, thereby ensuring safe operations at underground mining, personnel positioning, and specific items supervision and emergency response. Real-time data anomaly detection can predict the probability of occurrence of the abnormal event. However, massive heterogeneous monitoring data, poor wireless environment and data spatio-temporal association have posed a serious challenge to data anomaly detection for underground mining. Existing methods are mostly concerned about single data or processing at cloud platform, with little regard for the time and space association. Focus on the accuracy and timeliness of data anomaly detection, a novel multi-source multi-dimensional data anomaly detection scheme based on hierarchical edge computing model is presented in this paper. Firstly, a hierarchical edge computing model is proposed to realize load balance and low-latency data processing at the sensor end and base-station end. Then a single-source data anomaly detection algorithm is designed based on fuzzy theory, which can comprehensively analyze the anomaly detection results of multiple consecutive moments. Finally, a multi-source data anomaly detection algorithm executed at the base-station end is designed to consider the sensing data associated attributes of time and space. Experimental results reveal that the proposed scheme has higher detection accuracy and lower processing delay compared with traditional solutions.

A novel fusion diagnosis method for rotor system fault based on deep learning and multi-sourced heterogeneous monitoring data

Deep learning-based fault diagnosis has been acclaimed for its superiority in adaptively mining salient features. The monitoring data used as the input of deep learning typically includes only structured data (e.g. vibration signals, voltage signals, and acoustic emission signals), not unstructured data (e.g. infrared images), which provides another perspective on the mechanical health condition. To apply multi-sourced heterogeneous monitoring data fully, this paper presents a novel fusion diagnosis method integrating structured and unstructured data for rotor system faults. A novel multi-mode convolutional neural network (M-CNN) is first proposed to automatically learn fault-sensitive features from raw multisensory data composed of vibration signals and infrared images. M-CNN equipped with adjustable filter banks can identify data types and adaptively adopt the appropriate convolution mode. Then, t-distributed stochastic neighbor embedding (t-SNE) is introduced to fuse the deep features to further improve the quality of the learned features. Finally, the fused features are employed to conduct fault classification tasks. The effectiveness of the scheme is verified by fault diagnosis experiments in a rotor system, where it achieves a remarkable classification rate of 98.97%. Compared with similar methods, the proposed method exhibits outstanding performance, indicating the feasibility of using multi-sourced heterogeneous data for rotor system fault-diagnosis.

An Automatic Online Disaster Monitoring Network: Network Architecture and a Case Study Monitoring Slope Stability

Geosensor networks(GSN) is an important development direction of the disaster monitoring in the future. An online automatic unattended disaster monitoring system can prevent and reduce the geology disaster to protect the safety of life and property. At present, most GSN are independent and usually service for respective community. The observations data of GSN are bigger and complex , and GSN is mostly heterogeneous wireless sensor networks. So this paper proposes a novel GSN disaster monitoring overall architecture, This architecture can seamlessly integrate sensors for long- term, remote, and near-real-time monitoring. In the architecture, there are four layers are used to collect, manage , transport and processing observation data. Among them, the data server layer applies the OGC SWE standards to integrate and share heterogeneous monitoring data. sensor metadata and observation data are packaged into a virtual sensor that are is transported from data center to application layer through Sensor Observation Service (SOS). To demonstrates the applicability of our proposed method, we use a case named PS-MDMs which are developed and deployed to support mine disaster monitoring and modeling research.

Robust inversion method for jointly estimating parameters and variance components from heterogeneous monitoring data

In this paper, a novel inversion method is proposed for jointly robust estimation of parameters and variance components from disjunctive groups of observations affected by outliers. This method, named robust non-negative variance component estimation (RVCE), is a coupling of variance component estimation (VCE) technique with a robust estimation method, developed to cope with outliers and to avoid negative variance, leading thus, to an estimation reliable enough. The principle of RVCE method is based on the refinement of the stochastic model via an equivalent weight matrix established from the original measurement weight matrix and an adapted full weight matrix with hard rejection to outliers. This last one is derived from the robust standardized residuals, using a highly robust estimator, as an initial solution of the inverse problem, and a cut-off value adapted to sort out the good observations from the bad ones. Furthermore, because the original weight matrix is partly known, the integration of the VCE technique plays a key role to reach an optimal solution and to provide valuable information on the precision of the estimates. The performance of the proposed method is demonstrated by considering a rockfill dam as an example, where the material parameters and variance components are jointly estimated from geotechnical and geodetic measurements. The results of comparison study between RVCE method with other methods such as the classical NN-VCE, RIMCO, least squares and the combined Huber’s M-estimator with VCE (HVCE) for various configuration options have highlighted the pertinence of the proposed method.

The PeRvasive Environment Sensing and Sharing Solution

To stimulate better user behavior and improve environmental and economic sustainability, it is of paramount importance to make citizens effectively aware of the quality of the environment in which they live every day. In particular, we claim that users could significantly benefit from cost-effective efficient Internet-of-Things (IoT) solutions that provide them with up-to-date live information about air pollution in the areas where they live, suitably adapted to different situations and with different levels of dynamically selected granularities (e.g., at home/district/city levels). Our PeRvasive Environment Sensing and Sharing (PRESS) project has the ambition of increasing users’ awareness of the natural environment they live in, as a first step towards improved sustainability; the primary target is the efficient provisioning of real-time user-centric information about environmental conditions in the surroundings, and in particular about air pollution. To this purpose, we have designed, implemented, and thoroughly evaluated the PRESS framework, which is capable of achieving good flexibility and scalability while integrating heterogeneous monitoring data, ranging from sensed air pollution to user-provided quality perceptions. Among the elements of technical originality, PRESS exploits extended Kura IoT gateways with novel congestion detection and recovery mechanisms that allow us to optimize bandwidth allocation between in-the-field PRESS components and the cloud. The reported performance results show the feasibility of the proposed solution, by pointing out not only the scalability and efficiency of the adopted message-based solution that uses Message Queue Telemetry Transport (MQTT) and WebSockets, but also the capability of PRESS to quickly identify and manage traffic congestions, thus, ensuring good quality levels to final users.

Advanced Visualization and Accessibility to Heterogeneous Monitoring Data

AbstractStructural health monitoring (SHM) is a promising tool for better management of infrastructure. However, failure to manage large amounts of SHM data properly is a big barrier for its widespread application. Data management is especially challenging when heterogeneous data are involved and combined with camera images. Various sensors based on different technologies can measure many parameters such as strain, tilt, weather, etc., whereas live cameras can visualize traffic response. In addition, all the data streams can be registered both statically and dynamically. Data management is even more complex if multiple users access the data and have diverse backgrounds and interests (e.g., the owner/manager of the structure, operator, responsible engineer, and academic). In this article, general principles for SHM data management are researched, established, and proposed by the authors, and an original solution for data management based on these principles is presented. For validation purposes, the proposed principles were implemented in novel data management software and applied to a signature bridge. Feedback from interested groups including the managers, operators, engineers of record, and academics is used for validation. Thus, this article details the researched and proposed accessibility and visualization principles, and conceptual development of novel solution for data management, which combines computer science, internet omnipresence, and structural engineering.

Semantic data integration and monitoring in the railway domain

Information integration is a key for further growth of efficiency in management decisions for the railway domain. In the context of the EU project InteGRail (funded in the 6th framework programme) an integration approach leveraged by ontologies known from the semantic web and logic-based reasoning mechanisms has been successfully demonstrated. To this effect existing heterogeneous monitoring data acquired across the European railways (in the context of rolling stock, infrastructure, operations and traffic management) is logically integrated according to a formal information model. Based on distributed reasoning mechanism decentralised data and inferred knowledge does not have to be aggregated in a central repository but can be transparently accessed by applications independently from where it is acquired. We explain how the proposed techniques facilitate integration, analysis and interpretation of distributed observation data in the railway domain. In addition, the implementation of the presented approach...

Parameter identification in numerical modeling of tunneling using the Differential Evolution Genetic Algorithm (DEGA)

The paper investigates the potential use of the Differential Evolution Genetic Algorithm (DEGA) in the back-analysis of tunnel response in order to obtain improved estimates of model parameters by matching model prediction with monitored response. DEGA is implemented in the finite difference code FLAC using FLAC’s built-in language FISH. The performance of DEGA in back-analyzing tunnel response is demonstrated using two idealized tunnel geometries and synthetic monitoring data, and a real case involving a section of the Heshang Road Tunnel. The use of DEGA in back-analysis of tunnel response is analyzed in terms of the stability and efficiency of the procedure under highly non-linear problems, the sensitivity of the solution to the initial trial assumption and the sensitivity of the solution to the monitoring data. It is shown that DEGA exhibits excellent convergence and repeatability characteristics, and that the convergence is independent of the initial values of the parameters. Another strength of DEGA comes from its ability to handle highly nonlinear objective functions with potentially multiple optima resulting from the use of incomplete and relative monitoring data, and heterogeneous monitoring data from different sensors. The paper recommends DEGA as a very viable technique in the back analysis and computational modeling of tunnel response.

Integrating heterogeneous network monitoring data

In this paper, we investigate the integration of heterogeneous network monitoring data. Specifically, we will synchronize and integrate flow-level records, exemplified by Cisco NetFlow, and packet-level traces, exemplified by NLANR PMA. The integration can facilitate cross-validation and complementary utility. However, finding the correspondences of timestamps/flows/packets between the PMA and Netflow is non-trivial, because they have different levels of granularity, different sampling strategy, different time sources, and different IP address masking. To integrate heterogeneous monitoring data, we first synchronize their timestamps, and then match their masked IP addresses. Our key observation is that although the IP addresses are masked, some other header fields can be exploited to match different types of monitoring data. In order to reduce the search space and the processing overhead, we have adopted a top-down approach to limit the search scope, and iterative algorithms to reduce the matching errors step by step.