Failure Detection Capabilities Research Articles

A model for energy predictions and diagnostics of large-scale photovoltaic systems based on electric data and thermal imaging of the PV fields

The aim of this investigation is the development of robust models for the performance prediction and automatic monitoring of large photovoltaic systems, based on historical and real-time electric and thermal data. This issue is increasingly important due to the worldwide diffusion of large photovoltaic systems and their need to identify and predict failures and malfunctions, in order to promptly assess the convenience of maintenance actions. The present model describes the response to irradiance and temperature conditions of both modules and inverters and also it is able to predict shading conditions able to affect the energy yield. The model has been validated against real electric measurements in 6 large PV plants located in southern Italy and it demonstrated to be able to predict the real time power production within a 4.1 % error. Even more importantly, the model and its comparison with subhourly measurements over several years has demonstrated its effectiveness in detecting downtime conditions caused by inverter or string problems. Simulations and measurements revealed that missed energy production due to electrical grid coupling downtime can exceed 50 % on certain days and that the shading conditions (up to 5 % of the daily energy production) can be easily detected and separated from component problems, thus avoiding false alarms. Finally, the analysis of aerial infrared images allowed to further test the model in failure detection capability, assess the relationship between thermal anomalies and underperformance conditions and in predicting the yearly deterioration rate at the PV plants.

Neuron Semantic-Guided Test Generation for Deep Neural Networks Fuzzing

In recent years, significant progress has been made in testing methods for deep neural networks (DNNs) to ensure their correctness and robustness. Coverage-guided criteria, such as neuron-wise, layer-wise, and path-/trace-wise, have been proposed for DNN fuzzing. However, existing coverage-based criteria encounter performance bottlenecks for several reasons: ❶ Testing Adequacy : Partial neural coverage criteria have been observed to achieve full coverage using only a small number of test inputs. In this case, increasing the number of test inputs does not consistently improve the quality of models. ❷ Interpretability : The current coverage criteria lack interpretability. Consequently, testers are unable to identify and understand which incorrect attributes or patterns of the model are triggered by the test inputs. This lack of interpretability hampers the subsequent debugging and fixing process. Therefore, there is an urgent need for a novel fuzzing criterion that offers improved testing adequacy, better interpretability, and more effective failure detection capabilities for DNNs. To alleviate these limitations, we propose NSGen, an approach for DNN fuzzing that utilizes neuron semantics as guidance during test generation. NSGen identifies critical neurons, translates their high-level semantic features into natural language descriptions, and then assembles them into human-readable DNN decision paths (representing the internal decision of the DNN). With these decision paths, we can generate more fault-revealing test inputs by quantifying the similarity between original test inputs and mutated test inputs for fuzzing. We evaluate NSGen on popular DNN models (VGG16_BN, ResNet50, and MobileNet_v2) using CIFAR10, CIFAR100, Oxford 102 Flower, and ImageNet datasets. Compared to 12 existing coverage-guided fuzzing criteria, NSGen outperforms all baselines, increasing the number of triggered faults by 21.4% to 61.2% compared to the state-of-the-art coverage-guided fuzzing criterion. This demonstrates NSGen’s effectiveness in generating fault-revealing test inputs through guided input mutation, highlighting its potential to enhance DNN testing and interpretability.

A passive ladder-shaped sensor architecture with failure detection based on fiber Bragg grating

This study proposes a fiber Bragg grating (FBG)-based passive ladder-shaped sensor architecture with failure detection capabilities. The sensor architecture primarily includes optocoupler (OCP) lines and FBG lines and can detect multiple failure points. In order to verify the failure detection capability of the system, we set six failure modes and conducted simulation experiments. In the first four cases, the OCP lines are failure-free, while in the last two cases, the OCP line is faulty. The experimental results show that in the first four cases, the sensor network can detect multiple failure points; in the last two cases, although the break-point position cannot be determined, it can help to locate the failure area. In addition, strain sensing is also analyzed. The results show that the proposed sensor system can not only perform strain sensing, but also detect the failure point.

A lightweight adaptive random testing method for deep learning systems

AbstractIn recent years, deep learning (DL) systems are increasingly used in the safety‐critical fields such as autonomous driving, medical diagnosis, and financial service. Although these systems have demonstrated an outstanding performance in enhancing the accuracy of decision‐making, they pose significant challenges to the trustworthiness due to their limited interpretability and inherent uncertainty. Adaptive random testing (ART) has been proved as an effective approach for ensuring the reliability of DL systems. However, existing ART methods for DL systems incur a heavy overhead in test case selection due to the computation of distances. To address this issue, we propose a lightweight adaptive random testing (Lw‐ARTDL) method for DL systems. In our improved algorithm, we employ the K‐Means technique to divide the entire test suite into several subsets. Then, for a candidate test case, we only calculate distances between it and the test cases within the category to which it belongs. This partition strategy ensures that the selected test cases are more representative while significantly reducing the computational cost. To validate the proposed algorithm, the comparison experiments between Lw‐ARTDL and the original ARTDL algorithm are conducted on two typical DL systems. The experimental results show that Lw‐ARTDL significantly reduces the overhead of failure detection, and exhibits stronger failure detection capability compared to ARTDL in most similarity metrics.

Application of condition-based maintenance for electrical generators based on statistical control charts

Condition-based maintenance involves activities that are conducted based on the equipment's performance. Continuous monitoring of equipment will ensure that it will be maintained according to a relevant activity plan. This paper proposes a maintenance framework to analyze the application of statistical control charts for condition-based maintenance of electrical generators. The proposed framework consists of four components that collaboratively determine a performance threshold for a given piece of electrical equipment. Based on the slow progression and dynamics of mechanical failures, Long Short-term Memory (LSTM) and Useful Remaining Life (URL) models were used to assist in the maintenance decision-making process. The analysis is based on detecting the dynamics of the process parameters, including vibration, noise, and temperature, based on relevant control charts. With the help of experimental methodology, failures in the performance modes and defined modes are measured. Then empirical analysis reveals how control charts respond to failure detection. The results show that X-bar consistently demonstrates failure detection capability, while R charts sometimes fail when data deviates from normality.Moreover, heat monitoring surpassed vibration and noise in failure detection, where temperature control charts successfully identified failure. The overall results support the significant role of statistical charts in decision-making regarding condition-based maintenance for electrical equipment like generators.•Application of statistical control charts for condition-based maintenance of electrical generators.•Detecting dynamics of the process parameters, including vibration, noise, and temperature, based on relevant control charts.•Long Short-term Memory (LSTM) and Useful Remaining Life (URL) models were used to assist in the maintenance decision-making process

ARTE: Automated Generation of Realistic Test Inputs for Web APIs

Automated test case generation for web APIs is a thriving research topic, where test cases are frequently derived from the API specification. However, this process is only partially automated since testers are usually obliged to manually set meaningful valid test inputs for each input parameter. In this article, we present ARTE, an approach for the automated extraction of realistic test data for web APIs from knowledge bases like DBpedia. Specifically, ARTE leverages the specification of the API parameters to automatically search for realistic test inputs using natural language processing, search-based, and knowledge extraction techniques. ARTE has been integrated into RESTest, an open-source testing framework for RESTful APIs, fully automating the test case generation process. Evaluation results on 140 operations from 48 real-world web APIs show that ARTE can efficiently generate realistic test inputs for 64.9% of the target parameters, outperforming the state-of-the-art approach SAIGEN (31.8%). More importantly, ARTE supported the generation of over twice as many valid API calls (57.3%) as random generation (20%) and SAIGEN (26%), leading to a higher failure detection capability and uncovering several real-world bugs. These results show the potential of ARTE for enhancing existing web API testing tools, achieving an unprecedented level of automation.

A Novel Test Case Generation Approach for Adaptive Random Testing of Object-Oriented Software Using K-Means Clustering Technique

Random testing (RT) is considered important owing to the popularity of fuzzing techniques. Yet, its effectiveness has been questioned because it disregards the property that failures are usually conglomerated. Adaptive random testing (ART) has been developed to spread the test cases more evenly. A set of candidate test cases is first generated randomly. Then, the candidate farthest from the executed test cases is selected as the next test case. However, this process is time-consuming, especially for object-oriented programs. In order to improve the efficiency, a forgetting strategy may be added, taking into account only part of the executed test cases. Of course, the failure detection capability is compromised. In this paper, we propose a new approach that improves the efficiency of object-oriented program testing through K-means clustering. Frequency transform and improved wavelet transform are adopted to better determine the dissimilarity of test cases. Two measures, namely, the wavelet transform (WT) metric and the trisection frequency conversion (TFC) metric, are proposed. Based on the metrics, we develop two algorithms called WClustering-ART and TFClustering-ART. Experimental studies and statistical analysis have been conducted to evaluate the F-measure and the Fm-time of our approach. The results show that both algorithms demonstrate better effectiveness and efficiency than RT and object-oriented ART algorithms with forgetting.

Predicting locally manageable resource failures of high availability clusters

AbstractCritical services from domains as diverse as finance, manufacturing and healthcare are often delivered by complex enterprise applications (EAs). High‐availability clusters (HACs) are software‐managed IT infrastructures that enable these EAs to operate with minimum downtime. This article presents a novel Bayesian decision network model to improve the failure detection capabilities of the HACs components using a comprehensive set of characteristics for the analyzed component. The model then combines these characteristics to predict whether the failure of this component can be managed locally at the failed component level without propagating the failure to upper‐level components and causing a complete system failure. By improving the detection capabilities and predicting locally manageable failures, the model improves the decision‐making process of HACs, and has the potential to reduce the downtime and improve availability for the applications protected by HACs. The model uses the capabilities of the Bayesian decision networks, which combines Bayesian networks with the utility theory, to assign weights to different characteristics and consolidate the related variables to output the result. The model evaluation in a realistic testbed environment with three servers, an established HAC and a well‐known EA shows that the model can improve the area under the receiver operating characteristic curve for prediction of locally manageable failures by up to 9.05% compared to the baseline HAC results.

A Simulation-Based Approach to Aid Development of Software-Based Hardware Failure Detection and Mitigation Algorithms of a Mobile Robot System.

Mechatronic systems, like mobile robots, are fairly complex. They are composed of electromechanical actuation components and sensing elements supervised by microcontrollers running complex embedded software. This paper proposes a novel approach to aid mobile robotics developers in adopting a rigorous development process to design and verify the robot’s detection and mitigation capabilities against random hardware failures affecting its sensors or actuators. Unfortunately, assessing the interactions between the various safety/mission-critical subsystem is quite complex. The failure mode effect analysis (FMEA) alongside an analysis of the failure detection capabilities (FMEDA) are the state-of-the-art methodologies for performing such an analysis. Various guidelines are available, and the authors decided to follow the one released by AIAG&VDA in June 2019. Since the robot’s behavior is based on embedded software, the FMEA has been integrated with the hardware/software interaction analysis described in the ECSS-Q-ST-30-02C manual. The core of this proposal is to show how a simulation-based approach, where the mechanical and electrical/electronic components are simulated alongside the embedded software, can effectively support FMEA. As a benchmark application, we considered the mobility system of a proof-of-concept assistance rover for Mars exploration designed by the D.I.A.N.A. student team at Politecnico di Torino. Thanks to the adopted approach, we described how to develop the detection and mitigation strategies and how to determine their effectiveness, with a particular focus on those affecting the sensors.

A nearest-neighbor divide-and-conquer approach for adaptive random testing

• We propose a new ART approach to enhance testing efficacy and efficiency of FSCS- ART. • We report on simulations and empirical studies to investigate the proposed method. • Compared with FSCS-ART, our proposed method is more cost-effective. • Compared with other ART algorithms, our method has similar or even better results. • Our proposed method can alleviate the boundary effect problem of FSCS-ART. Adaptive Random Testing (ART) aims at enhancing the failure detection capability of Random Testing (RT) by evenly distributing test cases over the input domain. Many ART algorithms have been proposed to achieve an even spread of test cases, according to different notations. A well-known ART algorithm, namely Fixed-Size-Candidate-Set ART (FSCS-ART), chooses an element as the next test case such that it is farthest away from previously executed test cases. Previous studies have demonstrated that FSCS-ART could achieve good improvements in test effectiveness over RT, but suffers from some drawbacks such as high computational overhead and boundary effect problem. In this paper, we propose an alternative approach to enhance FSCS-ART, namely Nearest-Neighbor Divide-and-Conquer based ART (NNDC-ART), which combines the concepts of nearest-neighbor and divide-and-conquer . It attempts to reduce the computational overhead of test case generation, and also to alleviate the boundary effect problem; while maintaining the advantages of FSCS-ART. The simulation results show that compared with FSCS-ART, NNDC-ART requires much less computational overhead, and also achieves much better test case distribution; while maintaining better or comparable failure-detection effectiveness. Our empirical studies further show that the proposed approach is much more cost-effective than FSCS-ART. In addition, we further compare NNDC-ART against two enhancements of FSCS-ART based on the partitioning strategy, i.e., ART with Divide-and-Conquer (ART-DC) and ART by Bisection and Localization (ART-B-Loc), and find that our proposed approach can achieve similar or even better testing performances in some scenarios.

Candidate test set reduction for adaptive random testing: An overheads reduction technique

• We propose a cost-effective ART approach, FSCS-CTSR, to improve the efficiency of FSCS. • We examine the effectiveness and efficiency of FSCS-CTSR through simulations and experimental studies. • Compared with FSCS, FSCS-CTSR achieves similar effectiveness to FSCS, but significantly reduces the computational costs. Adaptive Random Testing (ART) is a family of testing techniques that were proposed as an enhancement of random testing (RT). ART achieves better failure-detection capability than RT by more evenly distributing test cases throughout the input domain. However, this process of selecting more diverse test cases incurs a heavy computational cost. In this paper, we propose a new ART method that improves on the efficiency of Fixed-Size-Candidate-Set ART (FSCS) by applying a test set reduction strategy. The proposed method, FSCS by Candidate Test Set Reduction (FSCS-CTSR), reduces the number of randomly generated candidate test cases, but supplements them with earlier, unused candidates that have lower similarity to the executed test cases. Simulations and experimental studies were conducted to examine the effectiveness and efficiency of the method, with the experimental results showing a comparable failure-detection effectiveness to FSCS, but with lower computational costs.

Item-based Reliability-centred Life-Cycle Costing using Monte Carlo Simulation

This paper presents a time-sequential probabilistic simulation model for the detailed design of maintenance strategies for turbine critical items. The term item shall refer to any part, component, device, subsystem, or functional unit of a wind turbine that can be individually described and considered. The model enables wind farm operators and turbine manufactures to find the most cost-effective maintenance strategy for each turbine critical item. Cost optimizations are realized through a better adaptation of the maintenance strategy to the item-specific failure modes, degradation processes, failure detection capabilities and the given operational configuration of the wind farm. Based on a time-sequential Monte Carlo simulation technique, the maintenance activities at turbine level are simulated over the windfarm’s operational lifetime, considering correlations between the stochastic variables. The results of the Monte Carlo simulation are evaluated using statistical means, thereby, determining the optimal maintenance strategy and associated parameters. The developed model is implemented as a Python application and equally applicable for onshore and offshore windfarms.

A Combined Approach for Model-Based PV Power Plant Failure Detection and Diagnostic

Photovoltaic (PV) technology allows large-scale investments in a renewable power-generating system at a competitive levelized cost of electricity (LCOE) and with a low environmental impact. Large-scale PV installations operate in a highly competitive market environment where even small performance losses have a high impact on profit margins. Therefore, operation at maximum performance is the key for long-term profitability. This can be achieved by advanced performance monitoring and instant or gradual failure detection methodologies. We present in this paper a combined approach on model-based fault detection by means of physical and statistical models and failure diagnosis based on physics of failure. Both approaches contribute to optimized PV plant operation and maintenance based on typically available supervisory control and data acquisition (SCADA) data. The failure detection and diagnosis capabilities were demonstrated in a case study based on six years of SCADA data from a PV plant in Slovenia. In this case study, underperforming values of the inverters of the PV plant were reliably detected and possible root causes were identified. Our work has led us to conclude that the combined approach can contribute to an efficient and long-term operation of photovoltaic power plants with a maximum energy yield and can be applied to the monitoring of photovoltaic plants.

Enabledness-based Testing of Object Protocols

A significant proportion of classes in modern software introduce or use object protocols, prescriptions on the temporal orderings of method calls on objects. This article studies search-based test generation techniques that aim to exploit a particular abstraction of object protocols (enabledness preserving abstractions (EPAs)) to find failures. We define coverage criteria over an extension of EPAs that includes abnormal method termination and define a search-based test case generation technique aimed at achieving high coverage. Results suggest that the proposed case generation technique with a fitness function that aims at combined structural and extended EPA coverage can provide better failure-detection capabilities not only for protocol failures but also for general failures when compared to random testing and search-based test generation for standard structural coverage.

Failure Mode & Effect Analysis and another Methodology for Improving Data Veracity and Validity

Failure Mode & Effect Analysis (FMEA) is a method that has been used to improve reliability of products, processes, designs, and software for different applications. In this paper we extend its usage for data veracity and validity improvement in the context of big data analysis and discuss its application in an electronics manufacturing test procedure which consists of a sequence of tests. Finally, we describe another methodology, developed as a result of the DVV-FMEA application which is aimed at improving the tests' repeatability and failure detection capabilities as well as monitoring their reliability.

Discovering process models for the analysis of application failures under uncertainty of event logs

Computer applications, such as servers, databases and middleware, ubiquitously emit execution traces stored in log files. The use of logs for the analysis of application failures is known since the early days of computers. Field data studies have shown that application logs are fraught with uncertainty, i.e., missing or noisy events in the logs. A body of research that has dealt successfully with uncertainty in event logs is process mining from the business process management community, specifically by discovering process models. The literature has shown the value of process mining across several domains, but as yet there is no study that quantifies possible improvements from using process models, and the impact of uncertainty in the context of application failures. This work addresses the use of process mining for detecting failures from application logs. First, process models are discovered from logs; then conformance checking is used to detect deviations from the models. We contribute to knowledge engineering research with a systematic measurement study that quantifies the failure detection capability of conformance checking in spite of missing events, and its accuracy with respect to process models obtained from noisy logs. Analysis is done with a dataset of 55,462 execution traces from three independent real-life applications. We obtain a mixed answer depending on the application under test; our measurements provide insights into the use of process mining for failure analysis.

Design of structural monitoring sensor network using surrogate modeling of stochastic sensor signal

Abstract Efficient Structural Health Monitoring (SHM) could reduce operation and maintenance costs, improve longevity, and enhance safety of the performance in complex mechanical systems. Traditional sensor network for SHM relied on simple sensor behavior without considering uncertain factors from system and environment. A probabilistic sensing model is required to simulate the realistic and stochastic sensor performance in the sensor network design process. In this paper, we introduce reliability-based design optimization for piezoelectric sensor network considering the detectability of different failure modes. The proposed method diagnoses failure based on the Mahalanobis Distance (MD) suitable for many SHM processes, while considering the uncertainties from structure properties and operation condition. Kriging is applied for surrogate modeling of the stochastic sensor signal and reduce computational cost. The optimal piezoelectric sensor network design is prototyped and its failure detection capability is experimentally verified.

A new swarm intelligence optimized multiclass multi-kernel relevant vector machine: An experimental analysis in failure diagnostics of diesel engines

This work attempts to introduce a new intelligent method for condition monitoring of diesel engines. Diesel engine is one of the most important power providers for various industrial applications, including automobiles, ships, agricultures, construction, and electrical machinery. Due to harsh working environment, diesel engines are vulnerable to failures. This article addresses a significant need to improve predictive maintenance activities in diesel engines. A new failure diagnostics approach was proposed based on the manifold learning and swarm intelligence optimized multiclass multi-kernel relevant vector machine. Three manifold learning algorithms were first respectively used to fuse the features that extracted from the original vibration data of the diesel engines into a new nonlinear space. The fused features contain the most distinct health information of the engine by discarding redundant features. Then, the swarm intelligence optimized multiclass multi-kernel relevant vector machine was proposed to identify the failures using the fused features. The contribution of this research is that the dragonfly algorithm is employed to optimize the weights of the multi-kernel functions in the multiclass relevant vector machine. It was also applied to establishing a weighted-sum model by combining the outputs of swarm intelligence optimized multiclass multi-kernel relevant vector machine models with different manifold learning algorithms. Robust failure detection of diesel engines is achieved owing to combined strengths of multiple kernel functions and weighted-sum strategy. The effectiveness of the proposed method is demonstrated by experimental vibration data collected from a commercial diesel engine. The failure detection capability of the proposed manifold learning and swarm intelligence optimized multiclass multi-kernel relevant vector machine method for diesel engines will potentially benefit the machine condition monitoring industry by improving budgeting/forecasting and/or enabling just-in-time maintenance.

Out of sight, out of mind: a distance-aware forgetting strategy for adaptive random testing

Adaptive random testing (ART) achieves better failure-detection effectiveness than random testing by increasing the diversity of test cases. However, the intention of ensuring even spread of test cases inevitably causes an overhead problem. Although two basic forgetting strategies (i.e. random forgetting and consecutive retention) were proposed to reduce the computation cost of ART, they only considered the temporal distribution of test cases. In the paper, we presented a distance-aware forgetting strategy for the fixed size candidate set version of ART (DF-FSCS), in which the spatial distribution of test cases is taken into consideration. For a given candidate, the test cases out of its “sight” are ignored to reduce the distance computation cost. At the same time, the dynamic adjustment for partitioning and the second-round forgetting are adopted to ensure the linear complexity of DF-FSCS algorithm. Both simulation analysis and empirical study are employed to investigate the efficiency and effectiveness of DF-FSCS. The experimental results show that DF-FSCS significantly outperforms the classical ART algorithm FSCS-ART in efficiency, and has comparable failure-detection effectiveness. Com-pared with two basic forgetting methods, DF-FSCS is better in both efficiency and effectiveness. In contrast with a typical linear-time ART algorithm RBCVT-Fast, our algorithm requires less computational overhead and exhibits the similar failure-detection capability. In addition, DF-FSCS has more reliable performance than RBCVT-Fast in detecting failures for the programs with high-dimensional input domain.

Error Trapping and Metamorphic Testing for Spreadsheet Failure Detection

This study deepens the research on error trapping (ET) and metamorphic testing (MT) for detecting spreadsheet failures. It is observed that some spreadsheet developers and testers are confused between ET and MT, because the two techniques are similar to each other in some aspects. Inspired by the observation, this paper first outlines the main concepts of ET and MT using several examples for illustration. This is followed by discussing an experiment with a view to investigating and comparing the failure detection capabilities of the two techniques. The results of the experiment indicate that neither technique is sufficient in spreadsheet testing. Rather, ET and MT are complementary and they should be used together in spreadsheet testing whenever possible.