Fault Detection Effectiveness Research Articles

Adaptive Partition Testing

Random testing and partition testing are two major families of software testing techniques. They have been compared both theoretically and empirically in numerous studies for decades, and it has been widely acknowledged that they have their own advantages and disadvantages and that their innate characteristics are fairly complementary to each other. Some work has been conducted to develop advanced testing techniques through the integration of random testing and partition testing, attempting to preserve the advantages of both while minimizing their disadvantages. In this paper, we propose a new testing approach, adaptive partition testing , where test cases are randomly selected from some partition whose probability of being selected is adaptively adjusted along the testing process. We particularly develop two algorithms, Markov-chain based adaptive partition testing and reward-punishment based adaptive partition testing , to implement the proposed approach. The former algorithm makes use of Markov matrix to dynamically adjust the probability of a partition to be selected for conducting tests; while the latter is based on a reward and punishment mechanism. We conduct empirical studies to evaluate the performance of the proposed algorithms using ten faulty versions of three large-scale open source programs. Our experimental results show that, compared with two baseline techniques, namely random partition testing (RPT) and dynamic random testing (DRT), our algorithms deliver higher fault-detection effectiveness with lower test case selection overhead. It is demonstrated that the proposed adaptive partition testing is an effective testing approach, taking advantages of both random testing and partition testing.

TSV Prebond Test Method Based on Switched Capacitors

As an integrated circuit (IC) technology develops, 3-D stacked ICs based on through-silicon-via (TSV) technology have attracted the attention of the semiconductor industry. In 3-D stacked ICs, the TSV forms signal paths that vertically connect different ICs. The signal integrity largely depends on the quality of the TSV. A key challenge to the commercial viability of 3-D ICs is developing a method of accomplishing prebond TSV tests. To address this problem, we propose a prebond TSV test method with easily applied Design for Testability (DfT) structures and a simple fault-detection process. The proposed method is based on switched-capacitor circuits, which can detect TSV leakage faults, open faults, and high-resistance faults with high resolution. The fault-detection effectiveness is evaluated via HSPICE simulations. We also present an overall assessment of the test resolution, test time, and DfT area cost.

Fault Detection Algorithm Design of Chemical Robots Based on Internet of Things and Deep Learning Feature Extraction Technology

In order to improve the fault detection effect of intelligent Chemical robots, this paper design fault detection algorithm of intelligent Chemical robots. Based on the Internet of Things and deep learning feature extraction technology, this paper combines WIFI and TCP/IP to optimize the RSSI path loss model, and obtains a path loss model with multi-path effects suitable for Chemical roadways to construct simplified model of robots. It is found that the robots could climb over vertical barriers with a height of 19 cm. It can be seen that the practicability and reliability of communication system are verified by the experiment. The stability conditions and theoretical barrier height of robots are obtained, which is significantly improved.

Oracles for Testing Software Timeliness with Uncertainty

Uncertainty in timing properties (e.g., detection time of external events) is a common occurrence in embedded software systems, since these systems interact with complex physical environments. Such time uncertainty leads to non-determinism. For example, time-triggered operations may either generate different valid outputs across different executions or experience failures (e.g., results not being generated in the expected time window) that occur only occasionally over many executions. For these reasons, time uncertainty makes the generation of effective test oracles for timing requirements a challenging task. To address the above challenge, we propose Stochastic Testing with Unique Input Output Sequences, an approach for the automated generation of stochastic oracles that verify the capability of a software system to fulfill timing constraints in the presence of time uncertainty. Such stochastic oracles entail the statistical analysis of repeated test case executions based on test output probabilities predicted by means of statistical model checking. Results from two industrial case studies in the automotive domain demonstrate that this approach improves the fault detection effectiveness of tests suites derived from timed automata compared to traditional approaches.

FBDTester 2.0: Automated test sequence generation for FBD programs with internal memory states

As Function Block Diagram (FBD) programs are used to implement safety-critical systems such as nuclear reactor protection systems, it is crucial to generate effective test cases. FBD programs are running on programmable logic controllers (PLCs) that execute the programs repeatedly within a particular scan time. Among the elements of FBD programs, function blocks and feedback variables save the inputs or outputs of the previous scan cycle into internal memories. The data stored in the internal memory are used to operate the function blocks of the next scan cycle. Researchers have recently developed an automated test generation technique aiming to satisfy several structural test coverage criteria for FBD programs. Among them, Jee et al.'s approach has an advantage of generating tests from FBD programs without requiring any intermediate models. However, test sequence generation for FBD programs with internal memory states was not fully automated in Jee et al.'s approach. This paper extends Jee et al.'s work by generating test sequences from FBD programs with internal memory states in a fully automated manner. The key technique involves explicitly unwinding FBD programs and solving the test requirements using an SMT solver. We conduct experiments on increasing the coverage of test requirements by unwinding cycles and evaluating the effectiveness of the test set using mutation analysis with several subject programs, including a real-world reactor protection system. The experimental results show that the proposed approach is able to generate effective test sequences for FBD programs. To demonstrate the efficiency of the proposed approach, we compared the test sequence generation time required for our approach with those for other approaches. The comparison results show that even excluding the generation time of test models of other approaches, the proposed approach increased the time-efficiency by more than 11 times. • Developing algorithms to generate test sets for Function Block Diagram programs. • Defining Function Block Conditions for all the standard function blocks. • Implementing an automated test sequence generation tool using Yices SMT solver. • Demonstrating improved fault-detection effectiveness and time efficiency.

False alarm reducing in PCA method for sensor fault detection in a nuclear power plant

Principal component analysis (PCA) is applied for fault detection of sensors in a nuclear power plant (NPP) in this paper. In order to reduce the false alarms of T2 and Q statistics during fault detection, two different methods are further proposed in this paper. One is statistics-based method which generates second confidence limits for T2 and Q statistics, and then false alarms are reduced based on the second confidence limit during test. The other is iteration-based method which reduces false alarms during modeling. Measurements beyond the first confidence limit of T2 or Q statistics are successively removed from the training data through iteration process. Finally, sensor measurements from a real NPP are acquired to train and test the proposed methods. On one hand, simulation results show that the proposed PCA model is capable of detecting the faulty sensors no matter with small or major failures. On the other hand, simulations also indicate that the PCA model combined with statistics-based and iteration-based methods simultaneously makes more contribution to the timeliness and effectiveness of sensor fault detection compared with the PCA model only with statistics-based or iteration-based method.

Investigating a Distributed and Scalable Model Review Process

[Context] Models play an important role in Software and Systems Engineering processes. Reviews are well-established methods for model quality assurance that support early and efficient defect detection. However, traditional document-based review processes have limitations with respect to the number of experts, resources, and the document size that can be applied. [Objective] In this paper, we introduce a distributed and scalable review process for model quality assurance to (a) improve defect detection effectiveness and (b) to increase review artifact coverage. [Method] We introduce the novel concept of Expected Model Elements (EMEs) as a key concept for defect detection. EMEs can be used to drive the review process. We adapt a best-practice review process to distinguish (a) between the identification of EMEs in the reference document and (b) the use of EMEs to detect defects in the model. We design and evaluate the adapted review process with a crowdsourcing tool in a feasibility study. [Results] The study results show the feasibility of the adapted review process. Further, the study showed that inspectors using the adapted review process achieved results for defect detection effectiveness, which are comparable to the performance of inspectors using a traditional inspection process, and better defect detection efficiency. Moreover, from a practical perspective the adapted review process can be used to complement inspection efforts conducted using the traditional inspection process, enhancing the overall defect detection effectiveness. [Conclusions] Although the study shows promising results of the novel process, future investigations should consider larger and more diverse review artifacts and the effect of using limited and different scopes of artifact coverage for individual inspectors.

SPIRITuS: a SimPle Information Retrieval regressIon Test Selection approach

Context:Regression Test case Selection (RTS) approaches aim at selecting only those test cases of a test suite that exercise changed parts of the System Under Test (SUT) or parts affected by changes.Objective:We present SPIRITuS (SimPle Information Retrieval regressIon Test Selection approach). It uses method code coverage information and a Vector Space Model to select test cases to be run. In a nutshell, the extent of a lexical modification to a method is used to decide if a test case has to be selected. The main design goals of SPIRITuS are to be: (i) easy to adapt to different programming languages and (ii) tunable via an easy to understand threshold.Method:To assess SPIRITuS, we conducted a large experiment on 389 faulty versions of 14 open-source programs implemented in Java. We were mainly interested in investigating the tradeoff between the number of selected test cases from the original test suite and fault detection effectiveness. We also compared SPIRITuS against well-known RTS approaches.Results:SPIRITuS selects a number of test cases significantly smaller than the number of test cases the other approaches select at the price of a slight reduction in fault detection capability.Conclusions:SPIRITuS can be considered a viable competitor of existing test case selection approaches especially when the average number of test cases covering a modified method increases (such information can be easily derived before test case selection takes place).

Asynchronous fault detection and robust control for switched systems with state reset strategy

This paper is concerned with the problem of simultaneous fault detection and control of switched systems under the asynchronous switching. A switching law and fault detection/control units called fault detector/controllers are designed to guarantee the fault sensitivity and robustness of the closed-loop systems. Different from the existing results, a state reset strategy is introduced in the process of fault detection/control, which reduces the conservatism caused by the jump of multiple Lyapunov functions at switching instants. Further, the proposed strategy is only dependent the state of fault detector/controllers, which is available when the system state is invalid. Finally, by using a performance gain transform technique, non-convex fault sensitivity conditions are converted into the convex error attenuation ones. This further improves the fault detection effect. A numerical example is given to demonstrate the effectiveness of the proposed results.

Test case prioritization for object-oriented software: An adaptive random sequence approach based on clustering

Test case prioritization (TCP) attempts to improve fault detection effectiveness by scheduling the important test cases to be executed earlier, where the importance is determined by some criteria or strategies. Adaptive random sequences (ARSs) can be used to improve the effectiveness of TCP based on white-box information (such as code coverage information) or black-box information (such as test input information). To improve the testing effectiveness for object-oriented software in regression testing, in this paper, we present an ARS approach based on clustering techniques using black-box information. We use two clustering methods: (1) clustering test cases according to the number of objects and methods, using the K-means and K-medoids clustering algorithms; and (2) clustered based on an object and method invocation sequence similarity metric using the K-medoids clustering algorithm. Our approach can construct ARSs that attempt to make their neighboring test cases as diverse as possible. Experimental studies were also conducted to verify the proposed approach, with the results showing both enhanced probability of earlier fault detection, and higher effectiveness than random prioritization and method coverage TCP technique.

The effectiveness of fault detection in common rail injectors examination methods

The article presents the effectiveness tests of fault detection in common rail injectors. 40 injectors with different wear levels were tested. Testing was made on two test benches of a completely different design. Research includes comparison of accuracy, reproducibility and testability to detect specific defects. A device was created for visualization of the fuel injector spraying steam.

A Similarity Metric for the Inputs of OO Programs and Its Application in Adaptive Random Testing

Random testing (RT) has been identified as one of the most popular testing techniques, due to its simplicity and ease of automation. Adaptive random testing (ART) has been proposed as an enhancement to RT, improving its fault-detection effectiveness by evenly spreading random test inputs across the input domain. To achieve the even spreading, ART makes use of distance measurements between consecutive inputs. However, due to the nature of object-oriented software (OOS), its distance measurement can be particularly challenging: Each input may involve multiple classes, and interaction of objects through method invocations. Two previous studies have reported on how to test OOS at a single-class level using ART. In this study, we propose a new similarity metric to enable multiclass level testing using ART. When generating test inputs (for multiple classes, a series of objects, and a sequence of method invocations), we use the similarity metric to calculate the distance between two series of objects, and between two sequences of method invocations. We integrate this metric with ART and apply it to a set of open-source OO programs, with the empirical results showing that our approach outperforms other RT and ART approaches in OOS testing.

Aircraft engine sensor fault diagnostics using an on-line OBEM update method.

This paper proposed a method to update the on-line health reference baseline of the On-Board Engine Model (OBEM) to maintain the effectiveness of an in-flight aircraft sensor Fault Detection and Isolation (FDI) system, in which a Hybrid Kalman Filter (HKF) was incorporated. Generated from a rapid in-flight engine degradation, a large health condition mismatch between the engine and the OBEM can corrupt the performance of the FDI. Therefore, it is necessary to update the OBEM online when a rapid degradation occurs, but the FDI system will lose estimation accuracy if the estimation and update are running simultaneously. To solve this problem, the health reference baseline for a nonlinear OBEM was updated using the proposed channel controller method. Simulations based on the turbojet engine Linear-Parameter Varying (LPV) model demonstrated the effectiveness of the proposed FDI system in the presence of substantial degradation, and the channel controller can ensure that the update process finishes without interference from a single sensor fault.

An Efficient Heuristic Based Test Suite Minimization Approach

Objectives: Development of an efficient test suite minimization approach in order to reduce the size of a previously acquired test suite and produce a new representative suite which will guarantee the same requirement coverage that was achieved before minimization for an effective and efficient regression testing. Method: Test suite minimizations techniques try to reduce the size and redundancy of test suite by removing certain test cases since requirement covered by them are already covered by other test cases. But, it has been found that the acquired test cases after minimization severely lacks ability to achieve the desirable code coverage because the minimization was done based on a single test adequacy criteria. In this paper, we propose an efficient heuristic based test suite minimization algorithm which will reduce the size of the test suites with respective to multiple test adequacy criterions in order to preserve the fault detection effectiveness and code coverage characteristics of the final test suite. Findings: Our experimental results indicate that a significant percentage of reduction in the test suite size is achieved when the minimization is performed with respect to multiple test adequacy criterions. Our approach is unique compared to the existing approaches in the sense that, we carried out minimization based on multiple test adequacy criterions while most of the existing approaches usually take one or two criterions into consideration. The proposed approach is evaluated based on two well known software testing metrics; one indicate the percentage of reduction in test suite size and the second one indicate the percentage of code coverage achieved by the minimized test suite. Our experimental results indicate that a significant percentage of reduction in the size as well as significant code coverage characteristics is achieved when the minimization is done according to the proposed approach. Improvements: The important contribution of this study is that, it presents a novel and efficient test suite minimization technique that optimizes the test suite size based on multiple adequacy criterions.

An Improvement in Defect Detection Efficiency -A Review

Objectives: The defect detection efficiency has to be improved by comparing different software development lifecycles and finding the best of defect detection methodology, along with the accurate defect rate analysis and classification and to achieve 100% efficiency in defect detection and attain 100% customer satisfaction. Methods and Statistical Analysis:- Agile methodology, with In-memory analytics is employed to improve the effectiveness of defect detection efficiency. Along with Agile methodology, defect comparison and classification of defects based on defect rate with respect to the standard threshold values, at each stage of the design process can be employed. In memory analytics can be employed for Defect classification. This method provides effectiveness of defect classification and rates defect as low, high and medium. This method easier the tasks of designer to detect the severity of defect and rectify, to avoid the defect being added to subsequent phases. Findings: It is found that, In Waterfall model, the product is tested only after the product has been completely manufactured. The defect detection effectiveness considering an average of 15-20 percentages of defects originating at each phase is 50 percent. In Six Sigma approach, the defect detection effectiveness which is improved to 99.9997 percent with the same percentage of defect originating at each phase. While 100 percent defect detection effectiveness is not practically possible. Hence the greatest challenge is how the testing engineers can meet 100 percent testing standard. The testing engineers need to adapt a unique technique to remove the defects before they get added to the system. Such technique will not only helps to detect defects faster but also reduces the high cost of poor quality products.

Online Monitoring of Electromagnetic Losses in an Electric Motor Indirectly Through Temperature Measurement

The condition monitoring of an electric motor is critical to its stable and reliable operation. The consequence of a fault is the variation in output performance and changes in the input electrical signature. One of the most common methods of fault detection is by analyzing the input current signature but this often requires expensive sensors. The other symptom of a faulty machine is when it runs “hot” due to an increase in losses generated. Temperature changes could be a useful indicator of the underlying fault. In this paper, a method of condition monitoring using temperature sensors embedded in an electric motor is presented. The components of electromagnetic losses generated are determined indirectly from temperature measurement in real time for a continuous drive cycle simulating normal and faulty operations. This is achieved through the inversion of the thermal model of the machine coupled with the application of a recursive parameter estimation method. Using temperature measurement for condition monitoring is advantageous since temperature sensors are low cost and already embedded in the machine for thermal protection. However, application of the method to a faulty motor and evaluation of its effectiveness in fault detection will be reported in a future work.

An Empirical Study on Mutation Testing of WS-BPEL Programs

Nowadays, applications are increasingly deployed as Web services in the globally distributed cloud computing environment. Multiple services are normally composed to fulfill complex functionalities. Business Process Execution Language for Web Services (WS-BPEL) is an XML-based service composition language that is used to define a complex business process by orchestrating multiple services. Compared with traditional applications, WS-BPEL programs pose many new challenges to the quality assurance, especially testing, of service compositions. A number of techniques have been proposed for testing WS-BPEL programs, but only a few studies have been conducted to systematically evaluate the effectiveness of these techniques. Mutation testing has been widely acknowledged as not only a testing method in its own right but also a popular technique for measuring the fault-detection effectiveness of other testing methods. Several previous studies have proposed a family of mutation operators for generating mutants by seeding various faults into WS-BPEL programs. In this study, we conduct a series of empirical studies to evaluate the applicability and effectiveness of various mutation operators for WS-BPEL programs. The experimental results provide insightful and comprehensive guidance for mutation testing of WS-BPEL programs in practice. In particular, our work is the systematic study in the selection of effective mutation operators specifically for WS-BPEL programs.

Evaluation and analysis of incorporating Fuzzy Expert System approach into test suite reduction

ContextSoftware has become increasingly important in our modern society. However, when new features are developed due to user requests, such requests could make the sizes of test-case pools bigger. Many techniques are proposed to solve this problem, such as test suite reduction. However, the ability to expose faults may be weakened when reducing the sizes of the test suites. In this paper, we propose some methods using fuzzy logic in order to improve existing test-suite reduction techniques. ObjectiveThe main purpose of this research is to use a Fuzzy Expert System approach in order to enhance the effectiveness of fault detection during software testing. MethodIncorporating a Fuzzy Expert System into traditional test suite reduction techniques is presented and studied. More objective criteria are used in order to compare the performance of our proposed and selected test suite reduction methods. Some important measures (and metrics) will also be obtained and discussed. Application of the Fuzzy Expert System approach for test case prioritization is also discussed. ResultsThe experiments in three improved test-suite reduction techniques show that the modified algorithms can reduce the size of test suites, which have improved the fault detection quality. ConclusionDuring software testing, test data are generally classified with Boolean logic. This method can classify data into groups easily. However, there may be ambiguity in classifications due to similar properties for certain data. Ambiguous data can be classified in each group. In this study, Boolean logic will be replaced by fuzzy logic. Incorporating the Fuzzy Expert System approach into three traditional test suite reduction techniques (i.e., HGS, GRE, and Greedy) is presented and evaluated. The experiments, based on nine real subject programs ranging from 173 LOCs to 35,545 LOCs, have demonstrated that our proposed Fuzzy-HGS, Fuzzy-GRE, and Fuzzy-Greedy algorithms can significantly reduce the sizes of test suites while also improving fault detection effectiveness. For instance, Fuzzy-HGS, Fuzzy-GRE, and Fuzzy-Greedy algorithms have almost the same reduction capability of test suite as traditional HGS, GRE, and Greedy algorithms. But in terms of the percentage of fault detection effectiveness loss (FDE loss), both Fuzzy-HGS and Fuzzy-GRE algorithms are averagely decreased by 21% and 5%, respectively. Additionally, Fuzzy-Greedy algorithm still provide the lower FDE loss for large subject programs compared to traditional Greedy algorithm. Based upon the integrated theoretical foundation, the approaches presented in this paper offer an efficient, useful software testing scheme in the testing and debugging phases.

Scope-aided test prioritization, selection and minimization for software reuse

Abstract Software reuse can improve productivity, but does not exempt developers from the need to test the reused code into the new context. For this purpose, we propose here specific approaches to white-box test prioritization, selection and minimization that take into account the reuse context when reordering or selecting test cases, by leveraging possible constraints delimiting the new input domain scope. Our scope-aided testing approach aims at detecting those faults that under such constraints would be more likely triggered in the new reuse context, and is proposed as a boost to existing approaches. Our empirical evaluation shows that in test suite prioritization we can improve the average rate of faults detected when considering faults that are in scope, while remaining competitive considering all faults; in test case selection and minimization we can considerably reduce the test suite size, with small to no extra impact on fault detection effectiveness considering both in-scope and all faults. Indeed, in minimization, we improve the in-scope fault detection effectiveness in all cases.

Data dimensionality reduction and data fusion for fast characterization of green coffee samples using hyperspectral sensors.

Hyperspectral sensors represent a powerful tool for chemical mapping of solid-state samples, since they provide spectral information localized in the image domain in very short times and without the need of sample pretreatment. However, due to the large data size of each hyperspectral image, data dimensionality reduction (DR) is necessary in order to develop hyperspectral sensors for real-time monitoring of large sets of samples with different characteristics. In particular, in this work, we focused on DR methods to convert the three-dimensional data array corresponding to each hyperspectral image into a one-dimensional signal (1D-DR), which retains spectral and/or spatial information. In this way, large datasets of hyperspectral images can be converted into matrices of signals, which in turn can be easily processed using suitable multivariate statistical methods. Obviously, different 1D-DR methods highlight different aspects of the hyperspectral image dataset. Therefore, in order to investigate their advantages and disadvantages, in this work, we compared three different 1D-DR methods: average spectrum (AS), single space hyperspectrogram (SSH) and common space hyperspectrogram (CSH). In particular, we have considered 370 NIR-hyperspectral images of a set of green coffee samples, and the three 1D-DR methods were tested for their effectiveness in sensor fault detection, data structure exploration and sample classification according to coffee variety and to coffee processing method. Principal component analysis and partial least squares-discriminant analysis were used to compare the three separate DR methods. Furthermore, low-level and mid-level data fusion was also employed to test the advantages of using AS, SSH and CSH altogether. Graphical Abstract Key steps in hyperspectral data dimenionality reduction.