Testing Techniques For Evaluation Research Articles

Testing, Validation, and Verification of Robotic and Autonomous Systems: A Systematic Review

We perform a systematic literature review on testing, validation, and verification of robotic and autonomous systems (RAS). The scope of this review covers peer-reviewed research papers proposing, improving, or evaluating testing techniques, processes, or tools that address the system-level qualities of RAS. Our survey is performed based on a rigorous methodology structured in three phases. First, we made use of a set of 26 seed papers (selected by domain experts) and the SERP-TEST taxonomy to design our search query and (domain-specific) taxonomy. Second, we conducted a search in three academic search engines and applied our inclusion and exclusion criteria to the results. Respectively, we made use of related work and domain specialists (50 academics and 15 industry experts) to validate and refine the search query. As a result, we encountered 10,735 studies, out of which 195 were included, reviewed, and coded. Our objective is to answer four research questions, pertaining to (1) the type of models, (2) measures for system performance and testing adequacy, (3) tools and their availability, and (4) evidence of applicability, particularly in industrial contexts. We analyse the results of our coding to identify strengths and gaps in the domain and present recommendations to researchers and practitioners. Our findings show that variants of temporal logics are most widely used for modelling requirements and properties, while variants of state-machines and transition systems are used widely for modelling system behaviour. Other common models concern epistemic logics for specifying requirements and belief-desire-intention models for specifying system behaviour. Apart from time and epistemics, other aspects captured in models concern probabilities (e.g., for modelling uncertainty) and continuous trajectories (e.g., for modelling vehicle dynamics and kinematics). Many papers lack any rigorous measure of efficiency, effectiveness, or adequacy for their proposed techniques, processes, or tools. Among those that provide a measure of efficiency, effectiveness, or adequacy, the majority use domain-agnostic generic measures such as number of failures, size of state-space, or verification time were most used. There is a trend in addressing the research gap in this respect by developing domain-specific notions of performance and adequacy. Defining widely accepted rigorous measures of performance and adequacy for each domain is an identified research gap. In terms of tools, the most widely used tools are well-established model-checkers such as Prism and Uppaal, as well as simulation tools such as Gazebo; Matlab/Simulink is another widely used toolset in this domain. Overall, there is very limited evidence of industrial applicability in the papers published in this domain. There is even a gap considering consolidated benchmarks for various types of autonomous systems.

Small specimen test techniques for evaluation of tensile flow properties – Evolution and developmental activities at IGCAR, Kalpakkam

The small specimen test techniques have been a subject of active research for the past 30 years. The idea of using small or miniature specimens for mechanical testing had actually originated in nuclear industry caused by space limitations in nuclear reactors for irradiation material testing with the added advantage of reduced radiation dose from smaller volumes during post-irradiation examination. Considering the potential applications of this technology for material characterization, the programme was initiated at IGCAR with the ab initio development of non conventional test techniques based on the use of small disc specimens of 3–8 mm diameters and 0.3–1.0 mm thick. The techniques that have been extensively studied are shear punch, small punch and ball-indentation tests. This paper will present the evolution of the research and developmental activities on punch and indentation based test techniques at the authors’ laboratory along with their salient results and highlight the outcome of few applications of small specimen tests employed for evaluating tensile flow properties.

A Review of Wind Tunnel Based Virtual Flight Testing Techniques for Evaluation of Flight Control Systems

Wind tunnel based Virtual Flight Testing (VFT) is a dynamic wind tunnel test for evaluating flight control systems (FCS) proposed in recent decades. It integrates aerodynamics, flight dynamics, and FCS as a whole and is a more realistic and reliable method for FCS evaluation than traditional ground evaluation methods, such as Hardware-in-the-Loop Simulation (HILS). With FCS evaluated by VFT before flight test, the risk of flight test will be further reduced. In this paper, the background, progress, and prospects of VFT are systematically summarized. Specifically, the differences among VFT, traditional dynamic wind tunnel methods, and traditional FCS evaluation methods are introduced in order to address the advantages of evaluating FCS with VFT. Secondly, the progress of VFT is reviewed in detail. Then, the test system and key technologies of VFT for FCS evaluation are analyzed. Lastly, the prospects of VFT for evaluating FCS are described.

Accounting for defect characteristics in evaluations of testing techniques

As new software-testing techniques are developed, before they can achieve widespread acceptance, their effectiveness at detecting defects must be evaluated. The most common way of evaluating testing techniques is with empirical studies, in which one or more techniques are tried out on software with known defects. However, the defects used can affect the performance of the techniques. To complicate matters, it is not even clear how to effectively describe or characterize defects. To address these problems, this article describes an experiment architecture for empirically evaluating testing techniques which takes both defect and test-suite characteristics into account. As proof of concept, an experiment on GUI-testing techniques is conducted. It provides evidence that the defect characteristics proposed do help explain defect detection, at least for GUI testing, and it explores the relationship between the coverage of defective code and the detection of defects.

JDAMA: Java database application mutation analyser

AbstractDatabase application programs are ubiquitous; hence, good techniques for testing them are needed. Several approaches to testing database applications have been proposed recently, including test generation techniques and adequacy criteria. Tuya et al. proposed mutation‐based adequacy criteria for database queries and developed a set of mutation operators for Structured Query Language SELECT statements. This paper extends that approach by integrating it with analysis and instrumentation of the application bytecode. The design and implementation of a tool, Java Database Application Mutation Analyser (JDAMA), are described. JDAMA instruments application bytecode so as to compare the results of queries executed by the application under test to the results of mutants of those queries. JDAMA can be used to support mutation testing for Java database applications. A second, potentially more useful application of JDAMA is to support experiments evaluating testing techniques for database applications, in which query mutation is used to seed faults into subject applications. The paper illustrates these uses of JDAMA with experiments comparing two approaches to test generation and experiments evaluating coverage criteria. Copyright © 2011 John Wiley & Sons, Ltd.

Comparison of delivered reliability of branch, data flow and operational testing

Many analytical and empirical studies of software testing effectiveness have used the probability that a test set exposes at least one fault as the measure of effectiveness. That measure is useful for evaluating testing techniques when the goal of testing is to gain confidence that the program is free from faults. However, if the goal of testing is to improve the reliability of the program (by discovering and removing those faults that are most likely to cause failures when the software is in the field) then the measure of test effectiveness must distinguish between those faults that are likely to cause failures and those that are unlikely to do so. Delivered reliability was previously introduced as a means of comparing testing techniques in that setting. This paper empirically compares reliability delivered by three testing techniques, branch testing, the all-uses data flow testing criterion, and operational testing. The subject program is a moderate-sized C-program (about 10,000 LOC) produced by professional programmers and containing naturally occurring faults.

Abrasivity characteristics of rocks from India and Nepal

Over the years, many attempts have been made to define and characterise rock abrasiveness. However, it is found that abrasiveness of rocks is not only difficult to define but also hard to measure. A number of relative rock abrasiveness tests have emerged but their results do not always faithfully reflect the characteristics of the rocks that may be useful in the excavation process. Under the circumstances, an in­ depth study of rock abrasiveness is warranted. The present study may provide additional help to fill a gap in this area.
 The study has addressed a number of issues on abrasivity of rocks and the objectives included inter-alia the following: 1. Developing and standardising appropriate testing techniques for evaluation of the abrasiveness and microhardness of the mine rocks. 2. Evaluating the abrasivity characteristics of mine rocks from Mosaboni copper mine, Jhinkpani limestone quarry and Indian coal-measure strata, Jharia, all from the Bihar State of India. 3. Evaluating the characteristics of a suite of rocks from Nepal (limestone from Nigale and quartzite from Bhedetar, Dhankuta District, Nepal).
 In this study a number of testing methods have been used to evaluate rock abrasiveness of rocks, specially the Cerchar Abrasivity Index (CAI), Schimazek Index and Indian School of Mines Abrasivity Test (ISMAT). The tests were performed in the rock mechanics laboratory of Indian School of Mines (ISM), Dhanbad, India. The results of these tests are discussed in detail. Hopefully, this study on abrasivity characteristics of rocks will aid in the selection of appropriate tools and excavation systems thereby enhancing the work performances and reducing the costs in excavations.

Conventional and Advanced Non-Destructive Test Techniques for Evaluation of Welds

Conventional and Advanced Non-Destructive Test Techniques for Evaluation of Welds