Test Adequacy Research Articles

A hybrid approach to quantify software reliability in nuclear safety systems

Technological advancements have led to the use of computer based systems in safety critical applications. As computer based systems are being introduced in nuclear power plants, effective and efficient methods are needed to ensure dependability and compliance to high reliability requirements of systems important to safety. Even after several years of research, quantification of software reliability remains controversial and unresolved issue. Also, existing approaches have assumptions and limitations, which are not acceptable for safety applications.This paper proposes a theoretical approach combining software verification and mutation testing to quantify the software reliability in nuclear safety systems. The theoretical results obtained suggest that the software reliability depends on three factors: the test adequacy, the amount of software verification carried out and the reusability of verified code in the software. The proposed approach may help regulators in licensing computer based safety systems in nuclear reactors.

An intuitive approach to determine test adequacy in safety-critical software

Safety-critical software must adhere to stringent quality standards and is expected to be thoroughly tested. However, exhaustive testing of software is usually impractical. The two main challenges faced by a software testing team are generation of effective test cases and demonstration of testing adequacy. This paper proposes an intuitive and conservative approach to determine the test adequacy in safety-critical software. The approach is demonstrated through a case study: the core temperature monitoring system of a nuclear reactor. We combine conservative test coverage of unique execution path test cases, and the results from mutation testing to determine the test adequacy. Although mutation testing is a powerful technique, the difficulty in identifying equivalent mutants has limited its practical utility. To gain confidence on the computed test adequacy: (i) faults during mutation testing must be induced at all possible execution paths of the code, (ii) properties of unkilled mutants must be studied, and (iii) all equivalent mutants must be detected. In this regard; results of static, dynamic and coverage analysis of the mutants is presented, and a technique to identify the likely equivalent mutants is proposed.

Automatic Generation and Execution of Mutants

To obtain the high quality software, there is a use of Mutation testing to measure the quality of our test suite. Fault insertion based techniques have been used for measuring test adequacy and testability of programs. Mutants are generated by introducing the faults in the original program. Tests Cases are adequate if they detect all the mutants. This paper describes a survey study to investigate the generation and execution of mutants. General Terms Software Testing

Software Reliability Test Based on Markov Usage Model

The reliability test for embedded software system is very difficult due to its complex structure and large scale. Markov usage model which describes the software usage with Markov process is widely used for statistical test. Software statistical test based on Markov usage model is an effective approach to the generation of test cases with high efficiency and the evaluation of software reliability in a quantitative way. This paper mainly focuses on the generation of Markov usage model of software system and the method of software reliability test based on it. Firstly, a method to build Markov usage model based on improved state transition matrix (STM), which is a table-based modeling language, is proposed. Then a software reliability test method including test case generation and test adequacy determination based on Markov usage model is presented. An improved Kullback discriminant is selected as the judgment criteria of convergence from the test chain to the usage chain in order to measure whether the testing process is sufficient. Finally, a Markov test cases generation tool MTCG is developed which implements the methods put forward in this paper. Experimental verification of test adequacy and efficiency is made through a comparison between the Markov usage model-based method and a completely random test method, the results of which show that software reliability test based on Markov usage model is approving, high-efficient and promising.

Generating Test Cases by an Innovative Way

Generate sufficient test cases is difficult and expensive, especially for testing software systems whose input is structurally complex. This paper proposes a new approach to generating a large number of test data from a few seed test cases, called data mutation. In the new approach, mutation operators transform the input data rather than the program under test or the specification of the software. It is not a test adequacy measurement. Instead, it generates test cases. The paper also reports a case study with the method on testing a modeling tool and illustrates the applicability of the proposed method. Experimental results validate the effectiveness of the method.

Analysis on Characteristic Features of L-Mode Guided Wave in Tube Testing

Applications of long range ultrasonic testing are on cylindrical products(pipes and tubes) and planar products(plates and sheets).Guided wave testing using circumferential and longitudinal modes are getting more and more importance as there are huge requirements for testing pipes and tubes in power generators, Chemical plants, steam generators etc.. There will be infinite number of modes with charectestic propagation and wave structure. This propagation characteristic and wave structure plays a major role in deciding its suitability for use in specific application. It is highly essential to use mode control and to understand the generated modes and its characteristics based on which only testing adequacy can be evaluated. In this paper the characteristic features of L-mode generated using a source design parameter for tube testing are explained. The dispersion characteristic of L-mode are discussed using constructed Dispersion Curves, phase and group velocity, its multiple defect detection capability, pulse shape and amplitude dynamics with propagation range and inclination. The details of controlled experiment conducted and the observations are discussed. The derived inferences are explained based on the applicable particle displacement equations and dispersion characteristic equations.

Assurance Technology of Testing Adequacy Based on Functional Analysis

Sufficient and efficient functional testing could efficiently find the defects existed in product development. However, due to the growing complexity of product, sufficient functional and performance testing is more difficult. Integrality of test objectives and correctness of test approach are essential to ensure the sufficient testing of products. This paper focused on the problem of integrality of test objectives and proposed a new functional analysis approach based on physical view to list all the functions and interrelationships between them.

The Test–Retest Reliability and Predictive Validity of a Battery of Newly Developed Occupational Performance Assessments

The purpose of the author in this study was to investigate the test–retest reliability and predictive validity of new occupational performance assessments. The researcher used a test–retest design with mixed quantitative and qualitative methods. The test–retest reliability of the perceived adequacy of engagement in occupations of priority to participants as measured on the assessments was r (15) = .54, p < .05. Perceived test adequacy predicted 27% [B = .517, t = 2.18, R2 = .27, F (1, 13) = 4.75, p = .048] and retest adequacy predicted 67% [B = .820, t = 5.16, R2 = .67, F (1, 13) = 26.61, p = .000] of variability in the retest frequency of engagement in occupations seen as a priority by research participants. Test satisfaction scores predicted 50% of variability in the test frequency of engagement in occupations [B = .707, t = 3.61, R2 = .50, F (1, 13) = 13.01, p = .003]. It was concluded that adequacy and satisfaction scores could be used by occupational therapists in planning therapeutic interventions to facilitate future performance of occupations seen as important by clients.

표준보육과정에 기초한 영아관찰척도 개발을 위한 기초 연구

The purpose of this study is to develop a valid and reliable infant observation scale based on the national curriculum for childcare centers. The study was carried out using the final draft of the Infant Observation Scale for 1 year old infants. The scale consists of 42 items in 6 areas. Whereas the final draft of the Infant Observation Scale for 2 year old infants consists of 52 items in 6 areas. The subjects of this observation scale were 120 infants from the city of Busan. The results of the Infant Observation Scale for the 1 year old infant analysis for the purpose of test adequacy indecates that the internal consistency reliability is from .88 to .97. The results of the Infant Observation Scale for 2 year old infant analysis for test adequacy shows that the internal consistency reliability is from .76 to .97.

Study on the Method for Fault Sample Selection Based on Fuzzy Clustering

A new method of fault sample selection is presented. First, the mapping relationship is established between the system components and system functions attributes according to the similarity of the system components and system functions attributes, the components are clustered to the class by fuzzy clustering method; then the fault sample is selected in the class according to relation of the faults propagation, the selected fault sample is validated by using the function adequacy and test adequacy. Finally From experimental results, the conclusion can be drown that the cost of this method is lower and the fault detection ratio is high. Compare with the other method the new method has certain advantages.

Test Case Generation for Adequacy of Floating-point to Fixed-point Conversion

Porting an application written for personal computer to embedded devices requires conversion of floating-point numbers and operations into fixed-point ones. Testing the conversion hence requires the latter be as close as possible to the former. The closeness is orthogonal to code coverage and requires different strategies to generate a test suite that reveals the gap between the two functions. We introduce a new test adequacy criterion and propose several metrics to quantify the closeness of two functions. After that we propose a method to generate a better test suite from a given one for the test adequacy criteria. We also show experimental results on some well-known mathematical functions.

Studentische Evaluation einer objektiven, strukturierten klinischen Prüfungsmethode (OSCE) im Fach Chirurgie und Orthopädie

The objective structured clinical exam (OSCE) has become an established form of examination. However, for general and orthopedic surgery it has barely been evaluated. Therefore, the present study was performed to analyze the OSCE in surgery by the students of the University of Ulm. In total 304 medical students undertaking the OSCE were included in the study. The students were asked to fill out a standardized questionnaire which contained different evaluation parameters, such as test adequacy, comprehensibility, balance, difficulty, atmosphere and clinical relevance as well as self-assessment and overall rating. In the overall rating the OSCE was rated as having a clinical relevance. The preferred preparation strategies were the clinical traineeship and standard medical textbooks. Altogether, the OSCE was chosen as the preferred future examination method, followed by multiple choice testing and clinical practical examination. The evaluation of the OSCE by the medical students at the University of Ulm showed a high acceptance rate as well as a high clinical relevance.

A Novel Approach for Optimized Test Case Generation Using Activity and Collaboration Diagram

Testing is the process of building confidence of the programmer that shows, the software does what it is intended to do, which in turn improves the reliability of the software. And automation of software testing process helps in achieving it with reduced cost and time. Test case generation is one part of the testing process with description of a test and independent of designed system, intended to find errors. The advantage of generation of test cases from specifications and design is that they can be available during early phase of the software development life cycle and there is no need to wait for development of codes to test the software. Additionally early test case generation reduces errors, inconsistencies and ambiguities, during the life cycle, because developers can use test cases to control their program to conform to the software specification. In this paper we have applied Constraint-based Genetic Algorithm technique to generate optimized test cases from UML Activity diagram and Collaboration diagram. Our proposed approach is more effective, which uncovers more number of errors, by using combinatorial optimization technique such as genetic algorithm with transition coverage, a test adequacy criterion as a constraint. We have defined an error minimization technique in our approach, which works as a basic principle for optimized test case generation. That means the generated test case have lower chance of presence of errors, by discarding the rest. The proposed approach is discussed by considering ATM cash withdrawal as a case study. The full text of the article is not available in the cache. Kindly refer the IJCA digital library at www.ijcaonline.org for the complete article. In case, you face problems while downloading the full-text, please send a mail to editor at editor@ijcaonline.org

ASSESSING TEST SUITES FOR BUFFER OVERFLOW VULNERABILITIES

Over the last few years, numerous vulnerabilities have been reported in software, and successful exploitations of these vulnerabilities have resulted in severe consequences such as denial of services and application state corruptions. Researches have shown that effective quality assurance methods can prevent such consequences when applied during software (or applications) development processes. Software security testing is a popular assurance method in this direction. However, effective testing involves obtaining an effective test suite (or collection of test cases) that can reveal specific faults. Over the last few years, different testing approaches have been applied for revealing vulnerabilities in software. However, only few works have assessed the effectiveness of test suites for revealing vulnerabilities. We believe that bringing the idea of mutation-based assessment of test adequacy for vulnerabilities can help in detecting and removing vulnerabilities proactively. In this work, we apply mutation-based adequate testing for one of the worst vulnerabilities namely buffer overflow (BOF). We propose 16 mutation operators to force the generation of adequate test suites for BOF vulnerabilities. A prototype tool is developed to automatically generate mutants and perform mutation analysis with input test cases. The effectiveness of the operators is evaluated by using several benchmark programs having BOF vulnerabilities, and the results indicate that the proposed operators are effective for testing BOF vulnerabilities. Moreover, we present an analysis to find selective mutation operators for reducing the cost of mutation-based testing of BOF vulnerabilities.

Research on Formal Verification Technique for Aircraft Safety-Critical Software

As an important part of airborne avionics system, aircraft safety critical software (ASCS) plays an essential role to the safety of the aircraft, and to ensure its quality and reliability is one of the key problems we are facing. Formal methods have become important means for modeling and verifying safety critical software. In this paper, formal method is introduced into the ASCS verification field and the real-time extended finite state machine model (RT-EFSM) is studied, which includes the detailed real-time extension schemes and its validation methods. The verification process of ASCS based on RT-EFSM is also proposed. Furthermore, combined with the verification of an aircraft inertial/satellite navigation systems, a timed unique input/output sequence (t_UIO) is presented and the automatic generation algorithm of sub-transfer sequences based on t_UIO is given. Finally, the test adequacy criteria are discussed and a time condition coverage criterion is proposed. The actual engineering project application shows that the method proposed in this paper is of great value for the ASCS, which can be generally and effectively used in engineering.

Automated software test optimisation framework – an artificial bee colony optimisation-based approach

Software test suite optimisation is one of the most important problems in software engineering research. To achieve this optimisation, a novel approach based on artificial bee colony (ABC) optimisation is proposed here. The work applied in this approach is motivated by the intelligent behaviour of honey bees. Since the ABC system combines local search methods carried out by employed and onlooker bees with global search methods managed by scouts, the approach attains global or near-global optima. Here, the parallel behaviour of the three bees is used to reach the solution generation faster. The performance of the proposed approach is investigated based on coverage-based test adequacy criteria by comparing it with sequential ABC, random testing and genetic algorithm-based approaches. Based on the experimental results, it has been proved that the proposed parallel ABC approach outperforms the other approaches in test suite optimisation.

Profiling the operational behavior of OS device drivers

As the complexity of modern Operating Systems (OS) increases, testing key OS components such as device drivers (DD) becomes increasingly complex given the multitude of possible DD interactions. Currently, DD testing entails a broad spectrum of techniques, where static (requiring source code) and dynamic (requiring the executable image) and static-dynamic testing combinations are employed. Despite the sustained and improving test efforts in the field of driver development, DDs still represent a significant cause of system outages as the coverage is invariably limited by test resources and release time considerations. The basic factor is the inability to exhaustively assess and then cover the operational states, leading to releases of inadequately tested DDs. Consequently, if representative operational activity profiles of DDs within an OS could be obtained, these could significantly improve the understanding of the actual operational DD state space and help focus the test efforts. Focusing on characterizing DD operational activities while assuming no access to source code, this paper proposes a quantitative technique for profiling the runtime behavior of DDs using a set of occurrence and temporal metrics obtained via I/O traffic characterization. Such profiles are used to improve test adequacy against real-world workloads by enabling similarity quantification across them. The profiles also reveal execution hotspots in terms of DD functionalities activated in the field, thus allowing for dedicated test campaigns. A case study on actual Windows XP and Vista drivers using various performance and stability benchmarks as workloads substantiates our proposed approach.

Quality of routine spirometry tests in Dutch general practices.

Spirometry is an indispensable tool for diagnosis and monitoring of chronic airways disease in primary care. To establish the quality of routine spirometry tests in general practice, and explore associations between test quality and patient characteristics. Analysis of routine spirometry test records. Fifteen general practices which had a working agreement with a local hospital pulmonary function laboratory for spirometry assessment regarding test quality and interpretation. Spirometry tests were judged by a pulmonary function technician and a chest physician. Proportions of test adequacy were analysed using markers for manoeuvre acceptability and test reproducibility derived from the 1994 American Thoracic Society spirometry guideline. Associations between quality markers and age, sex, and severity of obstruction were examined using logistic regression. Practices performed a mean of four (standard deviation = 2) spirometry tests per week; 1271 tests from 1091 adult patients were analysed; 96.4% (95% confidence interval [CI] = 95.6 to 97.2) of all tests consisted of ≥3 blows. With 60.6% of tests, forced expiratory time was the marker with the lowest acceptability rate. An overall 38.8% (95% CI = 36.0 to 41.6) of the tests met the acceptability as well as reproducibility criteria. Age, sex, and severity of obstruction were associated with test quality markers. The quality of routine spirometry tests was better than in previous reports from primary care research settings, but there is still substantial room for improvement. Sufficient duration of forced expiratory time is the quality marker with the highest rate of inadequacy. Primary care professionals should be aware of patient characteristics that may diminish the quality of their spirometry tests. Further research is needed to establish to what extent spirometry tests that are inadequate, according to stringent international expert criteria, result in incorrect clinical interpretations in general practice.

A methodology for evaluating test coverage criteria of high levelPetri nets

High level Petri nets have been extensively used for modeling concurrent systems; however, their strong expressive power reduces their ability to be easily analyzed. Currently there are few effective formal analysis techniques to support the validation of high level Petri nets. The executable nature of high level Petri nets means that during validation they can be analyzed using test criteria defined on the net model. Recently, theoretical test adequacy coverage criteria for concurrent systems using high level Petri nets have been proposed. However, determining the applicability of these test adequacy criteria has not yet been undertaken. In this paper, we present an approach for evaluating the proposed test adequacy criteria for high level Petri nets through experimentation. In our experiments we use the simulation functionality of the model checker SPIN to analyze various test coverage criteria on high level Petri nets.

Nonparametric Bootstrap Tests for Independence of Generalized Errors

In this paper, we develop a general method of testing for independence when unobservable generalized errors are involved. Our method can be applied to testing for serial independence of generalized errors, and testing for independence between the generalized errors and observable covariates. The former can serve as a unified approach to testing adequacy of time series models, as model adequacy often implies that the generalized errors obtained after a suitable transformation are independent and identically distributed. The latter is a key identification assumption in many nonlinear economic models. Our tests are based on a classical sample dependence measure, the Hoeffding-Blum-Kiefer-Rosenblat-type empirical process applied to generalized residuals. We establish a uniform expansion of the process, thereby deriving an explicit expression for the parameter estimation effect, which causes our tests not to be nuisance parameter-free. To circumvent this problem, we propose a multiplier-type bootstrap to approximate the limit distribution. Our bootstrap procedure is computationally very simple as it does not require a reestimation of the parameters in each bootstrap replication. In a simulation study, we apply our method to test the adequacy of ARMA-GARCH and Hansen (1994) skewed t models, and document a good finite sample performance of our test. Finally, an empirical application to daily exchange rate data highlights the merits of our approach.