Adaptive Testing Strategy Research Articles

Bayesian integrated testing strategy (ITS) for skin sensitization potency assessment: a decision support system for quantitative weight of evidence and adaptive testing strategy.

The presented Bayesian network Integrated Testing Strategy (ITS-3) for skin sensitization potency assessment is a decision support system for a risk assessor that provides quantitative weight of evidence, leading to a mechanistically interpretable potency hypothesis, and formulates adaptive testing strategy for a chemical. The system was constructed with an aim to improve precision and accuracy for predicting LLNA potency beyond ITS-2 (Jaworska et al., J Appl Toxicol 33(11):1353-1364, 2013) by improving representation of chemistry and biology. Among novel elements are corrections for bioavailability both in vivo and in vitro as well as consideration of the individual assays' applicability domains in the prediction process. In ITS-3 structure, three validated alternative assays, DPRA, KeratinoSens and h-CLAT, represent first three key events of the adverse outcome pathway for skin sensitization. The skin sensitization potency prediction is provided as a probability distribution over four potency classes. The probability distribution is converted to Bayes factors to: 1) remove prediction bias introduced by the training set potency distribution and 2) express uncertainty in a quantitative manner, allowing transparent and consistent criteria to accept a prediction. The novel ITS-3 database includes 207 chemicals with a full set of in vivo and in vitro data. The accuracy for predicting LLNA outcomes on the external test set (n = 60) was as follows: hazard (two classes)-100 %, GHS potency classification (three classes)-96 %, potency (four classes)-89 %. This work demonstrates that skin sensitization potency prediction based on data from three key events, and often less, is possible, reliable over broad chemical classes and ready for practical applications.

Estimating confidence interval of software reliability with adaptive testing strategy

Software reliability assessment is a critical problem in safety-critical and mission-critical systems. In the reliability assessment of such a system, both an accurate reliability estimate and a tight confidence interval are required. Adaptive testing (AT) is an on-line testing framework, which dynamically selects test cases from different subdomains to achieve some optimization object. Although AT has been proved effective in minimizing reliability estimator variance, its performance on providing the corresponding confidence interval has not been investigated. In order to address this issue, an AT strategy combined with Bayesian inference (AT-BI) is proposed in this study. The novel AT-BI strategy is expected to be effective in providing both a low-variance estimator and a tight confidence interval. Experiments are set up to validate the effectiveness of the AT-BI strategy.

Bilgisayar ortamında bireye uyarlanmış test stratejilerinin karşılaştırılması

Bu arastirmada, bilgisayar ortaminda bireye uyarlanmis test (BOBUT) yontemi ile geleneksel kâgit kalem test yonteminin karsilastirilmasi ve BOBUT yontemine iliskin farkli stratejilerin karsilastirilmasi amaclanmistir. Temel arastirma modelindeki arastirmanin verileri, Ankara Universitesi Yabanci Diller Yuksekokulu bunyesinde, hazirlik sinifinda ogrenim gormekte olan toplam 1166 ogrenciden toplanmistir. Farkli stratejilerin karsilastirilmasi amaciyla R programi ile simulatif veriler de olusturulmustur. Arastirmada veri toplama araci olarak Ingilizce Kelime Testi kullanilmistir. BOBUT uygulamalarinin yapilabilmesi icin arastirmaci tarafindan bir cevrimici ortam gelistirilmistir. Arastirma sonucunda, cevrimici ortam kullanilarak yapilan bireye uyarlanmis test uygulamasinda, kâgit kalem testine gore madde sayilarinda buyuk oranda tasarruf saglandigi saptanmistir. Bireye uyarlanmis ve kâgit kalem test uygulamalarindan elde edilen yetenek parametreleri arasinda da pozitif yonde yuksek korelâsyon katsayilari bulunmustur. Farkli stratejiler ve kagit kalem testinden elde edilen yetenek parametreleri arasinda pozitif yonde yuksek korelasyon katsayilari bulunarak, arastirma kapsaminda ele alinan 18 farkli strateji ile kagit kalem testlerinin cok benzer yetenek parametrelerinin kestirildigi ortaya konmustur. Ayni sekilde farkli stratejiler ile kestirilen yetenek parametrelerinin kendi aralarinda pozitif yonde yuksek korelasyon katsayilari elde edilmistir. Sonlandirma kurallari dikkate alindiginda, farkli stratejilerden elde edilen yetenek kestirimlerinin gerek kagit kalem testinden elde edilen yetenek parametreleri ile arasinda gerekse kendi aralarinda en dusuk korelasyon katsayilarinin sonlandirma kurali olarak standart hatanin 0.50’den kucuk olmasi durumunda elde edildigi saptanmistir. ML, EAP ve MAP yetenek kestirim yontemlerinden kaynakli, kestirilen yetenek parametrelerinde farklilik olmadigi gorulmustur

On the Asymptotic Behavior of Adaptive Testing Strategy for Software Reliability Assessment

In software reliability assessment, one problem of interest is how to minimize the variance of reliability estimator, which is often considered as an optimization goal. The basic idea is that an estimator with lower variance makes the estimates more predictable and accurate. Adaptive Testing (AT) is an online testing strategy, which can be adopted to minimize the variance of software reliability estimator. In order to reduce the computational overhead of decision-making, the implemented AT strategy in practice deviates from its theoretical design that guarantees AT's local optimality. This work aims to investigate the asymptotic behavior of AT to improve its global performance without losing the local optimality. To this end, a new AT strategy named Adaptive Testing with Gradient Descent method (AT-GD) is proposed. Theoretical analysis indicates that AT-GD, a locally optimal testing strategy, converges to the globally optimal solution as the assessment process proceeds. Simulation and experiments are set up to validate AT-GD's effectiveness and efficiency. Besides, sensitivity analysis of AT-GD is also conducted in this study.

Software engineering and their conflicts in development of software

In software engineering there are many different uses that will be mainly helpful in the development of suitable software that is required by the particular user in order to achieve the reliability of the user. So natively we used the technique called we use an adaptive testing strategy for testing software components. This strategy (AT_RLSEc with c indicating components) applies a recursive least squares estimation (RLSE) method to estimate parameters such as failure detection rate. It is different from the genetic algorithm-based adaptive testing (AT_GA) where a genetic algorithm is used for parameter estimation. By using this method we will not be getting the desired result so there will be some defects in the software. To overcome these defects we use qualitative and quantitative approaches, the points until the separately generated results are interpreted. It will show efficiency of the conflicts and risks that are presented that are occurred by using the existing approach. So t will be helping the developers to give perfect software that is required by the user.

A Python test suite for statistical properties of probabilistic networks with and without spatial structure

A Python test suite for statistical properties of probabilistic networks with and without spatial structure

Enhancing software reliability estimates using modified adaptive testing

ContextMost software reliability models are based on a binary notion of correctness, i.e. “successful” or “failed.” However, in several instances, it is important to account of failure severity to obtain more descriptive and accurate estimates of the reliability of the software. ObjectiveIn this paper, we develop a set of extended metrics based on the Nelson’s software reliability model to account for information gained from a user’s point of view regarding the severity of the observed failures. Model formulation based on multi-granularity failure severity is provided, and the proposed metrics are proved to be backward compatible. MethodIn order to estimate the software reliability through testing, an extended adaptive testing strategy, namely Modified Adaptive Testing (MAT) is proposed. The use of test history information allows the resulting test process to be adaptive in the selection of tests under limited test budget. Simulations and experiments on real-life programs are conducted to evaluate the effectiveness of MAT. ResultsData show that the reliability estimates obtained using MAT (a) are closer to the “true” reliability than those obtained using random testing and (b) lead to lower variance than the techniques used for comparison, which means MAT can be applied to help testers and reliability engineers better understand the reliability of their programs. ConclusionIt is concluded that the proposed approach can enhance the software reliability estimation testing by guiding the test case selection process by providing more descriptive and accurate results.

Test Application for Analog/RF Circuits With Low Computational Burden

In this paper, we propose an adaptive test strategy that tailors the test sequence with respect to the properties of each individual instance of a circuit. Reducing the test set by analyzing the dropout patterns during characterization and eliminating the unnecessary tests has always been the approach for high volume production in the analog domain. However, once determined, the test set remains typically fixed for all devices. We propose to exploit the statistical diversity of the manufactured devices and adaptively eliminate tests that are determined to be unnecessary based on information obtained on the circuit under test. Test time information is incorporated in the method to yield short test time. The proposed methodology is computationally efficient and imposes very little overhead on the tester. We compare our results with other similar specification-based test reduction techniques for a low noise amplifier (LNA) circuit and an analog industrial circuit. Results show 85% test quality improvement for the same test time or 24% test time reduction for the same test quality for the LNA circuit. Moreover, near zero defective parts per million is achieved for the industrial circuit.

Rank-Based Analyses of Stratified Experiments: Alternatives to the van Elteren Test

The van Elteren test, a popular extension of the Wilcoxon rank sum test for comparing two treatments in a stratified experiment, uses within-stratum ranks and assigns larger weight to rank sums from smaller strata. It performs well if the treatment effect is constant across strata, but it can be grossly inefficient otherwise. Existing alternatives to the van Elteren test include tests that use overall ranks and/or equal stratum weights, and the aligned rank test of Hodges and Lehmann. We develop two new testing procedures. The first replaces the stratum weights in the van Elteren test with corresponding weights based on estimated treatment effects and sample sizes. The second uses the aligned rank test and a version of the van Elteren test with equal stratum weights in a sequential manner via an adaptive multiple testing strategy. The new and existing tests are compared to the van Elteren test via extensive simulations. The results show that the aligned rank test and the proposed adaptive testing procedure convincingly outperform the van Elteren test in terms of statistical power; the latter is recommended as a default based on overall performance. Data from two clinical trials are used for illustration.

AN IMPROVED APPROACH TO ADAPTIVE TESTING

Adaptive testing is the counterpart of adaptive control in software testing. It means that software testing strategy should be adjusted on-line by using the testing data collected during software testing as our understanding of the software under test is improved. Previous studies on adaptive testing involved a simplified Controlled Markov Chain (CMC) model for software testing which employs several unrealistic assumptions. In this paper we propose a new adaptive software testing approach in the context of an improved and namely, general CMC model which aims to eliminate such threats to validity. A set of more realistic basic assumptions on the software testing process is proposed and several unrealistic assumptions are replaced by less unrealistic assumptions. A new adaptive testing strategy based on the general CMC is developed and implemented. Mathematical simulations and experiments on real life software are conducted to demonstrate the effectiveness of the new strategy.

ON THE ONLINE PARAMETER ESTIMATION PROBLEM IN ADAPTIVE SOFTWARE TESTING

Software cybernetics is an emerging area that explores the interplay between software and control. The controlled Markov chain (CMC) approach to software testing supports the idea of software cybernetics by treating software testing as a control problem, where the software under test serves as a controlled object modeled by a controlled Markov chain and the software testing strategy serves as the corresponding controller. The software under test and the corresponding software testing strategy form a closed-loop feedback control system. The theory of controlled Markov chains is used to design and optimize the testing strategy in accordance with the testing/reliability goal given explicitly and a priori. Adaptive software testing adjusts and improves software testing strategy online by using the testing data collected in the course of software testing. In doing so, the online parameter estimations play a key role. In this paper, we study the effects of genetic algorithm and the gradient method for doing online parameter estimation in adaptive software testing. We find that genetic algorithm is effective and does not require prior knowledge of the software parameters of concern. Although genetic algorithm is computationally intensive, it leads the adaptive software testing strategy to an optimal software testing strategy that is determined by optimizing a given testing goal, such as minimizing the total cost incurred for removing a given number of defects. On the other hand, the gradient method is computationally favorable, but requires appropriate initial values of the software parameters of concern. It may lead, or fail to lead, the adaptive software testing strategy to an optimal software testing strategy, depending on whether the given initial parameter values are appropriate or not. In general, the genetic algorithm should be used instead of the gradient method in adaptive software testing. Simulation results show that adaptive software testing does work and outperforms random testing.

An experimental study of adaptive testing for software reliability assessment

Adaptive testing is a new form of software testing that is based on the feedback and adaptive control principle and can be treated as the software testing counterpart of adaptive control. Our previous work has shown that adaptive testing can be formulated and guided in theory to minimize the variance of an unbiased software reliability estimator and to achieve optimal software reliability assessment. In this paper, we present an experimental study of adaptive testing for software reliability assessment, where the adaptive testing strategy, the random testing strategy and the operational profile based testing strategy were applied to the Space program in four experiments. The experimental results demonstrate that the adaptive testing strategy can really work in practice and may noticeably outperform the other two. Therefore, the adaptive testing strategy can serve as a preferable alternative to the random testing strategy and the operational profile based testing strategy if high confidence in the reliability estimates is required or the real-world operational profile of the software under test cannot be accurately identified.

Adaptive software testing with fixed-memory feedback

Adaptive software testing is the counterpart of adaptive control in software testing. It means that software testing strategy should be adjusted on-line by using the testing data collected during software testing as our understanding of the software under test is improved. In this paper we propose a new strategy of adaptive software testing in the context of software cybernetics. This new strategy employs fixed-memory feedback for on-line parameter estimations and is intended to circumvent the drawbacks of the assumption that all remaining defects are equally detectable at constant rate and to reduce the underlying computational complexity of on-line parameter estimations. A comprehensive case study with the Space program demonstrates that the new adaptive testing strategy can really work in practice and may noticeably outperform the purely-random testing strategy and the random-partition testing strategy (or collectively, the random testing strategies) in terms of the number of tests used to detect and remove a given number of defects in a single process of software testing and the corresponding standard deviation. In addition, the case study shows that the input domain of the software under test should be partitioned non-evenly for the adaptive testing strategy.

RESTRICTED RANDOM TESTING: ADAPTIVE RANDOM TESTING BY EXCLUSION

Restricted Random Testing (RRT) is a new method of testing software that improves upon traditional Random Testing (RT) techniques. Research has indicated that failure patterns (portions of an input domain which, when executed, cause the program to fail or reveal an error) can influence the effectiveness of testing strategies. For certain types of failure patterns, it has been found that a widespread and even distribution of test cases in the input domain can be significantly more effective at detecting failure compared with ordinary RT. Testing methods based on RT, but which aim to achieve even and widespread distributions, have been called Adaptive Random Testing (ART) strategies. One implementation of ART is RRT. RRT uses exclusion zones around executed, but non-failure-causing, test cases to restrict the regions of the input domain from which subsequent test cases may be drawn. In this paper, we introduce the motivation behind RRT, explain the algorithm and detail some empirical analyses carried out to examine the effectiveness of the method. Two versions of RRT are presented: Ordinary RRT (ORRT) and Normalized RRT (NRRT). The two versions share the same fundamental algorithm, but differ in their treatment of non-homogeneous input domains. Investigations into the use of alternative exclusion shapes are outlined, and a simple technique for reducing the computational overheads of RRT, prompted by the alternative exclusion shape investigations, is also explained. The performance of RRT is compared with RT and another ART method based on maximized minimum test case separation (DART), showing excellent improvement over RT and a very favorable comparison with DART.

Adaptive Statistical Analysis Following Sample Size Modification Based on Interim Review of Effect Size

ABSTRACT In designing a comparative clinical trial, the required sample size is a function of the effect size, the value of which is unknown and at best may be estimated from historical data. Insufficiency in sample size as a result of overestimating the effect size can be destructive to the success of the clinical trial. Sample size re-estimation may need to be properly considered as a part of clinical trial planning. This paper is intended to give the motivations for the sample size re-estimation based partly on the effect size observed at an interim analysis and for a resulting simple adaptive test strategy. The performance of this adaptive design strategy is assessed by comparing it with a fixed maximum sample size design that is properly adjusted in anticipation of the possible sample size adjustment.

Optimal software testing in the setting of controlled Markov chains

The controlled Markov chains (CMC) approach to software testing treats software testing as a control problem, where the software under test serves as a controlled object that is modeled as controlled Markov chain, and the software testing strategy serves as the corresponding controller. In this paper we extend the CMC approach to software testing to the case that the number of tests that can be applied to the software under test is limited. The optimal testing strategy is derived if the true values of all the software parameters of concern are known a priori. An adaptive testing strategy is employed if the true values of the software parameters of concern are not known a priori and need to be estimated on-line during software testing by using testing data. A random testing strategy ignores all the related information (true values or estimates) of the software parameters of concern and follows a uniform probability distribution to select a possible test case. Simulation results show that the performance of an adaptive testing strategy cannot compete that of the optimal testing strategy, but should be better than that of a random testing strategy. This paper further justifies the idea of software cybernetics that is aimed to explore the interplay between software theory/engineering and control theory/engineering.

Optimal and adaptive testing for software reliability assessment

Optimal software testing is concerned with how to test software such that the underlying testing goal is achieved in an optimal manner. Our previous work shows that the optimal testing problem for software reliability growth can be treated as closed-loop or feedback control problem, where the software under test serves as a controlled object and the software testing strategy serves as the corresponding controller. More specifically, the software under test is modeled as controlled Markov chains (CMCs) and the control theory of Markov chains is used to synthesize the required optimal testing strategy. In this paper, we show that software reliability assessment can be treated as a feedback control problem and the CMC approach is also applicable to dealing with the optimal testing problem for software reliability assessment. In this problem, the code of the software under test is frozen and the software testing process is optimized in the sense that the variance of the software reliability estimator is minimized. An adaptive software testing strategy is proposed that uses the testing data collected on-line to estimate the required parameters and selects next test cases. Simulation results show that the proposed adaptive software testing strategy can really work in the sense that the resulting variance of the software reliability estimate is much smaller than that resulting from the random testing strategies. The work presented in this paper is a contribution to the new area of software cybernetics that explores the interplay between software and control.

Group sequential test strategies for superiority and non‐inferiority hypotheses in active controlled clinical trials

In a group sequential active controlled clinical trial, the study hypothesis may be a superiority hypothesis that an experimental treatment is more effective than the active control therapy or a non-inferiority hypothesis that the treatment is no worse than the active control within some non-inferiority range. When it is necessary to plan for testing the superiority and the non-inferiority hypotheses, we propose an adaptive group sequential closed test strategy by which the sample size is planned for testing superiority and is to be increased for showing non-inferiority given that it is deemed more plausible than superiority based on the observed sample path during the course of the trial. The proposed adaptive test strategy is valid in terms of having the type I error probability maintained at the targeted alpha level for both superiority and non-inferiority. It has power advantage or sample size saving over the traditional group sequential test designed for testing either superiority only or non-inferiority only.

Computerized adaptive personality assessment: an illustration with the Absorption scale.

This article introduces the theory behind and applications of adaptive personality assessment based on the item response theory. Two adaptive testing strategies were compared: (a) fixed test length and (b) clinical decision. Real-data simulations, based on the item responses from 1,000 subjects who had previously taken the 34-item Absorption scale (Tellegen, 1982) by means of paper-and-pencil format, were used to illustrate these strategies. Results suggest that computerized adaptive personality assessment works impressively well. With the fixed-test-length strategy, a 50% savings in administered items was achieved with little loss of measurement precision. In the clinical-decision testing strategy, individuals who were extreme on the Absorption trait were identified with perfect accuracy using, on average, 25% of the available items. The implications of these results for personality research and assessment are discussed.

Bias and Information of Bayesian Adaptive Testing

Monte carlo simulation was used to investigate score bias and information characteristics of Owen's Bayesian adaptive testing strategy and to examine pos sible causes of score bias. Factors investigated in three related studies included effects of an accurate prior θ estimate, effects of item discrimination, and effects of fixed versus variable test length. Data were generated from a three-parameter logistic model for 3,100 simu lees in each of eight data sets, and Bayesian adaptive tests were administered, drawing items from a "per fect" item pool. Results showed that the Bayesian adaptive test yielded unbiased θ estimates and rela tively flat information functions only in the situation in which an accurate prior θ estimate was used. When a constant prior θ estimate was used with a fixed test length, severe bias was observed that varied with item discrimination. A different pattern of bias was ob served with variable test length and a constant prior. Information curves for the constant prior conditions generally became more peaked and asymmetric with increasing item discrimination. In the variable test length condition, the test length required to achieve a specified level of the posterior variance of θ estimates was an increasing function of θ level. These results indicate that θ estimates from Owen's Bayesian adap tive testing method are affected by the prior θ estimate used and that the method does not provide measure ments that are unbiased and equiprecise except when an accurate prior θ estimate is used.