Adaptive Testing Strategy Research Articles

Empowering software reliability: Leveraging efficient fault detection and removal efficiency

Advancements in science and technology have led to the widespread use of computer systems in various applications, emphasizing the importance of software reliability. Software failures can have severe consequences, making thorough testing crucial. Software reliability growth models (SRGMs) play a significant role in enhancing reliability by predicting improvement over time. This article introduces a comprehensive approach to software reliability that incorporates a dynamic fault detection rate, along with fault removal efficiency. The fault detection rate measures the rate at which faults are identified during testing, reflecting the effectiveness of the testing process. By incorporating this dynamic component, the model provides a more accurate estimation of software reliability and enables adaptive testing strategies and resource allocation. Achieving a high fault detection rate is desirable, but organizations must consider the cost implications and strike a balance between reliability and time-to-market constraints. This article extends the analysis to calculate the optimal release time and optimal warranty period that minimize development costs, subject to the desired reliability. By considering these factors, development teams can make informed decisions regarding the timing of software release and the duration of the warranty period, optimizing both reliability and cost.

Adaptive group testing strategy for infectious diseases using social contact graph partitions

Mass testing is essential for identifying infected individuals during an epidemic and allowing healthy individuals to return to normal social activities. However, testing capacity is often insufficient to meet global health needs, especially during newly emerging epidemics. Dorfman’s method, a classic group testing technique, helps reduce the number of tests required by pooling the samples of multiple individuals into a single sample for analysis. Dorfman’s method does not consider the time dynamics or limits on testing capacity involved in infection detection, and it assumes that individuals are infected independently, ignoring community correlations. To address these limitations, we present an adaptive group testing (AGT) strategy based on graph partitioning, which divides a physical contact network into subgraphs (groups of individuals) and assigns testing priorities based on the social contact characteristics of each subgraph. Our AGT aims to maximize the number of infected individuals detected and minimize the number of tests required. After each testing round (perhaps on a daily basis), the testing priority is increased for each neighboring group of known infected individuals. We also present an enhanced infectious disease transmission model that simulates the dynamic spread of a pathogen and evaluate our AGT strategy using the simulation results. When applied to 13 social contact networks, AGT demonstrates significant performance improvements compared to Dorfman’s method and its variations. Our AGT strategy requires fewer tests overall, reduces disease spread, and retains robustness under changes in group size, testing capacity, and other parameters. Testing plays a crucial role in containing and mitigating pandemics by identifying infected individuals and helping to prevent further transmission in families and communities. By identifying infected individuals and helping to prevent further transmission in families and communities, our AGT strategy can have significant implications for public health, providing guidance for policymakers trying to balance economic activity with the need to manage the spread of infection.

An adaptive testing strategy for efficient utilization of healthcare resources during an epidemic

Lockdowns are found to be effective against rapidly spreading epidemics like COVID-19. Two downsides to strategies rooted in social distancing and lockdowns are that they adversely affect the economy and prolong the duration of the epidemic. The extended duration observed in these strategies is often due to the under-utilization of medical facilities. Even though an under-utilized health care system is preferred over an overwhelmed one, an alternate strategy could be to maintain medical facilities close to their capacity, with a factor of safety. We explore the practicality of this alternate mitigation strategy and show that it can be achieved by varying the testing rate. We present an algorithm to calculate the number of tests per day to maintain medical facilities close to their capacity. We illustrate the efficacy of our strategy by showing that it reduced the epidemic duration by 40% in comparison to lockdown-based strategies.

A Diagnostic Tree Model for Adaptive Assessment of Complex Cognitive Processes Using Multidimensional Response Options

A tree model for diagnostic educational testing is described along with Monte Carlo simulations designed to evaluate measurement accuracy based on the model. The model is implemented in an assessment of inferential reading comprehension, the Multiple-Choice Online Causal Comprehension Assessment (MOCCA), through a sequential, multidimensional, computerized adaptive testing (CAT) strategy. Assessment of the first dimension, reading comprehension (RC), is based on the three-parameter logistic model. For diagnostic and intervention purposes, the second dimension, called process propensity (PP), is used to classify struggling students based on their pattern of incorrect responses. In the simulation studies, CAT item selection rules and stopping rules were varied to evaluate their effect on measurement accuracy along dimension RC and classification accuracy along dimension PP. For dimension RC, methods that improved accuracy tended to increase test length. For dimension PP, however, item selection and stopping rules increased classification accuracy without materially increasing test length. A small live-testing pilot study confirmed some of the findings of the simulation studies. Development of the assessment has been guided by psychometric theory, Monte Carlo simulation results, and a theory of instruction and diagnosis.

Noisy Adaptive Group Testing via Noisy Binary Search

The group testing problem consists of determining a small set of defective items from a larger set of items based on a number of possibly-noisy tests, and has numerous practical applications. One of the defining features of group testing is whether the tests are adaptive (i.e., a given test can be chosen based on all previous outcomes) or non-adaptive (i.e., all tests must be chosen in advance). In this paper, building on the success of binary splitting techniques in noiseless group testing (Hwang, 1972), we introduce noisy group testing algorithms that apply noisy binary search as a subroutine. We provide three variations of this approach with increasing complexity, culminating in an algorithm that succeeds using a number of tests that matches the best known previously (Scarlett, 2019), while overcoming fundamental practical limitations of the existing approach, and more precisely capturing the dependence of the number of tests on the error probability. We provide numerical experiments demonstrating that adaptive group testing strategies based on noisy binary search can be highly effective in practice, using significantly fewer tests compared to state-of-the-art non-adaptive strategies.

Revising testing of composite precursors – A new framework for data capture in complex multi-material systems

Any composite manufacturing method requires an application of a carefully designed consolidation process to ensure the suppression of voids in the laminate, establish bonding in laminate layers and prevent dimensional or fibre-path defects. The optimisation of consolidation processes relies on the characterisation of the composite precursors’ deformability. There are multiple mechanisms occurring in consolidation and various experimental programmes have been suggested in the literature to describe these mechanisms and deduce relevant material properties. The selection of a testing methodology often relies on an initial hypothesis or prior knowledge regarding the deformation modes. This may be a source of significant errors. This paper poses questions on the testing rationales, on subjectivity in material testing and on how data-rich programmes should be designed. Two approaches are suggested – the first one is a real-time adaptive testing strategy that enables a “conversation with the material” – flexible autonomous steering of a testing programme reacting on the obtained output. This framework focuses on the identification of the underlying physical mechanisms rather than material properties identification in a rightly or wrongly assumed flow mode. The second approach examines favourable combinations of tests to maximise information obtained whilst minimising the amount of testing. The obtained results highlight a way forward in terms of rethinking experiments for materials used in manufacturing and beyond.

Development and Validation of Computerized Adaptive Assessment Tools for the Measurement of Posttraumatic Stress Disorder Among US Military Veterans

Veterans from recent and past conflicts have high rates of posttraumatic stress disorder (PTSD). Adaptive testing strategies can increase accuracy of diagnostic screening and symptom severity measurement while decreasing patient and clinician burden. To develop and validate a computerized adaptive diagnostic (CAD) screener and computerized adaptive test (CAT) for PTSD symptom severity. A diagnostic study of measure development and validation was conducted at a Veterans Health Administration facility. A total of 713 US military veterans were included. The study was conducted from April 25, 2017, to November 10, 2019. The participants completed a PTSD-symptom questionnaire from the item bank and provided responses on the PTSD Checklist for Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) (PCL-5). A subsample of 304 participants were interviewed using the Clinician-Administered Scale for PTSD for DSM-5. Of the 713 participants, 585 were men; mean (SD) age was 52.8 (15.0) years. The CAD-PTSD reproduced the Clinician-Administered Scale for PTSD for DSM-5 PTSD diagnosis with high sensitivity and specificity as evidenced by an area under the curve of 0.91 (95% CI, 0.87-0.95). The CAT-PTSD demonstrated convergent validity with the PCL-5 (r = 0.88) and also tracked PTSD diagnosis (area under the curve = 0.85; 95% CI, 0.79-0.89). The CAT-PTSD reproduced the final 203-item bank score with a correlation of r = 0.95 with a mean of only 10 adaptively administered items, a 95% reduction in patient burden. Using a maximum of only 6 items, the CAD-PTSD developed in this study was shown to have excellent diagnostic screening accuracy. Similarly, using a mean of 10 items, the CAT-PTSD provided valid severity ratings with excellent convergent validity with an extant scale containing twice the number of items. The 10-item CAT-PTSD also outperformed the 20-item PCL-5 in terms of diagnostic accuracy. The results suggest that scalable, valid, and rapid PTSD diagnostic screening and severity measurement are possible.

Adaptive Test Case Allocation, Selection and Generation Using Coverage Spectrum and Operational Profile

We present an adaptive software testing strategy for test case allocation, selection and generation, based on the combined use of operational profile and coverage spectrum, aimed at achieving high delivered reliability of the program under test. Operational profile-based testing is a black-box technique considered well suited when reliability is a major concern, as it selects the test cases having the largest impact on failure probability in operation. Coverage spectrum is a characterization of a program's behavior in terms of the code entities (e.g., branches, statements, functions) that are covered as the program executes. The proposed strategy - named covrel+ - complements operational profile information with white-box coverage measures, so as to adaptively select/generate the most effective test cases for improving reliability as testing proceeds. We assess covrel+ through experiments with subjects commonly used in software testing research, comparing results with traditional operational testing. The results show that exploiting operational and coverage data in an integrated adaptive way allows generally to outperform operational testing at achieving a given reliability target, or at detecting faults under the same testing budget, and that covrel+ has greater ability than operational testing in detecting hard-to-detect faults.

Chip test pattern reordering method using adaptive test to reduce cost for testing of ICs

With the continuous drive toward integrated circuits (ICs) scaling, more test patterns are required in testing. However, the large number of patterns continues to increase test time and test costs. Thus, test costs of ICs are becoming more crucial yet more challenging. In this paper, we propose a novel adaptive test strategy to reduce test costs without increasing test escape, and using shortest path first (SPF) algorithm combined with K-Nearest Neighbor (KNN) to reorder the test patterns. The patterns which identify faults earlier are moved forward and applied first so as to save test time. In addition, the optimal test patterns are searched by means of a polynomial regression function and it provides a trade-off between test cost and test escape. Consequently, the optimization problem is converted into a mathematical function, in order to achieve broader applicability and generality. Experimental results demonstrate that the proposed method achieved 54.2 seconds time savings but leads to almost negligible test escapes increasing compared with traditional methods. The test pattern reordering method aims to find defects as early as possible. Furthermore, the proposed algorithm is completely software based and incurs no additional hardware overhead.

Enhanced multipath mitigation method based on multi-resolution CNR model and adaptive statistical test strategy for real-time kinematic PPP

This study proposes an enhanced multipath mitigation method based on multi-resolution carrier-to-noise-ratio (CNR) model and adaptive statistical test strategy for real-time kinematic precise point positioning (PPP) applications. The multi-resolution CNR model is established with GPS observation data collected from DOY 152 to 181 of 2019 by 230 globally distributed IGS stations, which used to analyze the relevant factors affecting CNR. Statistical results indicate that the CNR is not only related to the satellite elevation, but also closely related to the receiver types and specific satellite. The maximum difference between different receivers can reach 20 dB for the same satellite at the same elevation. In addition, the performance of the CNR is also obviously different between each satellite, and the maximum difference between different satellites is about 10 dB for the same receiver at the same elevation. Hence, in terms of the method which is based on CNR information for multipath detection and mitigation, the independence of receiver types, satellite and frequency must be considered. With the above analysis, this study developed a multi-resolution CNR model based on different receiver types, different satellites and different elevation firstly. Then, combined with the adaptive statistical test strategy which is based on the difference of CNR between inter-frequency and the difference of CNR between adjacent epochs, the multipath can be detected effectively. For the epoch which affected by multipath, the down-weighted strategy based on CNR is adopted to mitigate the influence of multipath on positioning. Real-time kinematic PPP data are collected to assess the proposed method, and the results demonstrate that the proposed method can detect the multipath effectively, and the detection rate can reach 90.28%. Moreover, after adopting the mitigation strategy, the RMS bias of the east, north and up components are improved about 19.95%, 17.89% and 23.07% compared to the original results, respectively. It is worth noting that this proposed method is also suitable for other GNSS, such as GLONASS and BDS, but the corresponding CNR model must be established simultaneously.

Rapid, large-scale, and effective detection of COVID-19 via non-adaptive testing

Pooling of samples can increase lab capacity when using Polymerase chain reaction (PCR) to detect diseases such as COVID-19. However, pool testing is typically performed via an adaptive testing strategy which requires a feedback loop in the lab and at least two PCR runs to confirm positive results. This can cost precious time. We discuss a non-adaptive testing method where each sample is distributed in a prescribed manner over several pools, and which yields reliable results after one round of testing. More precisely, assuming knowledge about the overall incidence rate, we calculate explicit error bounds on the number of false positives which scale favourably with pool size and sample multiplicity. This allows for hugely streamlined PCR testing and cuts in detection times for a large-scale testing scenario. A viable consequence of this method could be real-time screening of entire communities, frontline healthcare workers and international flight passengers, for example, using the PCR machines currently in operation.

Adaptive Minimax Testing for Circular Convolution

Given observations from a circular random variable contaminated by an additive measurement error, we consider the problem of minimax optimal goodness-of-fit testing in a non-asymptotic framework. We propose direct and indirect testing procedures using a projection approach. The structure of the optimal tests depends on regularity and ill-posedness parameters of the model, which are unknown in practice. Therefore, adaptive testing strategies that perform optimally over a wide range of regularity and ill-posedness classes simultaneously are investigated. Considering a multiple testing procedure, we obtain adaptive i.e. assumption-free procedures and analyse their performance. Compared with the non-adaptive tests, their radii of testing face a deterioration by a log-factor. We show that for testing of uniformity this loss is unavoidable by providing a lower bound. The results are illustrated considering Sobolev spaces and ordinary or super smooth error densities.

Software reliability assessment: a software cybernetics perspective

Software reliability assessment is a key step that determines whether the software can be delivered for the use or not. Because only limited testing resources are allowed for software reliability assessment, this paper proposes a software cybernetics perspective that introduces feedback and self-adaptive control to guide the software testing process in the software reliability assessment. Also, an adaptive testing strategy based on gradient descent is proposed for software reliability assessment. The theoretical and experimental results show that reliability assessment methods based on software cybernetics can achieve highly precise and stable software-reliability-assessment results using restricted testing resources.

Synthesizing adaptive test strategies from temporal logic specifications

Constructing good test cases is difficult and time-consuming, especially if the system under test is still under development and its exact behavior is not yet fixed. We propose a new approach to compute test strategies for reactive systems from a given temporal logic specification using formal methods. The computed strategies are guaranteed to reveal certain simple faults in every realization of the specification and for every behavior of the uncontrollable part of the system’s environment. The proposed approach supports different assumptions on occurrences of faults (ranging from a single transient fault to a persistent fault) and by default aims at unveiling the weakest one. We argue that such tests are also sensitive for more complex bugs. Since the specification may not define the system behavior completely, we use reactive synthesis algorithms with partial information. The computed strategies are adaptive test strategies that react to behavior at runtime. We work out the underlying theory of adaptive test strategy synthesis and present experiments for a safety-critical component of a real-world satellite system. We demonstrate that our approach can be applied to industrial specifications and that the synthesized test strategies are capable of detecting bugs that are hard to detect with random testing.

Dynamic Random Testing: Technique and Experimental Evaluation

A particularly good software testing strategy is to achieve the underlying testing goal while solving the problems of tradeoffs between testing effectiveness and efficiency. To improve the fault detection effectiveness of software testing, the principle of feedback control theory was adopted, which motivated the proposal of dynamic random testing (DRT). The main idea behind DRT is using the testing results to guide the test case selection to increase the selection probabilities of the subdomains with higher fault detection rates. Previous works show that DRT strategy can achieve better effectiveness than random testing strategy and random partition testing strategy, and has significantly lower computational costs than adaptive testing strategy. However, the essential factors that affect the performance of DRT, i.e., adjusting parameters, initial profile, and test case classification have not been thoroughly investigated. Besides, some experimental assumptions are inconsistent with real scenarios. Therefore, this paper gives a series of investigations on DRT with a set of practical subject programs. More specifically, the effectiveness and efficiency of DRT are presented, and the extended experiments on DRT with relevant factors are conducted. The results indicate that the effectiveness of DRT is robust to different initial profiles and affected noticeably by the adjusting parameter settings and test case classification methods.

Neuropsychological tests of the future: How do we get there from here?

Objective: This article reviews current approaches to neuropsychological assessment, identifies opportunities for development of new methods using modern psychometric theory and advances in technology, and suggests a transition path that promotes application of novel methods without sacrificing validity. Methods: Theoretical/state-of-the-art review.Conclusions: Clinical neuropsychological assessment today does not reflect advances in neuroscience, modern psychometrics, or technology. Major opportunities for improving practice include both psychometric and technological strategies. Modern psychometric approaches including item response theory (IRT) enable linking procedures that can place different measures on common scales; adaptive testing algorithms that can dramatically increase efficiency of assessment; examination of differential item functioning (DIF) to detect measures that behave differently in different groups; and person fit statistics to detect aberrant patterns of responding of high value for performance validity testing. Opportunities to introduce novel technologies include computerized adaptive testing, Web-based assessment, healthcare- and bio-informatics strategies, mobile platforms, wearables, and the ‘internet-of-things’. To overcome inertia in current practices, new methods must satisfy requirements for back-compatibility with legacy instrumentation, enabling us to leverage the wealth of validity data already accrued for classic procedures. A path to achieve these goals involves creation of a global network to aggregate item-level data into a shared repository that will enable modern psychometric analyses to refine existing methods, and serve as a platform to evolve novel assessment strategies, which over time can revolutionize neuropsychological assessment practices world-wide.

Assessing Screening Guidelines for Cardiovascular Disease Risk Factors using Routinely Collected Data

This study investigates if laboratory data can be used to assess whether physician-retesting patterns are in line with established guidelines, and if these guidelines identify deteriorating patients in a timely manner. A total of 7594 patients with high cholesterol were studied, along with 2764 patients with diabetes. More than 90% of borderline high cholesterol patients are retested within the 3 year recommended period, however less than 75% of pre-diabetic patients have repeated tests within the suggested 1-year time frame. Patients with borderline high cholesterol typically progress to full high cholesterol in 2–3 years, and pre-diabetic patients progress to full diabetes in 1–2 years. Data from routinely ordered laboratory tests can be used to monitor adherence to clinical guidelines. These data may also be useful in the design of adaptive testing strategies that reduce unnecessary testing, while ensuring that patient deterioration is identified in a timely manner. Established guidelines for testing of total serum cholesterol for hypercholesterolemia are appropriate and are well-adhered to, whereas guidelines for glycated hemoglobin A1c testing for type 2 diabetes mellitus could be improved to bring them in line with current practice and avoid unnecessary testing.

Auto-adaptive Alpha Allocation: A Strategy to Mitigate Risk on Study Assumptions

In some clinical development programs, there are potential biomarkers with promising but uncertain predictive effect, while the probability of success in the overall population cannot be readily dismissed. It is risky to focus only on the overall population, or just the biomarker subpopulation. In 2009, Chen and Beckman proposed a Bayesian decision framework to optimize the type I error rate (alpha) allocation in a Phase III clinical study with possible predictive subset effect. The utilization of internal data in this framework is of particular interest because it provides an opportunity to mitigate the potential risk of misspecified study assumptions using an auto-adaptive strategy. In this paper, we examine this auto-adaptive strategy in detail through extensive numerical case studies and provide guidance on the appropriate use of partial current trial (internal) data in this data-driven optimization framework. We show that internal data can be used to inform the alpha allocation to hypothesis testing in the overall population and the subgroup. The resulting adaptive testing strategy is robust with respect to the uncertainty in the predictive subgroup effect and biomarker prevalence.

Distinguishing Sequences for Distributed Testing: Preset Distinguishing Sequences

This paper concerns the problem of testing from a finite state machine (FSM) $M$ modelling a system that interacts with its environment at multiple physically distributed interfaces, called ports. We assume that the distributed test architecture is used: there is a local tester at each port, the tester at port $p$ only observes events at $p$ and the testers do not interact during testing. This paper formalizes the notion of an adaptive test strategy and what it means for an adaptive test strategy to be controllable. We provide algorithms to check whether a global strategy is controllable and to generate a controllable adaptive distinguishing sequence (ADS). We prove that controllable ADS existence is PSPACE-Hard and that the problem of deciding whether $M$ has a controllable ADS with length $\ell $ is NP-Hard. In practice, there is likely to be a polynomial upper bound on the length of ADS in which we are interested and for this case the decision problem is NP-Complete.

Distinguishing Sequences for Distributed Testing: Adaptive Distinguishing Sequences

This paper concerns the problem of testing from a finite state machine (FSM) | $M$ | modelling a system that interacts with its environment at multiple physically distributed interfaces, called ports. We assume that the distributed test architecture is used: there is a local tester at each port, the tester at port | $p$ | only observes events at | $p$ | and the testers do not interact during testing. This paper formalizes the notion of an adaptive test strategy and what it means for an adaptive test strategy to be controllable. We provide algorithms to check whether a global strategy is controllable and to generate a controllable adaptive distinguishing sequence (ADS). We prove that controllable ADS existence is PSPACE-Hard and that the problem of deciding whether | $M$ | has a controllable ADS with length | $\ell $ | is NP-Hard . In practice, there is likely to be a polynomial upper bound on the length of ADS in which we are interested and for this case the decision problem is NP-Complete .