Test Prioritization Research Articles

Seeing the invisible: test prioritization for object detection system

Seeing the invisible: test prioritization for object detection system

TPSQLi: Test Prioritization for SQL Injection Vulnerability Detection in Web Applications

The rapid proliferation of network applications has led to a significant increase in network attacks. According to the OWASP Top 10 Projects report released in 2021, injection attacks rank among the top three vulnerabilities in software projects. This growing threat landscape has increased the complexity and workload of software testing, necessitating advanced tools to support agile development cycles. This paper introduces a novel test prioritization method for SQL injection vulnerabilities to enhance testing efficiency. By leveraging previous test outcomes, our method adjusts defense strength vectors for subsequent tests, optimizing the testing workflow and tailoring defense mechanisms to specific software needs. This approach aims to improve the effectiveness and efficiency of vulnerability detection and mitigation through a flexible framework that incorporates dynamic adjustments and considers the temporal aspects of vulnerability exposure.

Compatibility of medication admixtures in continuous subcutaneous infusions: prioritizing laboratory testing for common combinations.

Continuous subcutaneous infusions (CSCIs) are indicated as an alternative therapy when the oral route is not viable. However, despite their widespread use in palliative care, the evidence for admixture compatibility remains a limitation. It is estimated that a significant number of admixtures used in practice are not supported by laboratory studies, which may lead to suboptimal clinical outcomes. The study aimed to determine the frequency of admixtures used in clinical practice without compatibility data generated by laboratory studies, and thereby identifying the most commonly prescribed admixtures that require laboratory data, which can help to guide the prioritization of future testing. This study was conducted across five palliative care services (three inpatients and two communities) in Victoria, Australia between May and July 2021. Electronic or paper medication charts of CSCIs were reviewed across all participating sites for all infusions administered. Data collected included medication combinations, dose, diluent, final volume, duration of infusion, reports of infusion-related reactions, and observed incompatibility. A total of 616 infusions containing two to three medications were assessed. Only 60% of these infusions were validated by laboratory data. Eleven most commonly prescribed admixtures with no laboratory compatibility data were identified over the 3-month period. Laboratory testing for the identified admixtures should be advocated to promote the safe and effective use of these medications.

DeepSense: test prioritization for neural network based on multiple mutation and manifold spatial distribution

DeepSense: test prioritization for neural network based on multiple mutation and manifold spatial distribution

Prioritizing test cases for deep learning-based video classifiers

The widespread adoption of video-based applications across various fields highlights their importance in modern software systems. However, in comparison to images or text, labelling video test cases for the purpose of assessing system accuracy can lead to increased expenses due to their temporal structure and larger volume. Test prioritization has emerged as a promising approach to mitigate the labeling cost, which prioritizes potentially misclassified test inputs so that such inputs can be identified earlier with limited time and manual labeling efforts. However, applying existing prioritization techniques to video test cases faces certain limitations: they do not account for the unique temporal information present in video data. Unlike static image datasets that only contain spatial information, video inputs consist of multiple frames that capture the dynamic changes of objects over time. In this paper, we propose VRank, the first test prioritization approach designed specifically for video test inputs. The fundamental idea behind VRank is that video-type tests with a higher probability of being misclassified by the evaluated DNN classifier are considered more likely to reveal faults and will be prioritized higher. To this end, we train a ranking model with the aim of predicting the probability of a given test input being misclassified by a DNN classifier. This prediction relies on four types of generated features: temporal features (TF), video embedding features (EF), prediction features (PF), and uncertainty features (UF). We rank all test inputs in the target test set based on their misclassification probabilities. Videos with a higher likelihood of being misclassified will be prioritized higher. We conducted an empirical evaluation to assess the performance of VRank, involving 120 subjects with both natural and noisy datasets. The experimental results reveal VRank outperforms all compared test prioritization methods, with an average improvement of 5.76%∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim $$\\end{document}46.51% on natural datasets and 4.26%∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim $$\\end{document}53.56% on noisy datasets.

Model‐Based Decision Support using Test and Evaluation: A Lightweight Architecture Approach

AbstractA standardized decision support tool to support test prioritization and decision‐making is a prerequisite to achieving on‐time delivery of weapon systems that are adequately tested and vetted by decision makers within distributed organizations such as the defense industrial base. However, to achieve standardization, the core principles of systems thinking and systems engineering must be utilized in order to realize a holistic and evolvable process and product that integrates Test & Evaluation (T&E) data and information with decision‐making in a consistent format to support program offices. To address this need, a model‐based reference architecture for the standardization of the Integrated Decision Support Key (IDSK) data and decision formats is presented in this paper. The IDSK is a DoD‐mandated decision support tool (artifact) for capturing a program's decisions and the T&E information necessary to support the decisions. The digital‐IDSK reference architecture proposed is developed using a model‐based systems engineering approach and facilitates the generation of standardized data and decision formats that are tailorable, adaptable to incremental future changes, and easy to integrate with other digital engineering artifacts all within a digital engineering ecosystem. In addition, this approach is also applicable not only to defense‐based organizations, but also finds utility within organizations pivoting to a digital engineering strategy for informing product T&E decision‐making.

Test Input Prioritization for 3D Point Clouds

3D point cloud applications have become increasingly prevalent in diverse domains, showcasing their efficacy in various software systems. However, testing such applications presents unique challenges due to the high-dimensional nature of 3D point cloud data and the vast number of possible test cases. Test input prioritization has emerged as a promising approach to enhance testing efficiency by prioritizing potentially misclassified test cases during the early stages of the testing process. Consequently, this enables the early labeling of critical inputs, leading to a reduction in the overall labeling cost. However, applying existing prioritization methods to 3D point cloud data is constrained by several factors: (1) inadequate consideration of crucial spatial information, and (2) susceptibility to noises inherent in 3D point cloud data. In this article, we propose PCPrior , the first test prioritization approach specifically designed for 3D point cloud test cases. The fundamental concept behind PCPrior is that test inputs closer to the decision boundary of the model are more likely to be predicted incorrectly. To capture the spatial relationship between a point cloud test and the decision boundary, we propose transforming each test (a point cloud) into a low-dimensional feature vector, toward indirectly revealing the underlying proximity between a test and the decision boundary. To achieve this, we carefully design a group of feature generation strategies, and for each test input, we generate four distinct types of features, namely spatial features, mutation features, prediction features, and uncertainty features. Through a concatenation of the four feature types, PCPrior assembles a final feature vector for each test. Subsequently, a ranking model is employed to estimate the probability of misclassification for each test based on its feature vector. Finally, PCPrior ranks all tests based on their misclassification probabilities. We conducted an extensive study based on 165 subjects to evaluate the performance of PCPrior, encompassing both natural and noisy datasets. The results demonstrate that PCPrior outperforms all of the compared test prioritization approaches, with an average improvement of 10.99% to 66.94% on natural datasets and 16.62% to 53% on noisy datasets.

Machine Learning Approach with Harmonized Multinational Datasets for Enhanced Prediction of Hypothyroidism in Patients with Type 2 Diabetes.

Type 2 diabetes (T2D) is a global health concern with increasing prevalence. Comorbid hypothyroidism (HT) exacerbates kidney, cardiac, neurological and other complications of T2D; these risks can be mitigated pharmacologically upon detecting HT. The current HT standard of care (SOC) screening in T2D is infrequent, delaying HT diagnosis and treatment. We present a first-to-date machine learning algorithm (MLA) clinical decision tool to classify patients as low vs. high risk for developing HT comorbid with T2D; the MLA was developed using readily available patient data from harmonized multinational datasets. The MLA was trained on data from NIH All of US (AoU) and UK Biobank (UKBB) (Combined dataset) and achieved a high negative predictive value (NPV) of 0.989 and an AUROC of 0.762 in the Combined dataset, exceeding AUROCs for the models trained on AoU or UKBB alone (0.666 and 0.622, respectively), indicating that increasing dataset diversity for MLA training improves performance. This high-NPV automated tool can supplement SOC screening and rule out T2D patients with low HT risk, allowing for the prioritization of lab-based testing for at-risk patients. Conversely, an MLA output that designates a patient to be at risk of developing HT allows for tailored clinical management and thereby promotes improved patient outcomes.

Artificial intelligence and scientific discovery: a model of prioritized search

We model a key step in the innovation process, hypothesis generation, as the making of predictions over a vast combinatorial space. Traditionally, scientists and innovators use theory or intuition to guide their search. Increasingly, however, they use artificial intelligence (AI) instead. We model innovation as resulting from sequential search over a combinatorial design space, where the prioritization of costly tests is achieved using a predictive model. The predictive model's ranked output is represented as a hazard function. Discrete survival analysis is used to obtain the main innovation outcomes of interest – the probability of innovation, expected search duration, and expected profit. We describe conditions under which shifting from the traditional method of hypothesis generation, using theory or intuition, to instead using AI that generates higher fidelity predictions, results in a higher likelihood of successful innovation, shorter search durations, and higher expected profits. We then explore the complementarity between hypothesis generation and hypothesis testing; potential gains from AI may not be realized without significant investment in testing capacity. We discuss the policy implications.

Potential health risk assessment of mixtures of heavy metals in drinking water

Despite the potential to cause chronic adverse effects, heavy metals screenings are not typically as frequent or exhaustive as those for bacterial contamination. Additionally, there is a pressing need for collectively assessing heavy metals as a mixture rather than individual allowed limits. We tested a hypothesis that comprehensive methodical wide screening for heavy metals can inform the magnitude of potential exposures and aid in classifying heavy metals collectively as hazardous rather than individual level comparisons. A total of 297 samples were taken by proportional cluster sampling from different households in Irbid governorate, depending on population density in each district. We analyzed the samples using Inductively Coupled Plasma-Mass Spectroscopy (ICP-MS). Our analysis looked at individual levels of heavy metal clustering in different water sources and regions and calculated a heavy metals pollution index (HPI) in five different types of drinking water: harvested rainwater, bottled water, spring water, water shop, and tap water. Pb and Fe were the main contaminants in drinking water, especially tap and harvested rainwater. High amounts of essential metals were found in tap, harvested rain, and spring waters compared to water shop and bottled water. HPI was above critical value in 29 samples, all of which were harvested rainwater samples, and most of which came from Wasateya and Taiba regions. Spring and tap water had similar heavy metal patterns. This experimental approach provides a guide for clustering water sources in their heavy metal patterns and for classification of heavy metals as mixtures rather than individual limits; it also fills crucial gaps in prioritization of heavy metal testing according to water type and metal correlations. Furthermore, this study presents the first data on drinking water quality in Jordan to be published in over a decade.

Regression test prioritization leveraging source code similarity with tree kernels

AbstractRegression test prioritization (RTP) is an active research field, aiming at re‐ordering the tests in a test suite to maximize the rate at which faults are detected. A number of RTP strategies have been proposed, leveraging different factors to reorder tests. Some techniques include an analysis of changed source code, to assign higher priority to tests stressing modified parts of the codebase. Still, most of these change‐based solutions focus on simple text‐level comparisons among versions. We believe that measuring source code changes in a more refined way, capable of discriminating between mere textual changes (e.g., renaming of a local variable) and more structural changes (e.g., changes in the control flow), could lead to significant benefits in RTP, under the assumption that major structural changes are also more likely to introduce faults. To this end, we propose two novel RTP techniques that leverage tree kernels (TK), a class of similarity functions largely used in Natural Language Processing on tree‐structured data. In particular, we apply TKs to abstract syntax trees of source code, to more precisely quantify the extent of structural changes in the source code, and prioritize tests accordingly. We assessed the effectiveness of the proposals by conducting an empirical study on five real‐world Java projects, also used in a number of RTP‐related papers. We automatically generated, for each considered pair of software versions (i.e., old version, new version) in the evolution of the involved projects, 100 variations with artificially injected faults, leading to over 5k different software evolution scenarios overall. We compared the proposed prioritization approaches against well‐known prioritization techniques, evaluating both their effectiveness and their execution times. Our findings show that leveraging more refined code change analysis techniques to quantify the extent of changes in source code can lead to relevant improvements in prioritization effectiveness, while typically introducing negligible overheads due to their execution.

Machine Learning for Test Case Prioritization in Continuous Integration: A Comprehensive Analysis

Machine Learning for Test Case Prioritization in Continuous Integration: A Comprehensive Analysis

Clinical prediction of GBA carrier status in Parkinson’s disease

IntroductionGiven the unique natural history of GBA-related Parkinson’s disease (GBA-PD) and the potential for novel treatments in this population, genetic testing prioritization for the identification of GBA-PD patients is crucial for prognostication, individualizing treatment, and stratification for clinical trials. Assessing the predictive value of certain clinical traits for the GBA-variant carrier status will help target genetic testing in clinical settings where cost and access limit its availability. MethodsIn-depth clinical characterization through standardized rating scales for motor and non-motor symptoms and self-reported binomial information of a cohort of subjects with PD (n = 100) from our center and from the larger cohort of the Parkinson’s Progression Marker Initiative (PPMI) was utilized to evaluate the predictive values of clinical traits for GBA variant carrier status. The model was cross-validated across the two cohorts. ResultsLeveraging non-motor symptoms of PD, we established successful discrimination of GBA variants in the PPMI cohort and study cohort (AUC 0.897 and 0.738, respectively). The PPMI cohort model successfully generalized to the study cohort data using both MDS-UPDRS scores and binomial data (AUC 0.740 and 0.734, respectively) while the study cohort model did not. ConclusionsWe assessed the predictive value of non-motor symptoms of PD for identifying GBA carrier status in the general PD population. These data can be used to determine a simple, clinically oriented model using either the MDS-UPDRS or subjective symptom reporting from patients. Our results can inform patient counseling about the expected carrier risk and test prioritization for the expected identification of GBA variants.

On the Effectiveness of Feature Selection Techniques in the Context of ML-Based Regression Test Prioritization

On the Effectiveness of Feature Selection Techniques in the Context of ML-Based Regression Test Prioritization

Unveiling the Tapestry of Learning: Exploring the Pedagogical Landscape for Transversal and Academic Competencies through the Lived Experiences of School Teachers and Students

This research investigates the pedagogical approaches employed for the development of transversal and academic competencies based on the lived experiences of students and teachers. Despite the presence of qualified educators in educational activities, schools face numerous challenges in fully realizing the potential of students. In the context of developing nations like Nepal, schools and teachers encounter unpreparedness in facilitating activities geared towards enhancing students' academic and transversal competencies. Despite teachers' expressed commitment in workshops to adopt student-centered methods, conventional lecture-based approaches persist due to the impact of high-stakes testing. The absence of clear guidelines and motivation further contributes to the prioritization of traditional testing over competency-based education. Furthermore, inadequate facilities and an unsupportive environment hinder the implementation of practice-based education in schools. Recognizing that students excel when engaged in enjoyable activities, it is imperative to align educational activities with learners' needs and interests while progressing towards predefined objectives. This study advocates for a blended management model, incorporating both top-down and bottom-up approaches, to encourage teachers in adopting student-centered methods, thereby playing a pivotal role in nurturing both academic and transversal competencies. Given that both teachers and students tend to prioritize aspects that align with assessment criteria, evaluating assessments covering both competencies become crucial for a comprehensive understanding of their educational progress.

GraphPrior: Mutation-based Test Input Prioritization for Graph Neural Networks

Graph Neural Networks (GNNs) have achieved promising performance in a variety of practical applications. Similar to traditional DNNs, GNNs could exhibit incorrect behavior that may lead to severe consequences, and thus testing is necessary and crucial. However, labeling all the test inputs for GNNs can be costly and time-consuming, especially when dealing with large and complex graphs, which seriously affects the efficiency of GNN testing. Existing studies have focused on test prioritization for DNNs, which aims to identify and prioritize fault-revealing tests (i.e., test inputs that are more likely to be misclassified) to detect system bugs earlier in a limited time. Although some DNN prioritization approaches have been demonstrated effective, there is a significant problem when applying them to GNNs: They do not take into account the connections (edges) between GNN test inputs (nodes), which play a significant role in GNN inference. In general, DNN test inputs are independent of each other, while GNN test inputs are usually represented as a graph with complex relationships between each test. In this article, we propose GraphPrior ( GNN -oriented Test Prior itization), a set of approaches to prioritize test inputs specifically for GNNs via mutation analysis. Inspired by mutation testing in traditional software engineering, in which test suites are evaluated based on the mutants they kill, GraphPrior generates mutated models for GNNs and regards test inputs that kill many mutated models as more likely to be misclassified. Then, GraphPrior leverages the mutation results in two ways, killing-based and feature-based methods. When scoring a test input, the killing-based method considers each mutated model equally important, while feature-based methods learn different importance for each mutated model through ranking models. Finally, GraphPrior ranks all the test inputs based on their scores. We conducted an extensive study based on 604 subjects to evaluate GraphPrior on both natural and adversarial test inputs. The results demonstrate that KMGP, the killing-based GraphPrior approach, outperforms the compared approaches in a majority of cases, with an average improvement of 4.76% ~49.60% in terms of APFD. Furthermore, the feature-based GraphPrior approach, RFGP, performs the best among all the GraphPrior approaches. On adversarial test inputs, RFGP outperforms the compared approaches across different adversarial attacks, with the average improvement of 2.95% ~46.69%.

Requirement based Test Case Prioritization for System Testing

System Testing encompasses a large number of test cases, which may not be able to get executed due to constrained time, budget and limitation of the resources. Therefore, the test cases must be prioritized in some order such that the critical and most required functionality can be tested early. In this paper, a hierarchical approach for system test case prioritization based on requirements has been proposed that maps requirements on the system test cases. This approach analyzes and assigns value to each requirement based on a comprehensive set of twelve factors thereby prioritizing the requirements. Further, the prioritized requirement is mapped on the highly relevant module and then prioritized set of test cases. To analyze the effectiveness of this approach, a case study of income tax calculator software [1] has been taken. The existing as well as the proposed approach were applied and analyzed on this software. The results show the efficacy of the proposed approach in terms of fault detection and severity early.

Component-Based Test Case Generation and Prioritization Using an Improved Genetic Algorithm

Developing test cases is the most challenging and crucial step in the software testing process. The initial test data must be optimized using a strong optimization technique due to many testing scenarios and poor testing effectiveness. Test prioritization is essential for testing the developed software products in a production line with a restricted budget in terms of time and money. A good understanding of the trade-off between costs (e.g. time and resources needed) and efficiency (e.g. component coverage) is necessary to prioritize test case scenarios for one or more software products. So, this paper proposes an efficient Multi-objective Test Case Generation and Prioritization using an Improved Genetic Algorithm (MTCGP-IGA) in Component-based Software Development (CSD). A random search-based method for creating and prioritizing multi-objective tests has been employed utilizing numerous cost and efficacy criteria. Specifically, the multi-objective optimization comprises maximizing the Prioritized Range of test cases (PR), Pairwise Coverage of Characteristics (PCC), Fault-Finding Capability (FFC), and minimizing Total Implementation Cost (TIC). For this test prioritizing problem, a unique fitness function is constructed with cost-effectiveness metrics. IGA is a robust search technique that exhibits excellent benefits and significant efficacy in resolving challenging issues, including ample space, multiple-peak, stochastic, and universal optimization. Relying on the use of IGA, this paper classifies, computes the objective function, introduces the Nondominated Sorting Genetic Algorithm-II (NSGA-II) method, evaluates each branch’s proximity on the handling route, and arranges the path set to get the best answer. The outcomes demonstrate that the proposed MTCGP-IGA with NSGA-II performed the best than other baseline algorithms in terms of prioritizing the test cases (mean value of 195.2), PCC (mean score of 0.7828), and FFC (mean score of 0.8136).

A data-derived reference mixture representative of European wastewater treatment plant effluents to complement mixture assessment

Aquatic environments are polluted with a multitude of organic micropollutants, which challenges risk assessment due the complexity and diversity of pollutant mixtures. The recognition that certain source-specific background pollution occurs ubiquitously in the aquatic environment might be one way forward to approach mixture risk assessment. To investigate this hypothesis, we prepared one typical and representative WWTP effluent mixture of organic micropollutants (EWERBmix) comprised of 81 compounds selected according to their high frequency of occurrence and toxic potential. Toxicological relevant effects of this reference mixture were measured in eight organism- and cell-based bioassays and compared with predicted mixture effects, which were calculated based on effect data of single chemicals retrieved from literature or different databases, and via quantitative structure-activity relationships (QSARs). The results show that the EWERBmix supports the identification of substances which should be considered in future monitoring efforts. It provides measures to estimate wastewater background concentrations in rivers under consideration of respective dilution factors, and to assess the extent of mixture risks to be expected from European WWTP effluents. The EWERBmix presents a reasonable proxy for regulatory authorities to develop and implement assessment approaches and regulatory measures to address mixture risks. The highlighted data gaps should be considered for prioritization of effect testing of most prevalent and relevant individual organic micropollutants of WWTP effluent background pollution. The here provided approach and EWERBmix are available for authorities and scientists for further investigations. The approach presented can furthermore serve as a roadmap guiding the development of archetypic background mixtures for other sources, geographical settings and chemical compounds, e.g. inorganic pollutants.

What are essential laparoscopic skills these days? Results of the SAGES Fundamentals of Laparoscopic Surgery (FLS) Committee technical skills survey.

The Fundamentals of Laparoscopic Surgery (FLS) program tests basic knowledge and skills required to perform laparoscopic surgery. Educational experiences in laparoscopic training and development of associated competencies have evolved since FLS inception, making it important to review the definition of fundamental laparoscopic skills. The Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) assigned an FLS Technical Skills Working Group to characterize technical skills used in basic laparoscopic surgery in current practice contexts and their possible application to future FLS tests. A group of subject matter experts defined an inventory of 65 laparoscopic skills using a Nominal Group Technique. From these, a survey was developed rating these items for importance, frequency of use, and priority for testing for FLS certification. This survey was distributed to SAGES members, recent recipients of FLS certification, and members of the Association of Program Directors in Surgery (APDS). Results were collected using a secure web-based survey platform. Complete data were available for 1742 surveys. Of these, 1143 comprised results for post-residency participants who performed advanced procedures. Seventeen competencies were identified for FLS testing prioritization by determining the proportion of respondents who identified them of highest priority, at median (50th percentile) of the maximum survey scale rating. These included basic peritoneal access, laparoscope and instrument use, tissue manipulation, and specific problem management skills. Sixteen could be used to show appropriateness of the domain construct by confirmatory factor analysis. Of these 8 could be characterized as manipulative tasks. Of these 5 mapped to current FLS tasks. This survey-identified competencies, some of which are currently assessed in FLS, with a high level of priority for testing. Further work is needed to determine if this should prompt consideration of changes or additions to the FLS technical skills test component.