Allocation Of Testing Resources Research Articles

Robust Resource Allocation for Calibration and Validation Tests

Model calibration and validation are two activities in system model development, and both of them make use of test data. Limited testing budget creates the challenge of test resource allocation, i.e., how to optimize the number of calibration and validation tests to be conducted. Test resource allocation is conducted before any actual test is performed, and therefore needs to use synthetic data. This paper develops a test resource allocation methodology to make the system response prediction “robust” to test outcome, i.e., insensitive to the variability in test outcome; therefore, consistent system response predictions can be achieved under different test outcomes. This paper analyzes the uncertainty sources in the generation of synthetic data regarding different test conditions, and concludes that the robustness objective can be achieved if the contribution of model parameter uncertainty in the synthetic data can be maximized. Global sensitivity analysis (Sobol’ index) is used to assess this contribution, and to formulate an optimization problem to achieve the desired consistent system response prediction. A simulated annealing algorithm is applied to solve this optimization problem. The proposed method is suitable either when only model calibration tests are considered or when both calibration and validation tests are considered. Two numerical examples are provided to demonstrate the proposed approach.

An Adaptive EM Algorithm for the Maximum Likelihood Estimation of Non-Homogeneous Poisson Process Software Reliability Growth Models

Non-homogeneous Poisson process (NHPP) software reliability growth models (SRGMa) enable quantitative metrics to guide decisions during the software engineering life cycle, including test resource allocation and release planning. However, many SRGM possess complex mathematical forms that make them difficult to apply. Specifically, traditional procedures solve a system of nonlinear equations to identify the numerical parameters that best characterize failure data. Recently, researchers have developed expectation-maximization (EM) algorithms for NHPP SRGM that exhibit better convergence properties and can therefore find maximum likelihood estimates with greater ease.This paper presents an adaptive EM (AEM) algorithm, which combines an earlier EM algorithm for NHPP SRGM with unconstrained search of the model parameter space. Our performance analysis shows that the AEM outperforms state-of-the-art EM algorithms for NHPP SRGM with very strong statistical significance, which is as much as hundreds of times faster on some data sets. Thus, the approach can fit SRGM very quickly. We also incorporate this high performance adaptive EM algorithm into a heuristic nested model selection procedure to objectively select a model of least complexity that best characterizes the failure data. Results indicate this heuristic approach often identifies the model possessing the best model selection criteria.aAcronyms are not pluralized.

Physiological conditions of parent and offspring Black-browed Albatrosses Thalassarche melanophris

ABSTRACTCapsule: In Black-browed Albatrosses Thalassarche melanophris nutritional condition is correlated between parents and their offspring.Aims: To test resource allocation hypotheses analysing the relationship between parental and offspring nutritional condition.Methods: We measured blood chemistry parameters related with nutritional condition in 24 parents and their nestlings in a colony of Black-browed Albatrosses.Results: There were no significant differences in blood parameters between sexes or location of the nest within the colony, neither among adults nor among nestlings. We found a significant positive correlation between parents and the nutritional condition of their offspring, measured as urea, uric acid and beta-hydroxybutyrate concentrations in blood.Discussion: These relationships demonstrate that condition of the young seems to be merely a reflection of parental condition. An interesting relationship between alkaline phosphatase concentration in adults and nutritional condition of their nestlings was found, suggesting that age of the parents would be a key factor explaining quality of the nestling.

Multiobjective Testing Resource Allocation Under Uncertainty

Testing resource allocation is the problem of planning the assignment of resources to testing activities of software components so as to achieve a target goal under given constraints. Existing methods build on software reliability growth models (SRGMs), aiming at maximizing reliability given time/cost constraints, or at minimizing cost given quality/time constraints. We formulate it as a multiobjective debug-aware and robust optimization problem under uncertainty of data, advancing the state-of-the-art in the following ways. Multiobjective optimization produces a set of solutions, allowing to evaluate alternative tradeoffs among reliability, cost, and release time. Debug awareness relaxes the traditional assumptions of SRGMs-in particular the very unrealistic immediate repair of detected faults-and incorporates the bug assignment activity. Robustness provides solutions valid in spite of a degree of uncertainty on input parameters. We show results with a real-world case study.

Testing regimes in clinical trials: Evidence from four polio vaccine trajectories

This paper highlights distinctive features of a neglected class of economic activity in the domain of medical innovation, namely the creation of testing regimes in clinical trials, asking how their nature might be expected to affect innovation of medical technology. It argues firstly that clinical trials are not simply about passively validating an already well-known technology and verifying its safety. Rather, clinical trials are part of a more active process of learning that allows pharmaceutical innovations to be useful outside the laboratory. It argues secondly that product development can proceed along a number of long and costly paths before a product’s behaviour in actual practice becomes clear, which can make selecting between alternative courses of action difficult. Thus, product choice and product development need to go hand-in-hand. To consider these arguments, the paper maps out four trajectories of polio vaccine development, tracing their paths through clinical trials since the 1950s, and describes some of the defining features of testing regimes for medical innovation. These include institutions that integrate knowledge and co-ordinate skills in testing processes, and capabilities for allocating testing resources, managing testability constraints, sharing knowledge and improving commensurability between testing communities.

Optimum breeding strategies using genomic selection for hybrid breeding in wheat, maize, rye, barley, rice and triticale.

A breeding strategy with moderate nursery selection followed by genomic selection and one-stage phenotypic selection maximizes annual selection gain for grain yield across a wide range of hybrid breeding scenarios. Genomic selection (GS) is a promising method for the selection of quantitatively inherited traits but its most effective implementation in routine hybrid breeding schemes requires further research. We compared five breeding strategies and varied their available budget, the costs for doubled haploid (DH) line and hybrid seed production as well as variance components for grain yield in a wide range. In contrast to previous studies, we included a nursery selection for disease resistance just before GS on grain yield. The breeding strategy GSrapid with moderate nursery selection followed by one stage GS and one final stage with phenotypic selection on grain yield had the highest annual selection gain across all strategies, budgets, costs and variance components considered and we, therefore, highly recommend its use in hybrid breeding of cereals. Although selecting on traits not correlated with grain yield in the observation nursery, this selection reduced the selection gain of grain yield, especially in the breeding schemes with GS and for selected fractions smaller than 0.3. Owing to the very high number of test candidates entering breeding strategies with GS, the costs for DH line production had a larger impact on the annual selection gain than the hybrid seed production costs. The optimum allocation of test resources maximizing annual selection gain in classical two-stage phenotypic selection on grain yield and for the recommended breeding strategy GSrapid is finally explored for maize, wheat, rye, barley, rice and triticale.

Determination of the Optimal Allocation of Testing Resource for Modular Software Reliability Growth Using LINGO

It is quite natural to produce reliable software systems since the breakdown of the computer systems, which is caused by software faults, results in a tremendous loss and damage for social life. Hence, software reliability is a key factor in software development process. Testing phase of software begins with module testing whereby, modules are tested independently to remove substantial amount of faults within a specified testing resource. Therefore, in this paper four optimization problems are discussed for optimal allocation of testing resources for a modular software system. These optimization problems are formulated as nonlinear programming problems (NLPP), which incorporates log-logistic testing-effort function into inflection S-shaped software reliability growth model based on a non-homogeneous Poisson process. The NLPPs are solved using LINGO, a user-friendly software package for optimization. Finally, numerical example is given to demonstrate and to validate the performance of proposed strategies. In addition, the results of the strategies proposed in this paper are compared with [1]. It is revealed that the proposed strategies yield a gain in efficiency.

A Markov Chain-Based Testability Growth Model With a Cost-Benefit Function

In this paper, we propose a Markov chain-based testability growth model (TGM) for the just in-time fix program. This model can help the system designers to manage the testability growth process during system maturation. We also derive a cost-benefit model for allocating test resources to optimize a specified testability metric subject to a constraint on cumulative test cost. Bayesian inference, coupled with a hybrid genetic and particle swarm optimization method, is used to estimate the parameters of the TGM from evolving data, and the resulting model is utilized to track and project the testability metric. A near-optimal Lagrangian relaxation-based algorithm is applied to solve the test resource allocation problem. The testability growth and resource allocation models are validated via simulation examples. Results show that the model and algorithms presented in this paper have the potential to efficiently manage the testability growth problem.

Determining the optimal allocation of testing resource for modular software system using dynamic programming

Reliability is a major concern in the process of software development because unreliable software can cause failure in the computer system that can be hazardous. A way to enhance the reliability of software is to detect and remove the faults during the testing phase, which begins with module testing wherein modules are tested independently to remove a substantial number of faults within a limited resource. Therefore, the available resource must be allocated among the modules in such a way that the number of faults is removed as much as possible from each of the modules to achieve higher software reliability. In this article, we discuss the problem of optimal resource allocation of the testing resource for a modular software system, which maximizes the number of faults removed subject to the conditions that the amount of testing-effort is fixed, a certain percentage of faults is to be removed and a desired level of reliability is to be achieved. The problem is formulated as a non linear programming problem (NLPP), which is modeled by the inflection S-shaped software reliability growth models (SRGM) based on a non homogeneous Poisson process (NHPP) which incorporates the exponentiated Weibull (EW) testing-effort functions. A solution procedure is then developed using a dynamic programming technique to solve the NLPP. Furthermore, three special cases of optimum resource allocations are also discussed. Finally, numerical examples using three sets of software failure data are presented to illustrate the procedure developed and to validate the performance of the strategies proposed in this article. Experimental results indicate that the proposed strategies may be helpful to software project managers for making the best decisions in allocating the testing resource. In addition, the results are compared with those of Kapur et al. (2004), Huang and Lyu (2005), and Jha et al. (2010) that are available in the literature to deal the similar problems addressed in this article. It reveals that the proposed dynamic programming method for the testing-resource allocation problem yields a gain in efficiency over other methods.

실사격 시험시스템의 효율적인 개발을 위해 안전도 반영을 통해 개선된 시스템 성숙도 모델에 관한 연구

The live fire test has been playing a critical role in evaluating the goals-to-meet of the weapon systems which utilize the power of explosives. As such, the successful development of the test systems therein is quite important. The test systems development covers that of ranges and facilities including system-level key components such as mission control, instrumentation or observation, safety control, electric power, launch pad, and so on. In addition, proper operational guidelines are needed with well-trained test and operation personnel. The emerging weapon systems to be deployed in future battle field would thus have to be more precise and dynamic, smarter, thereby requiring more elaboration. Furthermore, the safety consideration is becoming more serious due to the ever-increasing power of explosives. In such a situation, development of live fire test systems seems to be challenging. The objective of the paper is on how to incorporate the safety and other requirements in the development. To achieve the goal, an architectural approach is adopted by utilizing both the system components relationship and safety requirement when advanced instrumentation technology needs to be developed and deteriorated components of the range are replaced. As an evaluation method, it is studied how the level of maturity of the test systems development can be assessed particularly with the safety requirement considered. Based on the concepts of both systems engineering and SoS (System-of-Systems) engineering process, an enhanced model for the system readiness level is proposed by incorporating safety. The maturity model proposed would be helpful in assessing the maturity of safety-critical systems development whereas the costing model would provide a guide on how the reasonable test resource allocation plan can be made, which is based on the live fire test scenario of future complex weapon systems such as SoS.

Using Defect Taxonomies for Testing Requirements

Defect taxonomies collect and organize experts' domain knowledge and project experience and are valuable for requirements-based testing. They provide systematic backup for the test design, support decisions for allocating testing resources, improve the review of requirements, and are suitable for measuring product quality. Researchers have devised requirements-based testing that exploits defect taxonomies' advantages. The taxonomies can be seamlessly integrated into a standard test process; the researchers successfully applied this approach to industrial projects at a public health insurance institution.

Genomic selection in wheat: optimum allocation of test resources and comparison of breeding strategies for line and hybrid breeding.

The implementation of genomic selection in breeding programs can be recommended for hybrid and line breeding in wheat. High prediction accuracies from genomic selection (GS) were reported for grain yield in wheat asking for the elaboration of efficient breeding strategies applying GS. Our objectives were therefore, (1) to optimize the number of lines, locations and testers in different multi-stage breeding strategies with and without GS, (2) to elaborate the most efficient breeding strategy based on the selection gain and its standard deviation, and (3) to investigate the potential of GS to improve the relative efficiency of hybrid versus line breeding in wheat. We used the open source software package "selectiongain" to optimize the allocation of resources in different breeding strategies by predicting the expected selection gain for a fixed budget. Classical two-stage phenotypic selection was compared with three GS breeding strategies for line and hybrid breeding in wheat. The ranking of the alternative breeding strategies varied largely in dependence of the GS prediction accuracy. Fast-track breeding strategies based solely on GS were only advantageous for high GS prediction accuracies that is >0.50 and >0.65 for hybrid and line breeding, respectively. However, a GS prediction accuracy across breeding cycles of 0.3 or even less must be assumed as realistic for grain yield in wheat. For this low GS prediction accuracy, the use of GS is advantageous for line but especially for hybrid breeding in wheat. Furthermore, the use of GS in hybrid wheat breeding increased the relative efficiency of hybrid versus line breeding and, thus, might be an important pillar for the establishment of hybrid wheat.

An Architecture-Based Multi-Objective Optimization Approach to Testing Resource Allocation

Software systems are widely employed in society. With a limited amount of testing resource available, testing resource allocation among components of a software system becomes an important issue. Most existing research on the testing resource allocation problem takes a single-objective optimization approach, which may not adequately address all the concerns in the decision-making process. In this paper, an architecture-based multi-objective optimization approach to testing resource allocation is proposed. An architecture-based model is used for system reliability assessment, which has the advantage of explicitly considering system architecture over the reliability block diagram (RBD)-based models, and has good flexibility to different architectural alternatives and component changes. A system cost modeling approach which is based on well-developed software cost models is proposed, which would be a more flexible, suitable approach to the cost modeling of software than the approach adopted by others which is based on an empirical cost model. A multi-objective optimization model is developed for the testing resource allocation problem, in which the three major concerns in the testing resource allocation problem, i.e., system reliability, system cost, and the total amount of testing resource consumed, are taken into consideration. A multi-objective evolutionary algorithm (MOEA), called multi-objective differential evolution based on weighted normalized sum (WNS-MODE), is developed. Experimental studies are presented, and the experiments show several results. 1) The proposed architecture-based multi-objective optimization approach can identify the testing resource allocation strategy which has a good trade-off among optimization objectives. 2) The developed WNS-MODE is better than the MOEA developed in recent research, called HaD-MOEA, in terms of both solution quality and computational efficiency. 3) The WNS-MODE seems quite robust from the sensitivity analysis results.

A Learning-to-Rank Approach to Software Defect Prediction

Software defect prediction can help to allocate testing resources efficiently through ranking software modules according to their defects. Existing software defect prediction models that are optimized to predict explicitly the number of defects in a software module might fail to give an accurate order because it is very difficult to predict the exact number of defects in a software module due to noisy data. This paper introduces a learning-to-rank approach to construct software defect prediction models by directly optimizing the ranking performance. In this paper, we build on our previous work, and further study whether the idea of directly optimizing the model performance measure can benefit software defect prediction model construction. The work includes two aspects: one is a novel application of the learning-to-rank approach to real-world data sets for software defect prediction, and the other is a comprehensive evaluation and comparison of the learning-to-rank method against other algorithms that have been used for predicting the order of software modules according to the predicted number of defects. Our empirical studies demonstrate the effectiveness of directly optimizing the model performance measure for the learning-to-rank approach to construct defect prediction models for the ranking task.

Optimal testing strategy in semiconductor testing process

We present an optimal test resource allocation strategy using uncertainty reduction in an environment where resource capacity changes dynamically according to engineering activity. The dynamics of test capacity change are modeled using a linear programming model and then extended and generalized to a Markov decision process. We analyze the model to develop structural results and illustrate its behavior with numerical examples. To the best of our knowledge, this model is the first to define, formalize, and analyze the decision-making process associated with reducing final test time in an environment where capacity may be dynamically increased, depending on engineering activity results.

Selecting software reliability growth models and improving their predictive accuracy using historical projects data

During software development two important decisions organizations have to make are: how to allocate testing resources optimally and when the software is ready for release. SRGMs (software reliability growth models) provide empirical basis for evaluating and predicting reliability of software systems. When using SRGMs for the purpose of optimizing testing resource allocation, the model's ability to accurately predict the expected defect inflow profile is useful. For assessing release readiness, the asymptote accuracy is the most important attribute. Although more than hundred models for software reliability have been proposed and evaluated over time, there exists no clear guide on which models should be used for a given software development process or for a given industrial domain.Using defect inflow profiles from large software projects from Ericsson, Volvo Car Corporation and Saab, we evaluate commonly used SRGMs for their ability to provide empirical basis for making these decisions. We also demonstrate that using defect intensity growth rate from earlier projects increases the accuracy of the predictions. Our results show that Logistic and Gompertz models are the most accurate models; we further observe that classifying a given project based on its expected shape of defect inflow help to select the most appropriate model.

Optimal allocation of testing resources for statistical simulations

Statistical estimates from simulation involve uncertainty caused by the variability in the input random variables due to limited data. Allocating resources to obtain more experimental data of the input variables to better characterize their probability distributions can reduce the variance of statistical estimates. The methodology proposed determines the optimal number of additional experiments required to minimize the variance of the output moments given single or multiple constraints. The method uses multivariate t-distribution and Wishart distribution to generate realizations of the population mean and covariance of the input variables, respectively, given an amount of available data. This method handles independent and correlated random variables. A particle swarm method is used for the optimization. The optimal number of additional experiments per variable depends on the number and variance of the initial data, the influence of the variable in the output function and the cost of each additional experiment. The methodology is demonstrated using a fretting fatigue example.

Cost optimization of a software reliability growth model with imperfect debugging and a fault reduction factor

In modern society people depend on both hardware and software systems. A software system is embedded in every activity of a computer system. The desired performance of a software system is an important issue for many critical systems. Over the past decades, many software models were proposed for estimating the growth of reliability. To improve software quality, software reliability growth models (SRGM) play an important role. The present investigation deals with a SRGM with imperfect debugging, change points and a fault reduction factor (FRF). A FRF is the net number of faults removed in proportion to the failures experienced. This article proposes a new scheme for constructing a SRGM based on a non-homogeneous Poisson process by considering a constant FRF. The main focus is to provide an efficient parametric decomposition for a SRGM. Numerical examples are given to illustrate the validity of analytical results. References Ahmad, N., Khan, M. G. M. and Rafi, L. S. Analysis of an inflection S-shaped software reliability model considering log-logistic testing-effort and imperfect debugging, Int. J. Comput. Sci. Net. Sec. , 11(1)161–171, 2011. http://paper.ijcsns.org/07_book/201101/20110125.pdf . Chang, Y.-C. and Liu, C. A generalized JM model with applications to imperfect debugging in software reliability, Appl. Math. Model. , 33(9)3578–3588, 2009. doi:10.1016/j.apm.2008.11.018 . Goel, A. L. and Okumoto, K. Time-dependent error-detection rate model for software reliability and other performance measures, IEEE T. Reliab. , 28(3):206–211, 1979. doi:10.1109/TR.1979.5220566 . Hsu, C.-J., Huang, C.-Y. and Chang, J.-R. Enhancing software reliability modelling and prediction through the introduction of time-variable fault reduction factor, Appl. Math. Model. , 35(1):506–521, 2011. doi:10.1016/j.apm.2010.07.017 . Jha, P. C., Gupta, D., Yang, B. and Kapur, P. K. Optimal testing resource allocation during module testing considering cost, testing effort and reliability, Comput. Int. Eng. , 57(3):1122–1130, 2009. doi:10.1016/j.cie.2009.05.001 . Kapur, P. K., Kumar, A., Yadav, K. and Khatri, S. K. Software reliability growth modelling for errors of different severity using change point, Int. J. Qual., Reliab., Safe. Eng. , 14(4):311–326, 2007. doi:10.1142/S0218539307002672 . Lyu, M. R. Handbook of Software Reliability Engineering, McGraw-Hill, New York, 1996. Lyu, M. R. and Nikora, A. Applying reliability models more effectively (software), IEEE Software , 9(4):43–52, 1992. doi:10.1109/52.143104 . Musa, J. D. A theory of software reliability and its application, IEEE T. Software Eng. , 3(1):312–327, 1975. doi:10.1109/TSE.1975.6312856 . Musa, J. D. The measurement and management of software reliability, P. IEEE , 68(9):1131–1143, 1980. doi:10.1109/PROC.1980.11812 . Musa, J. D., Lamino, A. and Okumoto, K. Software reliability: Measurement, Prediction, Application, McGraw-Hill, New York, 1987. Okumoto, K. and Goel, A. L. Optimum release time for software system based on reliability and cost criteria, J. Syst. Software , 1:315–318, 1980. doi:10.1016/0164-1212(79)90033-5 . Pham, H. Software Reliability, Springer-Verlag, Singapore, 2000. Pham, H. A software cost model with imperfect debugging, Random life cycle and penalty cost, Int. J. Syst. Sci. , 27(5):455–463, 1996. doi:10.1080/00207729608929237 . Quadri, S. M. K. and Ahmad, N. Software reliability growth modelling with new modified Weibull testing- effort and optimal release policy, Int. J. Comput. Appl. , 6(12), 2010. doi:10.5120/1127-1477 . Quadri, S. M. K., Ahmad, N. and Farooq, S. U. Software reliability growth modelling with generalized exponential testing-effort and optimal software release policy, Global J. Comput. Sci. Tech. , 11(2):27–42, 2011. http://computerresearch.org/stpr/index.php/gjcst/article/view/605 . Shyur, H.-J. A stochastic software reliability model with imperfect debugging and change-point, J. Syst. Software , 66(2):135–141, 2003. doi:10.1016/S0164-1212(02)00071-7 . Xie, M. Software Reliability Modelling, World Scientist, Singapore, 1991. Xie, M. and Yang, B. A study of the effect of imperfect debugging on software development cost model, IEEE T. Software Eng. , 29(5):471–473, 2003. doi:10.1109/TSE.2003.1199075 . Li, X., Xie. M. and Szu H. N. Sensitivity analysis of release time of software reliability models incorporating testing effort with multiple change-points, Appl. Math. Model. , 34(11):3560–3570, 2010. doi:10.1016/j.apm.2010.03.006 . Zhao, J., Liu, H.-W., Cui, G. and Yang, X.-Z. Software reliability growth model with change-point and environmental function, J. Syst. Software , 79(11):1578–1587, 2006. doi:10.1016/j.jss.2006.02.030 .

Dynamic test planning: a study in an industrial context

Testing accounts for a relevant part of the production cost of complex or critical software systems. Nevertheless, time and resources budgeted to testing are often underestimated with respect to the target quality goals. Test managers need engineering methods to perform appropriate choices in spending testing resources, so as to maximize the outcome. We present a method to dynamically allocate testing resources to software components minimizing the estimated number of residual defects and/or the estimated residual defect density. We discuss the application to a real-world critical system in the homeland security domain. We describe a support tool aimed at easing industrial technology transfer by hiding to practitioners the mathematical details of the method application.

Evaluation and Application of Bounded Generalized Pareto Analysis to Fault Distributions in Open Source Software

In general, one of the most important aspects of software development and project management is how to make predictions and assessments of quality and reliability for developed products. Project data usually will be systematically collected and analyzed during the process of software development. Practically, it would be helpful if developers could identify the most error-prone modules early so that they can optimize testing-resource allocation and increase fault detection effectiveness accordingly. In the past, many research studies revealed the applicability of the Pareto principle to software systems, and some of them reported that the Pareto distribution (PD) model can be used to predict the fault distribution of software. In this paper, a special form of the Generalized PD model, named the Bounded Generalized Pareto distribution (BGPD) model, is further proposed to investigate the fault distributions of Open Source Software (OSS). It can be seen that the BGPD model eliminates the issue which occurred in the classical PD model. Three methods of parameter estimation will be presented, and related experiments are performed based on real OSS failure data. Experimental results show that the BGPD model presents high fitness to the actual failure data of OSS. Finally, the possibility of using early limited fault data to predict the later software fault distribution is also studied. Numerical results indicate that the BGPD model can be trusted to consistently produce accurate estimates of fault predictions during the early stages of development. The findings can provide an effective foundation for managing the necessary activities of software development and testing.