Sequential Mastery Testing Research Articles

Variations on Stochastic Curtailment in Sequential Mastery Testing

In sequential mastery testing (SMT), assessment via computer is used to classify examinees into one of two mutually exclusive categories. Unlike paper-and-pencil tests, SMT has the capability to use variable-length stopping rules. One approach to shortening variable-length tests is stochastic curtailment, which halts examination if the probability of changing classification decisions is low. The estimation of such a probability is therefore a critical component of a stochastically curtailed test. This article examines several variations on stochastic curtailment where the key probability is estimated more aggressively than the standard formulation, resulting in additional savings in average test length (ATL). In two simulation sets, the variations successfully reduced the ATL, and in many cases the average loss, compared with the standard formulation.

On Using Stochastic Curtailment to Shorten the SPRT in Sequential Mastery Testing

Sequential mastery testing (SMT) has been researched as an efficient alternative to paper-and-pencil testing for pass/fail examinations. One popular method for determining when to cease examination in SMT is the truncated sequential probability ratio test (TSPRT). This article introduces the application of stochastic curtailment in SMT to shorten the TSPRT without substantially compromising error rates. Unlike the TSPRT, the stochastically curtailed procedure exhibits an optimality property known as weak admissibility. Error bounds of the two methods are provided in terms of one another. In two simulation sets, the stochastically curtailed procedure considerably improved the average test length of an SMT with only a slight decrease in accuracy.

The Wald–Wolfowitz Theorem Is Violated in Sequential Mastery Testing

The sequential probability ratio test (SPRT) has important optimality properties when sequential observations are independent and identically distributed (i.i.d.). Most famously, the Wald–Wolfowitz theorem states that under i.i.d., no other method has both better error rates and better average sample lengths. This article presents three counterexamples to demonstrate that the Wald–Wolfowitz theorem does not hold in sequential mastery testing (SMT), an application of the SPRT where observations are not i.i.d. Consequences of this violation are discussed.

Application of Sequential Probability Ratio Test to Computerized Criterion-Referenced Testing

The adaptive testing is an important testing method in the modern educational/psychological testings. In adaptive mastery testing, the items are selected adaptively according to the estimated information of the unknown latent ability levels, and given then to the test-takers, sequentially. Hence, the decision (master or non-master; pass or fail) for each test-taker is made sequentially based on each test-taker's responses to a particular sequence of items administered to him/her. Thus, statistically speaking, by the natural character of the adaptive mastery testing, it is a sequential problem with dependent observations. The Wald's [Wald, A. Sequential Analysis; Wiley: New York, 1947] SPRT (sequential probability ratio test) has been applied to this kind of mental testing problem by many researchers in the field of educational/psychological measurement theory; for example, Reckase [Reckase, M. A procedure for decision making using tailored testing. In New Horizons in Testing – Latent Trait Test Theory and Computerized Adaptive Testing; Weiss, D.J., Ed.; Academic Press: New York, 1983; 238–257], Kingsbury and Weiss [Kingsbury, G.; Weiss, D.J. A comparison of IRT-based adaptive mastery testing and a sequential mastery testing procedure. In New Horizons in Testing – Latent Trait Test Theory and Computerized Adaptive Testing; Weiss, D.J., Ed.; Academic Press: New York, 1983; 258–288] and Spray [Spray, J. Multiple-Category Classification Using a Sequential Probability Ratio Test, ACT Research Report Series; 1993; 93–7]. Most of their results are empirical studies of the performance of SPRT with different item selection schemes. In statistical literature, the SPRT with i.i.d. observations has been intensively studied by many statisticians since Wald [Wald, A. Sequential Analysis; Wiley: New York, 1947]. In this paper, we concentrate on the properties of the stopping time of the SPRT under adaptive mastery testing; i.e., the test items (observations) are adaptively selected. Not only are there a few people discussing the properties of SPRT under non-i.i.d. setup, but also most of them study just large sample properties, which provide very little information for test-makers to design good mastery tests. Without the independence property of the observations, it will require different approaches to analyze its performance. Here we apply some results of linear growth processes and a martingale extension of the Wald's [Wald, A. Sequential Analysis; Wiley: New York, 1947] equation to obtain a bound for the expectation of the stopping rules (i.e., the test length) used in adaptive mastery tests.

A Bayesian sequential procedure for determining the optimal number of interrogatory examples for concept-learning

The purpose of this paper is to derive optimal rules for sequential decision-making in intelligent tutoring systems. In a sequential mastery test, the decision is to classify a student as a master, a nonmaster, or to continue testing and administering another item. The framework of Bayesian sequential decision theory is used; that is, optimal rules are obtained by minimizing the posterior expected losses associated with all possible decision rules at each stage of testing and using techniques of backward induction. The main advantage of this approach is that costs of testing can be taken explicitly into account. The sequential testing procedure is demonstrated for determining the optimal number of interrogatory examples for concept-learning in the Minnesota adaptive instructional system. The paper concludes with an empirical example in which, for given maximum number of interrogatory examples for concept-learning in medicine, the appropriate action is indicated at each stage of testing for different number-correct score.

Applications of Bayesian Decision Theory to Sequential Mastery Testing

The purpose of this paper is to formulate optimal sequential rules for mastery tests. The framework for the approach is derived from Bayesian sequential decision theory. Both a threshold and linear loss structure are considered. The binomial probability distribution is adopted as the psychometric model involved. Conditions sufficient for sequentially setting optimal cutting scores are presented. Optimal sequential rules will be derived for the case of a subjective beta distribution representing prior true level of functioning. An empirical example of sequential mastery esting for concept-learning in medicine concludes the paper.

Applications of Bayesian Decision Theory to Sequential Mastery Testing

The purpose of this paper is to formulate optimal sequential rules for mastery tests. The framework for this approach is derived from empirical Bayesian decision theory. Both a threshold and linear loss structure are considered. The binomial probability distribution is adopted as the psychometric model involved. Conditions sufficient for sequentially setting optimal cutting scores are presented. Optimal sequential rules will be derived for the case of a beta distribution representing prior true level functioning. An empirical example of sequential mastery testing for concept-learning in medicine concludes the paper.

COMPUTERIZED MASTERY TESTING USING FUZZY SET DECISION THEORY

Lewis and Sheehan (1990) developed a computerized sequential mastery test procedure that utilizes Bayesian decision theory to make “master,” “nonmaster,” or “continue testing” decisions. While maintaining their general framework of administering sequential testlets, a fuzzy set approach was used to develop an alternative computerized mastery test. This new procedure differs from Lewis and Sheehan's in that the decision rule is determined using fuzzy set decision theory, and ability estimates are obtained using the Rasch model rather than a three‐parameter logistic model. This article describes this new approach and illustrates the differences between the fuzzy set and Bayesian methods by way of an example.

Computerized Mastery Testing Using Fuzzy Set Decision Theory

Lewis and Sheehan (1990) developed a computerized sequential mastery test procedure that utilizes Bayesian decision theory to make "master," "nonmaster," or "continue testing" decisions. While maintaining their general framework of administering sequential testlets, a fuzzy set approach was used to develop an alternative computerized mastery test. This new procedure differs from Lewis and Sheehan's in that the decision rule is determined using fuzzy set decision theory, and ability estimates are obtained using the Rasch model rather than a three-parameter logistic model. This article describes this new approach and illustrates the differences between the fuzzy set and Bayesian methods by way of an example.