Answer-until-correct Research Articles

Partial Credit in Answer-Until-Correct Multiple-Choice Tests Deployed in a Classroom Setting

ABSTRACT The answer-until-correct (AUC) method of multiple-choice (MC) testing involves test respondents making selections until the keyed answer is identified. Despite attendant benefits that include improved learning, broad student adoption, and facile administration of partial credit, the use of AUC methods for classroom testing has been extremely limited. This study presents scoring properties and item analysis for 26 AUC university course examinations, administered using a commercial scratch-card response system. Here, we show that beyond the traditional pedagogical advantages of AUC, the availability of partial credit adds psychometric advantages by boosting both the mean item discrimination and overall test-score reliability, when compared to tests scored dichotomously upon initial response. Furthermore we also find a strong correlation between students’ initial-response successes and the likelihood that they would obtain partial credit when they make incorrect initial responses. Thus, partial credit is being granted based on partial knowledge that remains latent in traditional MC tests. The fact that these advantages are realized in real-life classroom tests may motivate further expansion of the use of AUC MC tests in higher education.

Score Increase and Partial-Credit Validity When Administering Multiple-Choice Tests Using an Answer-Until-Correct Format

There are numerous benefits to answer-until-correct (AUC) approaches to multiple-choice testing, not the least of which is the straightforward allotment of partial credit. However, the benefits of granting partial credit can be tempered by the inevitable increase in test scores and by fears that such increases are further contaminated by a large random guessing component. We have measured the effects of using the immediate feedback assessment technique (IF-AT), a commercially available AUC response system, on the scores of a typical first-year chemistry multiple-choice test. We find that with a particular commonly used scoring scheme the test scores from IF-AT deployment are 6–7 percentage points higher than from Scantron deployment. This amount is less than that suggested by previous studies, where the mark increase was calculated in a purely post hoc manner and thus neglected affective changes of students’ behavior associated with the IF-AT technique. Furthermore, we have strong evidence that partial cr...

The effect of the nonhuman external regulator’s answer-until-correct (AUC) versus knowledge-of-result (KR) task feedback on children’s behavioral regulation during learning tasks

The present study was conducted to shed a new light on the nonhuman’s external regulation effect on children’s behavioral regulation through investigating the effect of the computer’s task feedback answer-until-correct (AUC) versus knowledge-of-result (KR) with 40 preschool children (boys/girls) divided into AUC-Condition versus KR-Condition. The Aginian’s methodology ( Agina, Kommers, & Steehouder, 2010) with the latest updates ( Agina, Kommers, & Steehouder, 2011) was used, which involves an isolated, computer-based learning system with three different computer’s agents for measuring self-regulation as a function of the task level selection, self-regulation as a function of task precision, and special agent for evaluating children’s satisfaction. It was hypothesized that AUC-Condition will outperform KR-Condition in verbalization intensity, manifested self-regulation, and the degree of satisfaction. Despite the results were not confirmed the hypothesis, the results generated by the game were consistent with the statistical results in which this consistency increases, to a great extent, the reliability of the Aginian’s measurements. However, both results were not confirmed Vygotsky’s view or Piaget’s view of self-regulation development as both concluded that thinking aloud and self-regulation have a reverse relationship and, therefore, thinking aloud, per se, can be used to explore the problems the children may not agree to talk about.

The Answer-Until-Correct Item Format Revisited

Current availability of computers has led to the use of a new series of response formats that are an alternative to the classical dichotomic format, and to the recovery of other formats, like the case of the answer-until-correct (AUC) format, whose efficient administration requires this kind of technology. The goal of the present study is to determine whether the use of the AUC format improves test reliability and validity in comparison to the classical dichotomic format. Three samples of 174, 431, and 1,446 Spanish students from secondary education, professional training, and high school, ages between 13 and 20 years, were used. A 100-item test and a 25-item test that assessed knowledge of Universal History were used, both tests administered by Internet with the AUC format. There were 56 experimental conditions, resulting from the manipulation of eight scoring models and seven test lengths. The data were analyzed from the perspective of the Classical Test Theory and also with Item Response Theory (IRT) models. Reliability and construct validity, analyzed from the classic perspective, did not seem to improve significantly when using the AUC format; however, when assessing reliability with the Information Function obtained by means of IRT models, the advantages of the AUC format versus the dichotomic format become clear. For low levels of the trait assessed, scores obtained with the AUC format provide more information than scores obtained with the dichotomic format. Lastly, these results are commented on, and the possibilities and limits of the AUC format in highly computerized psychological and educational contexts are analyzed.

The effects of different feedback strategies using computer-administered multiple-choice questions as instruction

This study investigated the effects of using different forms of computer-based feedback on high school students' learning of science material. The basic design consisted of two conditions of instructional support (text and questions vs. questions only), two testings (immediate vs. retention), five levels of similarity between lesson and posttest questions, and five feedback conditions: knowledge of correct response (KCR), delayed KCR, answer until correct (AUC), questions only (no feedback), and no questions. Results showed significant benefits for feedback over no feedback, with AUC becoming more advantageous and delayed feedback less so as lesson-posttest question similarity decreased. Also, with decreased question similarity and the availability of supporting text, overall feedback effects tended to decrease. The results are discussed in terms of the information processing effects of the different feedback forms, a factor that CBI designers often fail to exploit in planning feedback conditions.

THE EFFECTS OF ITEM BY ITEM FEEDBACK GIVEN DURING AN ABILITY TEST

Summary. Answer‐until‐correct (AUC) is a procedure for providing item by item feedback during a multiple‐choice test, giving an increased range of scores. The performance of a group of children on an ability test using AUC procedures was compared with a group using conventional instructions. AUC scores considerably enhanced reliability but not validity. A comparison of item analyses for the two groups showed no evidence of learning during the test. This suggests that attempts to derive measures of ‘ability to learn’ which are based on changes in responding during the course of a test administered with feedback should be treated with caution. Evidence should be provided that similar changes in response pattern do not occur in a conventionally administered test.

Solving Measurement Problems with an Answer-Until-Correct Scoring Procedure

Answer-until-correct (AUC) tests have been in use for some time. Pressey (1950) pointed to their ad vantages in enhancing learning, and Brown (1965) proposed a scoring procedure for AUC tests that appears to increase reliability (Gilman & Ferry, 1972; Hanna, 1975). This paper describes a new scoring procedure for AUC tests that (1) makes it possible to determine whether guessing is at ran dom, (2) gives a measure of how "far away" guess ing is from being random, (3) corrects observed test scores for partial information, and (4) yields a mea sure of how well an item reveals whether an ex aminee knows or does not know the correct re sponse. In addition, the paper derives the optimal linear estimate (under squared-error loss) of true score that is corrected for partial information, as well as another formula score under the assumption that the Dirichlet-multinomial model holds. Once certain parameters are estimated, the latter formula score makes it possible to correct for partial infor mation using only the examinee's usual number- correct observed score. The importance of this for mula score is discussed. Finally, various statistical techniques are described that can be used to check the assumptions underlying the proposed scoring procedure.

The Effect of Guessing on Item Reliability under Answer-Until-Correct Scoring

The answer-until-correct (AUC) procedure re quires that examinees respond to a multiple-choice item until they answer it correctly. The examinee's score on the item is then based on the number of responses required for the item. It was expected that the additional responses obtained under the AUC procedure would improve reliability by pro viding additional information on those examinees who fail to choose the correct alternative on their first attempt. However, when compared to the zero- one (ZO) scoring procedure, the AUC procedure has failed to yield consistent improvements in relia bility. Using a modified version of Horst's model for examinee behavior, this paper compares the ef fect of guessing on item reliability for the AUC pro cedure and the ZO procedure. The analysis shows that the relative efficiency of the two procedures de pends strongly on the nature of the item alterna tives and implies that the appropriate criteria for item selection are different for each procedure. Conflicting results reported for empirical compari sons of the reliabilities of the two procedures may result from a failure to control for the characteris tics of the items.