Multiple True-false Items Research Articles

Effect of different scoring approaches upon credit assignment when using Multiple True‐False items in dental undergraduate examinations

Various scoring approaches for Multiple True-False (MTF) items exist. This study aimed at comparing scoring results obtained with different scoring approaches and to assess the effect of item cues on each scoring approaches' result. Different scoring approaches (MTF, Count-2, Count-3, "Vorkauf-Method," PS50 , Dichotomized MTF, "Blasberg-Method," Multiple response (MR), Correction for Guessing, "Ripkey-Method," Morgan-Method, Balanced Scoring Method) were retrospectively applied to all MTF items used within electronic examinations of undergraduate dental students at the University Medical Center Göttingen in the winter term 2016/2017 (1297 marking events). Item quality was evaluated regarding formal parameters such as presence of cues and correctness of content. Differences between scoring results of all scoring approaches and the differences between each methods' scoring results of items with and without cues were calculated by Wilcoxon rank sum tests (P<.05). Average scoring results per item highly differed between the scoring approaches and ranged from 0.46 (MR) to 0.92 (Dichotomized MTF). Presence of cues leads to significantly higher scoring in case of all scoring approaches (P<.001; +0.14 on average). However, effect of cues differed amongst scoring approaches and ranged from +0.04 (Dichotomized MTF) to +0.20 (MR). Scoring of MTF items is complex. The data presented in this manuscript may help educators make informed choices about scoring algorithms.

Multiple true-false items: a comparison of scoring algorithms.

Multiple true-false (MTF) items are a widely used supplement to the commonly used single-best answer (Type A) multiple choice format. However, an optimal scoring algorithm for MTF items has not yet been established, as existing studies yielded conflicting results. Therefore, this study analyzes two questions: What is the optimal scoring algorithm for MTF items regarding reliability, difficulty index and item discrimination? How do the psychometric characteristics of different scoring algorithms compare to those of Type A questions used in the same exams? We used data from 37 medical exams conducted in 2015 (998 MTF and 2163 Type A items overall). Using repeated measures analyses of variance (rANOVA), we compared reliability, difficulty and item discrimination of different scoring algorithms for MTF with four answer options and Type A. Scoring algorithms for MTF were dichotomous scoring (DS) and two partial credit scoring algorithms, PS50 where examinees receive half a point if more than half of true/false ratings were marked correctly and one point if all were marked correctly, and PS1/n where examinees receive a quarter of a point for every correct true/false rating. The two partial scoring algorithms showed significantly higher reliabilities (αPS1/n=0.75; αPS50=0.75; αDS=0.70, αA=0.72), which corresponds to fewer items needed for a reliability of 0.8 (nPS1/n=74; nPS50=75; nDS=103, nA=87), and higher discrimination indices (rPS1/n=0.33; rPS50=0.33; rDS=0.30; rA=0.28) than dichotomous scoring and Type A. Items scored with DS tend to be difficult (pDS=0.50), whereas items scored with PS1/n become easy (pPS1/n=0.82). PS50 and Type A cover the whole range, from easy to difficult items (pPS50=0.66; pA=0.73). Partial credit scoring leads to better psychometric results than dichotomous scoring. PS50 covers the range from easy to difficult items better than PS1/n. Therefore, for scoring MTF, we suggest using PS50.

Language testing

06–515Dudley, Albert (Aichi Prefectural U, Japan), Multiple dichotomous-scored items in second language testing: investigating the multiple true–false item type under norm-referenced conditions. Language Testing (Hodder Arnold) 23.2 (2006), 198–228.06–516Lee, Yong-Won (Educational Testing Service, China), Dependability of scores for a new ESL speaking assessment consisting of integrated and independent tasks. Language Testing (Hodder Arnold) 23.2 (2006), 131–166.06–517Mcnamara, Tim (U Melbourne, Australia; tfmcna@unimelb.edu.au), 21st Century shibboleth: Language tests, identity and intergroup conflict. Language Policy (Springer) 4.4 (2005), 351–370.06–518Roever, Carsten (U Melbourne, Australia), Validation of a web-based test of ESL pragmalinguistics. Language Testing (Hodder Arnold) 23.2 (2006), 229–256.06–519Vincent, John (U Melbourne, Australia; jtv@unimelb.edu.au), Children writing: Multimodality and assessment in the writing classroom. Literacy (Blackwell) 40.1 (2006), 51–57.06–520Weir, Cyril J. &amp; Wu, Jessica R. W. (U Luton, UK &amp; Testing Center, Taiwan), Establishing test form and individual task comparability: a case study of a semi-direct speaking test. Language Testing (Hodder Arnold) 23.2 (2006), 167–197.

Multiple dichotomous-scored items in second language testing: investigating the multiple true-false item type under norm-referenced conditions

This study examined the multiple true-false (MTF) test format in second language testing by comparing multiple-choice (MCQ) and multiple true-false (MTF) test formats in two language areas of general English: vocabulary and reading. Two counter-balanced experimental designs - one for each language area - were examined in terms of the number of MCQ and MTF differentially responded to, MTF item dependency, reliability, and concurrent validity. The data were analysed by classical test theory (CTT) and Rasch analysis. The results showed a two- and three-fold increase in vocabulary and reading items answered, respectively. Participants responded to significantly more MTF items than MCQ, and further analysis revealed no item dependency for both language domains. Reliability increases were found in the reading tests. Item conversions did not alter the basic functioning of the MTF items, and common person equating plots demonstrated a steady relationship between MCQ and MTF person ability estimates.

The Generalizability of Ratings of Item Relevance

The relevance of test content to practice is essential for credentialing examinations and one way to ensure it is to collect ratings of item relevance from job incumbents. This study analyzed ratings of the 132 single-best-answer items and 117 multiple true-false item sets that formed the pretest books in a single administration of a medical certifying examination. Ratings collected from 57 practitioners were high (an average of more than 4 on a 5-point scale) and correlated with item difficulty (r = .31 to .34). The relationship between ratings and item discrimination is less clear (r = -.04 to .31). Application of generalizability theory to the ratings shows that reasonable estimates of item, stem, and total test relevance can be obtained with about 10 raters.

Item Type and Cognitive Ability Measured: The Validity Evidence for Multiple True-False Items in Medical Specialty Certification

This study compared the criterion-related validity evidence and other psycho- metric characteristics of multiple-choice (MCQ) and multiple true-false (MTF) items in medical specialty certifying examinations in internal medicine and its subspecialties. Results showed that MTF items were more reliable than MCQs and that the format scores were highly correlated. However, MCQs were more highly correlated with an independent performance measure than were MTF items. MTF items were classified primarily as measuring knowledge rather than synthesis or judgment. These results may have implications for examination construction, especially if criterion-related validity evidence is important.

A Brief Report on a Comparison of Six Scoring Methods for Multiple True-False Items

Six methods of scoring multiple true-false items were compared in terms of the reliabilities, difficulties, and discrimination. It was found that the differences in reliabilities and discrimination values were not statistically significant. After adjustments for scoring metrics were made, the multiple true-false, the correction-for-guessing, and the let-omit method were found to yield higher item means (i.e., easier), whereas the multiple response method yielded the lowest item mean. Results of this study suggest that, for norm-referenced score interpretations, there is insufficient evidence to support any one of the six methods as superior to others psychometrically. For criterion-referenced score interpretations, however, the effects of the scoring method on score interpretation and the determination of passing scores need to be taken into consideration.

Effectiveness of Multiple True-False Items

This study was designed to extend and partially replicate previous research comparing the multiple true-false (MTF) and multiple-choice (MC) item formats. Using content-parallel forms of a college-level social psychology midterm exam, the reliabilities, difficulties, concurrent validity, and response rates were investigated. The MTF measures were shown to yield higher reliabilities, lower adjusted means, and higher response rates when compared with those of MC. Also, correlations between subtests, when corrected for the unreliability, were very close to one. These results provide additional support for the use of MTF tests as an alternative to MC tests in the measurement of achievement.

ESTIMATING THE RELIABILITY OF MULTIPLE TRUE‐FALSE TESTS

This study was designed to examine the level of dependence within multiple true‐false (MTF) test item clusters by computing sets of item intercorrelations with data from a test composed of both MTF and multiple choice (MC) items. It was posited that internal analysis reliability estimates for MTF tests would be spurious due to elevated MTF within‐cluster intercorrelations. Results showed that, on the average, MTF within‐cluster dependence was no greater than that found between MTF items from different clusters, between MC items, or between MC and MTF items. But item for item, there was greater dependence between items within the same cluster than between items of different clusters.

The Ineffectiveness of Multiple True-False Test Items

Graduate students in education responded twice at the same sitting to 100 true-false questions on educational measurement. The items were presented as Part 1 and Part 2 of a midterm test. In Part 1, the items were presented separately with instructions to the students to mark each statement true or false. In Part 2, the same items were presented in clusters (32 triads, 2 dyads) with instructions to pick the one true (or the one false) statement. Scores on Part 1 were much more reliable than scores on Part 2. These results support the suggestion from test specialists that test constructors should avoid use of multiple true-false items. The relation between the difficulty of the component true-false items and of the multiple choice clusters was examined.