Abstract
Formula scoring (FS) is the use of a don’t know option (DKO) with subtraction of points for wrong answers. Its effect on construct validity and reliability of progress test scores, is subject of discussion. Choosing a DKO may not only be affected by knowledge level, but also by risk taking tendency, and may thus introduce construct-irrelevant variance into the knowledge measurement. On the other hand, FS may result in more reliable test scores. To evaluate the impact of FS on construct validity and reliability of progress test scores, a progress test for radiology residents was divided into two tests of 100 parallel items (A and B). Each test had a FS and a number-right (NR) version, A-FS, B-FS, A-NR, and B-NR. Participants (337) were randomly divided into two groups. One group took test A-FS followed by B-NR, and the second group test B-FS followed by A-NR. Evidence for impaired construct validity was sought in a hierarchical regression analysis by investigating how much of the participants’ FS-score variance was explained by the DKO-score, compared to the contribution of the knowledge level (NR-score), while controlling for Group, Gender, and Training length. Cronbach’s alpha was used to estimate NR and FS-score reliability per year group. NR score was found to explain 27 % of the variance of FS [F(1,332) = 219.2, p < 0.0005], DKO-score, and the interaction of DKO and Gender were found to explain 8 % [F(2,330) = 41.5, p < 0.0005], and the interaction of DKO and NR 1.6 % [F(1,329) = 16.6, p < 0.0005], supporting our hypothesis that FS introduces construct-irrelevant variance into the knowledge measurement. However, NR-scores showed considerably lower reliabilities than FS-scores (mean year-test group Cronbach’s alphas were 0.62 and 0.74, respectively). Decisions about FS with progress tests should be a careful trade-off between systematic and random measurement error.
Highlights
Progress testing is used to assess knowledge growth (McHarg et al 2005)
To study the effect of the don’t know option (DKO) on construct validity of Formula scoring (FS) test scores, we investigated risk taking tendency, which is influenced by gender and personality, as a major source of construct irrelevant variance in progress test scores, that are obtained under FS conditions
Our results show that for participants with the same training length, the larger part (27 %) of the FS score variance is explained by the variation in knowledge level, as represented by the number right (NR) score, indicating that FS score measures to a large extent a participant’s knowledge level
Summary
Progress testing is used to assess knowledge growth (McHarg et al 2005). Participants take tests about a knowledge domain across the full breadth of a curriculum at regular intervals during the course. A ‘‘don’t know’’ option (DKO) is frequently added to stimulate participants to recognize their knowledge deficit (McHarg et al 2005). With true–false tests the penalty for a wrong answer is -1, and the reward of a correct answer is ?1. The test score is defined as the number of correct minus the number of incorrect answers, expressed as a percentage of the maximum score (Kurz 1999). Participants can choose the DKO when they do not know the answer, otherwise they would be forced to guess which would result in random correct and incorrect answers. The internal consistency among test scores is expected to be higher in tests with DKO, and the reliability of these test scores likewise (Burton 2004; Diamond and Evans 1973; Mattson 1965)
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
