Abstract
BackgroundThe outcome of assessments is determined by the standard-setting method used. There is a wide range of standard – setting methods and the two used most extensively in undergraduate medical education in the UK are the norm-reference and the criterion-reference methods. The aims of the study were to compare these two standard-setting methods for a multiple-choice question examination and to estimate the test-retest and inter-rater reliability of the modified Angoff method.MethodsThe norm – reference method of standard -setting (mean minus 1 SD) was applied to the 'raw' scores of 78 4th-year medical students on a multiple-choice examination (MCQ). Two panels of raters also set the standard using the modified Angoff method for the same multiple-choice question paper on two occasions (6 months apart). We compared the pass/fail rates derived from the norm reference and the Angoff methods and also assessed the test-retest and inter-rater reliability of the modified Angoff method.ResultsThe pass rate with the norm-reference method was 85% (66/78) and that by the Angoff method was 100% (78 out of 78). The percentage agreement between Angoff method and norm-reference was 78% (95% CI 69% – 87%). The modified Angoff method had an inter-rater reliability of 0.81 – 0.82 and a test-retest reliability of 0.59–0.74.ConclusionThere were significant differences in the outcomes of these two standard-setting methods, as shown by the difference in the proportion of candidates that passed and failed the assessment. The modified Angoff method was found to have good inter-rater reliability and moderate test-retest reliability.
Highlights
The outcome of assessments is determined by the standard-setting method used
There is a wide range of standard -setting methods but the most popular ones in undergraduate medical education in the UK are the norm – reference methods and the criterion reference methods
The Intra Class Correlation Coefficient (ICC) calculated as the average measure of reliability and by using two-way random effects model was 0.82
Summary
The outcome of assessments is determined by the standard-setting method used. There is a wide range of standard – setting methods and the two used most extensively in undergraduate medical education in the UK are the norm-reference and the criterion-reference methods. Standards are generally classed as absolute (criterion based) or relative (norm based) [3,4,5]. An absolute standard determines the pass/ fail outcome by how well a candidate performs and he/ she is usually judged against an arbitrarily set external standard. It is independent of the performance of the group. A relative standard on the other hand, compares how well the examinee has performed compared to others who took the test and the outcome (pass/ fail) is dependent on the performance of the group
Sign-in/Register to view highlights & summary 
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call