Abstract
BackgroundStandard setting of assessment is critical in quality assurance of medical programs. The aims of this study were to identify and compare the impact of methods used to establish the passing standard by the 13 medical schools who participated in the 2014 Australian Medical Schools Assessment Collaboration (AMSAC).MethodsA survey was conducted to identify the standard setting procedures used by participating schools. Schools standard setting data was collated for the 49 multiple choice items used for benchmarking by AMSAC in 2014. Analyses were conducted for nine schools by their method of standard setting and key characteristics of 28 panel members from four schools.ResultsSubstantial differences were identified between AMSAC schools that participated in the study, in both the standard setting methods and how particular techniques were implemented. The correlation between the item standard settings data by school ranged from − 0.116 to 0.632. A trend was identified for panel members to underestimate the difficulty level of hard items and overestimate the difficulty level of easy items for all methods. The median derived cut-score standard across schools was 55% for the 49 benchmarking questions. Although, no significant differences were found according to panel member standard setting experience or clinicians versus scientists, panel members with a high curriculum engagement generally had significantly lower expectations of borderline candidates (p = 0.044).ConclusionThis study used a robust assessment framework to demonstrate that several standard setting techniques are used by Australian medical schools, which in some cases use different techniques for different stages of their program. The implementation of the most common method, the Modified Angoff standard setting approach was found to vary markedly. The method of standard setting used had an impact on the distribution of expected minimally competent student performance by item and overall, with the passing standard varying by up to 10%. This difference can be attributed to the method of standard setting because the ASMSAC items have been shown over time to have consistent performance levels reflecting similar cohort ability. There is a need for more consistency in the method of standard setting used by medical schools in Australia.
Highlights
Standard setting of assessment is critical in quality assurance of medical programs
The aims of this study were firstly to identify the methods of standard setting used by Australian medical schools who participated in Australian Medical Schools Assessment Collaboration (AMSAC) in 2014; secondly to assess the impact of different methods on the school cut-scores for the 2014 AMSAC Multiple choice question (MCQ) items; and thirdly to assess the effects of characteristics of panel members on the standard setting results for each item
Standard setting methods used by Australian medical school Ten of the 12 AMSAC schools that completed the survey used a criterion referenced method of standard setting, with two using a compromise method; the Cohen and the Hofstee [21, 22]
Summary
Standard setting of assessment is critical in quality assurance of medical programs. The aims of this study were to identify and compare the impact of methods used to establish the passing standard by the 13 medical schools who participated in the 2014 Australian Medical Schools Assessment Collaboration (AMSAC). The General Medical Council in the United Kingdom is planning a national medical licensing examination and implementing the examination fully in 2022 [4]. Other countries, such as Australia, are still debating the need for a national approach to setting standards for graduating doctors [5, 6]. The most common method used was the ‘modified’ version of the Angoff method, which is consistent with recent findings for UK medical education [23] and a recent study by Taylor et al [24] which revealed differences in passing standards for a common set of items across a number of UK medical schools
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
