Angoff Procedure Research Articles

Are we passing the acceptable? Standard setting of theoretical proficiency tests for foreign-trained dentists.

Dentists educated outside the EU, EEA and Switzerland have the option to undergo proficiency testing to practice dentistry in Sweden. A standardised evidence-based procedure for proficiency tests is crucial. This paper explores if the standard setting of the theoretical examination in the Swedish proficiency test for foreign dentists is fit to differentiate candidates who meet the acceptable standard for licensure from those who do not, by analysis of inter-rater reliability and credibility of Angoff panels and the test's ability to differentiate candidates regarding learning outcomes and dental disciplines. In a cohort study design, test results were collected from seven examinations during 2018-2019 (n=316), combined with Angoff judgements and demographic data from The National Board of Health and Welfare. Intraclass correlation, Pearson correlation and linear regression were calculated. All exams demonstrated substantial judge inter-rater reliability using the Angoff procedure (0.65-0.82). Item difficulty predicted by the expert panels showed significant positive correlations with the item difficulty by candidate results (0.69-0.77). Candidates who did not meet the requirements for pass mark in the theoretical test revealed a significantly reduced performance overall in professional qualifications (i.e., learning outcomes) and reduced performance within dental disciplines. The results indicate that a modified Angoff method is a feasible, simple, and defensible method for setting the pass mark for the theoretical examinations. The standard setting appears to differentiate candidates who do not meet the requirements of the professional qualifications of the dental degree. Adjustments could be undertaken to further discriminate for competence within single dental disciplines.

A cross-sectional survey of nursing students' patient safety knowledge

BackgroundKnowledge provides a foundation for safe and effective nursing practice. However, most previous studies have focused on exploring nursing students' self-reported perceptions of, or confidence in, their level of patient safety knowledge, rather than examining their actual levels of knowledge. ObjectiveThe overarching objective of this study was to examine final year nursing students' levels of knowledge about key patient safety concepts. DesignA cross-sectional design was used for this study. Data collection was undertaken during 2018 using a web-based patient safety quiz with 45 multiple choice questions informed by the Patient Safety Competency Framework for Nursing Students. A Modified Angoff approach was used to establish a pass mark or ‘cut score’ for the quiz. Setting and participantsNursing students enrolled in the final year of a pre-registration nursing program in Australia or New Zealand were invited to participate in the study. ResultsIn total, 2011 final year nursing students from 23 educational institutions completed the quiz. Mean quiz scores were 29.35/45 or 65.23% (SD 5.63). Participants achieved highest scores in the domains of person-centred care and therapeutic communication, and lowest scores for infection prevention and control and medication safety.Based on the pass mark of 67.3% determined by the Modified Angoff procedure, 44.7% of students (n = 899) demonstrated passing performance on the quiz. For eight of the institutions, less than half of their students achieved a passing mark. ConclusionsGiven the pivotal role that nurses play in maintaining patient safety, the results from this quiz raise important questions about the preparation of nursing students for safe and effective clinical practice. The institutional results also suggest the need for increased curricula attention to patient safety.

Effect of Trainee Performance Data on Standard-Setting Judgments Using the Mastery Angoff Method.

Mastery learning in health professions education requires learners to learn and undergo assessment until they demonstrate a high level of competence. Setting defensible standards is key to accurately assessing educational outcomes in mastery learning. The Mastery Angoff method was proposed recently to set a minimum passing standard (MPS) for mastery learning curricula. However, it is unknown whether prior knowledge of trainee performance affects judges' decisions about setting an MPS using the Mastery Angoff method. We sought to determine the effect of introducing baseline data about trainee performance on faculty judges' decisions about the Mastery Angoff MPS for a written examination. We developed a mastery learning curriculum to train internal medicine residents and cardiology fellows about the correct interpretation of inpatient telemetry monitoring. All learners were required to meet or exceed an MPS on a 35-item written examination at the end of training. The MPS was set in 2017 by judges who used the item-based Mastery Angoff method without prior examinee performance information. The judges subsequently reevaluated the test items after receiving baseline data about trainee performance collected during pilot testing. Mastery Angoff MPSs with and without baseline performance data were compared. Twelve judges participated in the standard-setting exercise. The initial MPS was similar to the repeat MPS set after judges received trainee performance data (86.2% versus 86.9%, P = .23). Prior knowledge about medical trainee performance data did not affect MPS as determined by the Mastery Angoff procedure.

Standards of an Examination (cut points), Construction of Scenarios, Measurement of lmpacts in the Decision Making Process

The National Board of Medical Examiners (NBME®) applies rigorous examinations to determine the impact on candidates taking the United States Medical License Examination (USMLE®) for the purpose of licensing physicians in the US. UU UU These standards apply to all levels of exam development, administration and qualification. The standards apply to the following processes:• Validity of the inferences of the articles: content: the content tested must be appropriate, asking the correct questions, the format of the element, or the design of the test and the design, or the process by which the test is performed, the documentation• Accuracy of the scores: reliability, focalization, information on the cutting edge, standard errors• Determination and application of cut points: methods, modified Angoff procedure, misclassification error.The main focus is on the last standard, although the first two levels are also briefly discussed. Different methods to establish an approval standard will be discussed and the method used for USMLE will be described. Errors of misclassification are also presented and how to minimize them.

Non-numeric Intrajudge Consistency Feedback in an Angoff Procedure

The credibility of standard-setting cut scores depends in part on two sources of consistency evidence: intrajudge and interjudge consistency. Although intrajudge consistency feedback has often been provided to Angoff judges in practice, more evidence is needed to determine whether it achieves its intended effect. In this randomized experiment with 36 judges, non-numeric item-level intrajudge consistency feedback was provided to treatment-group judges after the first and second rounds of Angoff ratings. Compared to the judges in the control condition, those receiving the feedback significantly improved their intrajudge consistency, with the effect being stronger after the first round than after the second round. To examine whether this feedback has deleterious effects on between-judge consistency, I also examined interjudge consistency at the cut score level and the item level using generalizability theory. The results showed that without the feedback, cut score variability worsened; with the feedback, idiosyncratic item-level variability improved. These results suggest that non-numeric intrajudge consistency feedback achieves its intended effect and potentially improves interjudge consistency. The findings contribute to standard-setting feedback research and provide empirical evidence for practitioners planning Angoff procedures.

Fortune-tellers or content specialists: challenging the standard setting paradigm in medical education programmes

The veracity of Objective Standard Setting (OSS) as a modern approach to criterion-referenced standard setting has been reported for healthcare student assessment in the USA, while in other countries, OSS remains unrecognized. OSS upholds the foundational principle for itemized tests that judges should base their decisions on test item content. Moreover, it presents judges with a conceptually transparent decision procedure. This contrasts with the predictions concerning a hypothetical borderline candidate which typify Angoff procedures. Furthermore, the iterative process involved in the Angoff standard setting task incurs financial and administrative burdens, thus creating the potential to cut corners through recruiting fewer judges. The underlying objective of homogenizing the test standard undermines its validity, while circumventing reputable standard setting principles. While the Rasch model offers an objective approach to predicting successful outcomes, combining Rasch and Angoff procedures does not resolve the validity problem for Angoff-based pass marks. This commentary highlights the virtues of OSS relative to the modified Angoff approach in the standard setting of itemized tests. It also identifies gaps in the research literature that should be addressed to strengthen the case for using OSS on an international scale for high-stakes assessments within healthcare disciplines as a testing ground for other disciplines.

Cross-cultural adaptation and validation of the health literacy assessment tool METER in the Portuguese adult population

ObjectiveWe aimed to culturally adapt and validate METER in the Portuguese population, and to define cut-off values for adequate health literacy. MethodsWe used the standard procedure for the adaptation of the words and surveyed health professionals to select the non-words. The instrument was administered to a total sample of 249 participants and retested in a sub-sample of 45 after three months. Cut-offs were defined using the modified Angoff procedure. Construct validity was assessed through association with educational attainment and health-related occupation. ResultsExploratory factor analysis revealed two dimensions of the instrument, one for words and another for non-words. METER showed a high degree of internal consistency, and acceptable test–retest reliability. Adequate health literacy was defined as scoring at least 35/40 in words and 18/30 in non-words. Physicians scored higher than any other group, followed by health researchers, researchers from other areas and by people with progressively lower levels of education (p<0.001). ConclusionWe culturally adapted a brief and simple instrument for health literacy assessment, and showed it was valid and reliable. Practice implicationsThe Portuguese version of METER can be used to assess health literacy in Portuguese adults and to explore associations with health outcomes.

Musculoskeletal Ultrasound Training and Competency Assessment Program for Rheumatology Fellows

The purpose of this study was to establish standards for musculoskeletal ultrasound competency through knowledge and skills testing using criterion-referenced methods. Two groups of rheumatology fellows trained in musculoskeletal ultrasound through a standardized curriculum, which required submission of ultrasound studies for review over 8 months. Both groups then completed written and practical examinations in musculoskeletal ultrasound. Instructors, advanced users, and intermediate users of musculoskeletal ultrasound served as comparison groups. A passing score (competency) was established for the written examination by the Angoff procedure and for the practical examination by the borderline method. Thirty-eight fellows (19 in each group) took the final examination. Five fellows failed the written examination, and 1 failed the practical examination, whereas none of the advanced users failed. Written examination scores did not differ between the two fellow groups (74% versus 70%; P > .05), were reliable, and were able to discriminate between the intermediate and advanced groups. Practical and written examination results correlated in both groups (first group, r = 0.70; P = .0008; second group, r = 0.59; P = .009). Criterion-referenced methods were used for the first time to determine fellow musculoskeletal ultrasound competency. The examination used to determine competency was reproducible, was reliable, and could differentiate musculoskeletal ultrasound users with different levels of experience. Most rheumatology fellows completing our program passed the written and practical examinations, suggesting achievement of basic musculoskeletal ultrasound competency.

An application of Multifaceted Rasch measurement in the Yes/No Angoff standard setting procedure

When implementing standard setting procedures, there are two major concerns: variance between panelists and efficiency in conducting multiple rounds of judgments. With regard to the former, there is concern over the consistency of the cutoff scores made by different panelists. If the cut scores show an inordinately wide range then further rounds of group discussion are required to reach consensus, which in turn leads to the latter concern. The Yes/No Angoff procedure is typically implemented across several rounds. Panelists revise their original decisions for each item based on discussion with co-panelists between each round. The purpose of this paper is to demonstrate a framework for evaluating the judgments in the standard setting process. The Multifaceted Rasch model was applied as a tool to evaluate the quality of standard setting in a context of language assessment. The results indicate that the Multifaceted Rasch model offers a promising approach to examination of the variability in the standard setting procedures. In addition, this model can identify aberrant decision making for each panelist, which can be used as feedback for both standard setting designers and panelists.

Judges' Use of Examinee Performance Data in an Angoff Standard‐Setting Exercise for a Medical Licensing Examination: An Experimental Study

Although the Angoff procedure is among the most widely used standard setting procedures for tests comprising multiple‐choice items, research has shown that subject matter experts have considerable difficulty accurately making the required judgments in the absence of examinee performance data. Some authors have viewed the need to provide performance data as a fatal flaw for the procedure; others have considered it appropriate for experts to integrate performance data into their judgments but have been concerned that experts may rely too heavily on the data. There have, however, been relatively few studies examining how experts use the data. This article reports on two studies that examine how experts modify their judgments after reviewing data. In both studies, data for some items were accurate and data for other items had been manipulated. Judges in both studies substantially modified their judgments whether the data were accurate or not.

The ACGME Resident Survey Aggregate Reports: An Analysis and Assessment of Overall Program Compliance

The Accreditation Council for Graduate Medical Education (ACGME) uses a 29-question Resident Survey for yearly residency program assessments. This article describes methodology for aggregating Resident Survey data into 5 discrete areas of program performance for use in the accreditation process. This article also describes methodology for setting thresholds that may assist Residency Review Committees in identifying programs with potential compliance problems. A team of ACGME staff and Residency Review Committee chairpersons reviewed the survey for content and proposed thresholds (through a modified Angoff procedure) that would indicate problematic program functioning. Interrater agreement was high for the 5 content areas and for the threshold values (percentage of noncompliant residents), indicating that programs above these thresholds may warrant follow-up by the accrediting organization. Comparison of the Angoff procedure and the actual distribution of the data revealed that the Angoff thresholds were extremely similar to 1 standard deviation above the content area mean. Data from the ACGME Resident Survey may be aggregated into internally consistent and consensually valid areas that may help Residency Review Committees make more targeted and specific judgments about program compliance.

Performance and views of examiners in the Applied Knowledge Test for the nMRCGP licensing examination

A new computer-based Applied Knowledge Test (AKT) has been developed for the licensing examination for general practice administered by the Royal College of General Practitioners. The aim of this evaluation was to assess the acceptability, feasibility, and validity of the test as well as its transfer to a computerised format at local test centres. Computer-based test and postal questionnaire. Panel of examiners, Membership of the Royal College of General Practitioners (MRCGP) examination, UK. Self-administered postal questionnaires were sent to examiners not involved with the development of the test after completing it. Their performance scores were compared with those of candidates. The majority of participants (80.9%) were satisfied with the new computer-based test. Responses relating to content and attitudes to the test were also positive overall, but some problems with content were highlighted. Fewer examiners (61.9%) were positive about the physical comfort of the test centre, including seating, heating, and lighting. Examiners had significantly higher scores (mean 83.3%, range 69 to 93%, 95% confidence interval [CI] = 81.9 to 84.7%) than 'real' candidates (mean 75.0%, range 45 to 94%, 95% CI = 74.6 to 75.5%), who subsequently took an identical test. The new computer-based licensing test (the AKT) was found to be acceptable to the majority of examiners. The pass-fail standard, determined by routine methods including an Angoff procedure, was supported by the higher success rate of examiners compared with candidates. The use of selected groups to assess high-stakes (licensing) examinations can be useful for assessing test validity.

Validity Evidence for an OSCE to Assess Competency in Systems-Based Practice and Practice-Based Learning and Improvement: A Preliminary Investigation

To determine the psychometric properties and validity of an OSCE to assess the competencies of Practice-Based Learning and Improvement (PBLI) and Systems-Based Practice (SBP) in graduate medical education. An eight-station OSCE was piloted at the end of a three-week Quality Improvement elective for nine preventive medicine and endocrinology fellows at Mayo Clinic. The stations assessed performance in quality measurement, root cause analysis, evidence-based medicine, insurance systems, team collaboration, prescription errors, Nolan's model, and negotiation. Fellows' performance in each of the stations was assessed by three faculty experts using checklists and a five-point global competency scale. A modified Angoff procedure was used to set standards. Evidence for the OSCE's validity, feasibility, and acceptability was gathered. Evidence for content and response process validity was judged as excellent by institutional content experts. Interrater reliability of scores ranged from 0.85 to 1 for most stations. Interstation correlation coefficients ranged from -0.62 to 0.99, reflecting case specificity. Implementation cost was approximately $255 per fellow. All faculty members agreed that the OSCE was realistic and capable of providing accurate assessments. The OSCE provides an opportunity to systematically sample the different subdomains of Quality Improvement. Furthermore, the OSCE provides an opportunity for the demonstration of skills rather than the testing of knowledge alone, thus making it a potentially powerful assessment tool for SBP and PBLI. The study OSCE was well suited to assess SBP and PBLI. The evidence gathered through this study lays the foundation for future validation work.

Is an Angoff Standard an Indication of Minimal Competence of Examinees or of Judges?

To establish credible, defensible and acceptable passing scores for written tests is a challenge for health profession educators. Angoff procedures are often used to establish pass/fail decisions for written and performance tests. In an Angoff procedure judges' expertise and professional skills are assumed to influence their ratings of the items during standard-setting. The purpose of this study was to investigate the impact of judges' item-related knowledge on their judgement of the difficulty of items, and second, to determine the stability of differences between judges. Thirteen judges were presented with two sets of 60 items on different occasions. They were asked to not only judge the difficulty of the items but also to answer them, without the benefit of the answer key. For each of the 120 items an Angoff estimate and an item score were obtained. The relationship between the Angoff estimate and the item score was examined by applying a regression analysis to the 60 items (Angoff estimate, score) for each judge at each occasion. This study shows that in standard-setting the individual judgement of the individual item is not only a reflection of the difficulty of the item but also of the inherent stringency of the judge and his/her subject-related knowledge. Considerable variation between judges in their stringency was found, and Angoff estimates were significantly affected by a judge knowing or not knowing the answer to the item. These findings stress the importance of a careful selection process of the Angoff judges when making pass/fail decisions in health professions education. They imply that judges should be selected who are not only capable of conceptualising the 'minimally competent student', but who would also be capable of answering all the items.

The Bookmark Procedure for Setting Cut-Scores and Finalizing Performance Standards: Strengths and Weaknesses

The Bookmark standard-setting procedure was developed to address the perceived problems with the most popular method for setting cut-scores: the Angoff procedure (Angoff, 1971). The purposes of this article are to review the Bookmark procedure and evaluate it in terms of Berk’s (1986) criteria for evaluating cut-score setting methods. The strengths and weaknesses of the Bookmark are critically examined and discussed. In general, the strengths of the Bookmark method are that it (a) accommodates constructed-response as well as selected-response test items; (b) efficiently accommodates multiple cut-scores and multiple test forms; and (c) reduces cognitive complexity for panelists. Despite unresolved issues like the choice and understanding of the response probability, the Bookmark method remains a promising procedure for setting cut-scores and finalizing performance standards.

Re-Evaluating the NCLEX-RN® Passing Standard

Setting passing standards is a critical component of the NCLEX examination process. This research was conducted to provide sufficient information to the National Council of State Boards of Nursing's (NCSBN) Board of Directors to,make a decision regarding the passing standard of the NCLEX-RN. This article illustrates the standard setting process that NCSBN uses. Surveys of educators and employers, a modified Angoff procedure, the Beuk compromise, and global assessments by content experts were methods used. The Rasch model and a presumed ability distribution were used as the framework to integrate these diverse perspectives regarding minimal competence. The revised passing standard was -0.28 logits. For many of the minimal competence estimates, the author did not have authorization to release the information. In those instances, estimates and results were fabricated to be similar to the actual results, yet different enough as to not disclose confidential information. The fabricated results are clearly marked. In conclusion, a variety of approaches, sources, and perspectives are necessary for the establishment of fair and appropriate standards on the NCLEX-RN.

Deficient knowledge of genetics relevant for daily practice among medical students nearing graduation

PurposeThe objective of this study was to investigate whether the knowledge of genetics relevant for daily practice among medical students nearing graduation in the Netherlands was sufficient to react appropriately to the change of relevance of genetics in medicine. MethodsA computer examination validated in a group of clinical geneticists, medical students nearing graduation, and nonmedical students. The examination consisted of 215 genetic questions classified by the designers into three categories of relevance: “essential” knowledge (requirement: > 95% correct answers), “desirable” knowledge (requirement: > 60% correct answers), and “too specialized” knowledge (no requirement). To set an independent standard, the questions were also judged by clinical geneticists and nongenetic health care providers in an Angoff procedure. In total, 291 medical students nearing graduation from seven out of the eight medical schools in the Netherlands participated. ResultsAs expected, the mean score for “essential” knowledge (71.63%, 95% CI 70.74–72.52) was higher than for “desirable” knowledge (55.99%, 95% CI 55.08–56.90); the mean score for “too specialized” knowledge (44.40%, 95% CI 43.19–45.62) was the lowest. According to passing scores set for “essential” knowledge as defined by the designers, the clinical geneticists, and the nongenetic health care providers, only 0%, 26%, and 3%, respectively, of the participants would have passed. ConclusionsMedical students nearing graduation lack genetic knowledge that is essential for daily practice. Therefore, changes should be made in the medical curricula.

Do students have sufficient knowledge of clinical anatomy?

Comparisons of anatomy knowledge levels of students from various curricula show either no differences or small differences to the detriment of innovative schools. To pass judgement on the general level of students' anatomy knowledge, we need an absolute standard. The purpose of this study was to compare students' levels of anatomy knowledge as measured by a case-based anatomy test with standards set by different groups of experts. A modified Angoff procedure was used to establish an absolute standard against which the students' results could be evaluated. Four panels of 9 anatomists, 7 clinicians, 9 recent graduates and 9 Year 4 students, respectively, judged 107 items of an anatomy test. The students' results on these items were compared with the standards obtained by the panels. If the standard established by the panel of Year 4 students was used, 64% of the students would fail the test. The standards established by the anatomists, clinicians and recent graduates would yield failure rates of 42%, 58% and 26%, respectively. According to the panels' standards, many students did not know enough about anatomy. The high expectations that the Year 4 students appeared to have of their peers may contribute to students' uncertainty about their level of anatomy knowledge.

Competence in the musculoskeletal system: assessing the progression of knowledge through an undergraduate medical course

Professional bodies have expressed concerns that medical students lack appropriate knowledge in musculoskeletal medicine despite its high prevalence of use within the community. Changes in curriculum and teaching strategies may be contributing factors to this. There is little evidence to evaluate the degree to which these concerns are justified. To design and evaluate an assessment procedure that tests the progress of medical students in achieving a core level of knowledge in musculoskeletal medicine during the course. A stratified sample of 136 volunteer students from all 5 years of the medical course at Sheffield University. The progress test concept was adapted to provide a cross-sectional view of student knowledge gain during each year of the course. A test was devised which aimed to provide an assessment of competence set at the standard required of the newly qualified doctor in understanding basic and clinical sciences relevant to musculoskeletal medicine. The test was blueprinted against internal and external guidelines. It comprised 40 multiple-choice and extended matching questions administered by computer. Six musculoskeletal practitioners set the standard using a modified Angoff procedure. Test reliability was 0.6 (Cronbach's alpha). Mean scores of students increased from 41% in Year 1 to 84% by the final year. Data suggest that, from a baseline score in Year 1, there is a disparate experience of learning in Year 2 that evens out in Year 3, with knowledge progression becoming more consistent thereafter. All final year participants scored above the standard predicted by the Angoff procedure. This short computer-based test was a feasible method of estimating student knowledge acquisition in musculoskeletal medicine across the undergraduate curriculum. Tested students appear to have acquired a satisfactory knowledge base by the end of the course. Knowledge gain seemed relatively independent of specialty-specific clinical training. Proposals from specialty bodies to include long periods of disciplinary teaching may be unnecessary.

Setting a standard for performance assessment of doctor-patient communication in general practice

Continuing professional development (CPD) of general practitioners. Criterion-referenced standards for assessing performance in the real practice of general practitioners (GPs) should be available to identify learning needs or poor performers for CPD. The applicability of common standard setting procedures in authentic assessment has not been investigated. To set a standard for assessment of GP-patient communication with video observation of daily practice, we investigated 2 well known examples of 2 different standard setting approaches. An Angoff procedure was applied to 8 written cases. A borderline regression method was applied to videotaped consultations of 88 GPs. The procedures and outcomes were evaluated by the applicability of the procedure, the reliability of the standards and the credibility as perceived by the stakeholders, namely, the GPs. Both methods are applicable and reliable; the obtained standards are credible according to the GPs. Both modified methods can be used to set a standard for assessment in daily practice. The context in which the standard will be used - i.e. the specific purpose of the standard, the moment the standard must be available or if specific feedback must be given - is important because methods differ in practical aspects.