Modified Angoff Method Research Articles

Implementation of standard setting for high-stakes objective structured clinical examinations

Background and purposeTo describe one institution's approach to transformation of high-stakes objective structure clinical examinations (OSCEs) from norm-referenced to criterion-referenced standards setting and to evaluate the impact of these changes on OSCE performance and pass rates. Educational activity and settingThe OSCE writing team at the college selected a modified Angoff method appropriate for high-stakes assessments to replace the two standard deviation method previously used. Each member of the OSCE writing team independently reviewed the analytical checklist and calculated a passing score for active stations on OSCEs. Then the group met to determine a final pass score for each station. The team also determined critical cut points for each station, when indicated. After administration of the OSCEs, scores, pass rates, and need for remediation were compared to the previous norm-referenced method. Descriptive statistics were used to summarize the data. FindingsOSCE scores remained relatively unchanged when switched to a criterion-referenced method, but the number of remediators increased up to 2.6 fold. In the first year, the average score increased from 86.8% to 91.7% while the remediation rate increased from 2.8% to 7.4%. In the third year, the average increased from 90.9% to 92% while the remediation rate increased from 6% to 15.6%. Likewise, the fourth-year average increased from 84.9% to 87.5% while the remediation rate increased from 4.4% to 9%. SummaryTransition to a modified Angoff method did not impact average OSCE score but did increase the number of remediations.

Establishing performance standards for child development: learnings from the ECDI2030

BackgroundStandards of early childhood development (ECD) are needed to determine whether children living in different contexts are developmentally on track. The Early Childhood Development Index 2030 (ECDI2030) is a population-level measure intended to be used in household surveys to collect globally comparable data on one of the indicators chosen to monitor progress toward target 4.2 of the Sustainable Development Goals: The proportion of children aged 24–59 months who are developmentally on track in health, learning and psychosocial well-being.MethodsTo define performance cut-scores for the ECDI2030 we followed a criterion-referenced standard setting exercise using the modified Angoff method. The exercise gauged the expectations from 15 global experts in ECD and was informed by representative population data collected in Mexico and the State of Palestine. The final calibrated age-specific performance cut-scores were applied to these data to estimate the proportion of children developmentally on track, disaggregated by background characteristics, including the child's sex and attendance to early childhood education.ResultsThrough a process of standard setting, we generated robust performance standards for the ECDI2030 by establishing five age-specific cut-scores to identify children as developmentally on track.ConclusionsThis paper demonstrated how the standard setting methodology, typically applied to measures in the health and education fields, could be applied to a measure of child development. By creating robust criterion-referenced standards, we have been able to ensure that the cut-scores related to age for the ECDI2030 are based on performance standards set by global experts in the ECD field for defining on and off track development.

Automated Item Generation: impact of item variants on performance and standard setting

BackgroundAutomated Item Generation (AIG) uses computer software to create multiple items from a single question model. There is currently a lack of data looking at whether item variants to a single question result in differences in student performance or human-derived standard setting. The purpose of this study was to use 50 Multiple Choice Questions (MCQs) as models to create four distinct tests which would be standard set and given to final year UK medical students, and then to compare the performance and standard setting data for each.MethodsPre-existing questions from the UK Medical Schools Council (MSC) Assessment Alliance item bank, created using traditional item writing techniques, were used to generate four ‘isomorphic’ 50-item MCQ tests using AIG software. Isomorphic questions use the same question template with minor alterations to test the same learning outcome. All UK medical schools were invited to deliver one of the four papers as an online formative assessment for their final year students. Each test was standard set using a modified Angoff method. Thematic analysis was conducted for item variants with high and low levels of variance in facility (for student performance) and average scores (for standard setting).ResultsTwo thousand two hundred eighteen students from 12 UK medical schools participated, with each school using one of the four papers. The average facility of the four papers ranged from 0.55–0.61, and the cut score ranged from 0.58–0.61. Twenty item models had a facility difference > 0.15 and 10 item models had a difference in standard setting of > 0.1. Variation in parameters that could alter clinical reasoning strategies had the greatest impact on item facility.ConclusionsItem facility varied to a greater extent than the standard set. This difference may relate to variants causing greater disruption of clinical reasoning strategies in novice learners compared to experts, but is confounded by the possibility that the performance differences may be explained at school level and therefore warrants further study.

Evaluation of the Prescribing Skills Assessment implementation, performance and medical student experience in Australia and New Zealand.

The UK Prescribing Safety Assessment was modified for use in Australia and New Zealand (ANZ) as the Prescribing Skills Assessment (PSA). We investigated the implementation, student performance and acceptability of the ANZ PSA for final-year medical students. This study used a mixed-method approach involving student data (n= 6440) for 2017-2019 (PSA overall score and 8 domain subscores). Data were also aggregated by medical school and included student evaluation survey results. Quantitative data were analysed using descriptive and multivariate analyses. The pass rate was established by a modified Angoff method. Thematic analyses of open-ended survey comments were conducted. The average pass rate was slightly higher in 2017 (89%) which used a different examination to 2018 (85%) and 2019 (86%). Little difference was identified between schools for the PSA overall performance or domain subscores. There was low intercorrelation between subscores. Most students provided positive feedback about the PSA regarding the interface and clarity of questions, but an average of 35% reported insufficient time for completion. Further, 70% on average felt unprepared by their school curricula for the PSA, which is in part explained by the low prescribing experience; 69% reported completing ≤10 prescriptions during training. The ANZ PSA was associated with high pass rates and acceptability, although student preparedness was highlighted as a concern for further investigation. We demonstrate how a collaboration of medical schools can adapt a medical education assessment resource (UK PSA) as a means for fulfilling an unmet need.

중학교 수리적성 영역 검사에 대한 Modified-Angoff와 Bookmark 준거설정 방법 비교

본 연구의 목적은 두 가지 준거설정 방법을 비교 분석하여 학교 현장에서의 적용 가능성을 탐색하고자 하는 것으로, 다양한 준거설정 방법들 중에서 현재까지 보편적으로 사용되고 있는 Modified-Angoff 방법과 문항반응이론을 적용한 Bookmark 방법을 사용하여 분할점수를 산출 및 비교하였다. 검사자료는 ‘종합진로직업적성검사(CACV) 중학생용(성태제 외, 2016)’을 사용하였으며 성취수준은 기초미달, 기초, 숙달의 3개 범주로 분류하였다. Modified-Angoff 방법과 Bookmark 방법으로 준거설정을 실시한 결과, Modified-Angoff 방법의 분할점수가 더 높았으며, 1차 평정에 비해 2차 평정에서 준거설정자 간 점수의 표준편차가 줄어들었다. 또한 준거설정자 내 일치도는 Modified-Angoff 방법에서, 준거설정자 간 일치도는 Bookmark 방법에서 더 높았다. 준거설정 전반에 대한 설문 결과, Modified-Angoff 방법과 Bookmark 방법에 대한 선호도가 유사하였으며, 준거설정자의 자격 요건에 관한 질문에는 현장 경력이 있는 교사를 가장 적합하게 여기는 것으로 나타났다. 결론적으로 학교 교사들을 준거설정자로 선정하여 준거를 설정함에 있어서 Modified-Angoff 방법과 Bookmark 방법이 실제적으로 적용하기에 용이하였BR다. 따라서 국내외 교육 분야에서 보편적으로 사용되는 Modified-Angoff 방법과 문항반응이론에 기초한 Bookmark 방법을 사용하고, 두 방법에 의한 결과를 비교하여 볼 것을 권장한다. 특히 Bookmark 방법은 문항별로 준거를 설정하는 것이 아니라 문항순서집에 의하여 준거점수를 결정하므로 준거설정자 간의 점수 차이가 적어 보다 안정적인 준거를 설정할 수 있다. 단, Bookmark 방법을 원활히 적용하기 위해서는 문항반응이론에 대한 기초적인 이해가 전제되어야 한다.

Are we passing the acceptable? Standard setting of theoretical proficiency tests for foreign-trained dentists.

Dentists educated outside the EU, EEA and Switzerland have the option to undergo proficiency testing to practice dentistry in Sweden. A standardised evidence-based procedure for proficiency tests is crucial. This paper explores if the standard setting of the theoretical examination in the Swedish proficiency test for foreign dentists is fit to differentiate candidates who meet the acceptable standard for licensure from those who do not, by analysis of inter-rater reliability and credibility of Angoff panels and the test's ability to differentiate candidates regarding learning outcomes and dental disciplines. In a cohort study design, test results were collected from seven examinations during 2018-2019 (n=316), combined with Angoff judgements and demographic data from The National Board of Health and Welfare. Intraclass correlation, Pearson correlation and linear regression were calculated. All exams demonstrated substantial judge inter-rater reliability using the Angoff procedure (0.65-0.82). Item difficulty predicted by the expert panels showed significant positive correlations with the item difficulty by candidate results (0.69-0.77). Candidates who did not meet the requirements for pass mark in the theoretical test revealed a significantly reduced performance overall in professional qualifications (i.e., learning outcomes) and reduced performance within dental disciplines. The results indicate that a modified Angoff method is a feasible, simple, and defensible method for setting the pass mark for the theoretical examinations. The standard setting appears to differentiate candidates who do not meet the requirements of the professional qualifications of the dental degree. Adjustments could be undertaken to further discriminate for competence within single dental disciplines.

Early detection measures: addressing Unemployability using Modified Angoff in situational judgment test

This is the first standard setting study in Malaysia aimed at developing Performance-Level Descriptors (PLDs) and determining cut scores that differentiate undergraduates with low, moderate and high employability skills using a newly developed Situational judgment Test (SJT). SJT items from five domains critical for future job employment: 1) critical thinking and problem solving, 2) enterprising, 3) integrity, 4) leadership and 5) teamwork were developed. Modified Angoff method involving PLDs and cut scores were developed by five subject-matter experts. Two test instruments were used: 10 practice and 36 original bilingual Malay-English SJT items. The PLDs and two cut scores to distinguish low, moderate and high employability skills were validly developed, without compromising consequential validity. The findings benefit undergraduates with low employability as they can be trained with current global job readiness skills by the universities, which also forms the practice implication of this study. Another practical implication is academicians can be further encouraged to understand the theoretical framework of establishing standard setting for employability.

A Competency-Based Approach to Pass/Fail Decisions: An Observational Study

Any high-stakes assessment that leads to an important decision requires careful consideration in determining whether a student passes or fails. Despite the implementation of many standard-setting methods in clinical examinations, concerns remain about the reliability of pass/fail decisions in high stakes assessment, especially clinical assessment. This observational study proposes a defensible pass/fail decision based on the number of failed competencies. In the study conducted in Erbil, Iraq, in June 2018, results were obtained for 150 medical students on their final objective structured clinical examination. Cutoff scores and pass/fail decisions were calculated using the modified Angoff, borderline, borderline-regression, and holistic methods. The results were compared with each other and with a new competency method using Cohen’s kappa. Rasch analysis was used to compare the consistency of competency data with Rasch model estimates. The competency method resulted in 40 (26.7%) students failing, compared with 76 (50.6%), 37 (24.6%), 35 (23.3%), and 13 (8%) for the modified Angoff, borderline, borderline regression, and holistic methods, respectively. The competency method demonstrated a sufficient degree of fit to the Rasch model (mean outfit and infit statistics of 0.961 and 0.960, respectively). In conclusion, the competency method was more stringent in determining pass/fail, compared with other standard-setting methods, except for the modified Angoff method. The fit of competency data to the Rasch model provides evidence for the validity and reliability of pass/fail decisions.

A comparison of different standard-setting methods for professional qualifying dental examination.

The outcome of assessments is determined by the standard-setting method used. Standard setting is the process of deciding what is good enough. A cutoff score of 50% was commonly used in dental schools in Malaysia. This study aims to compare the conventional, norm-referenced, and modified-Angoff standard-setting methods. The norm-referenced method of standard setting was applied to the real scores of 40 final-year dental students on a multiple-choice question (MCQ), a short answer question (SAQ), and an objective structured clinical examination (OSCE). A panel of 10 judges set the standard using the modified-Angoff method for the same paper in one sitting. One judge set the passing score of 10 OSCE questions after 2weeks. A comparison of the grades and pass/fail rates derived from the absolute standard, norm-referenced, and modified-Angoff methods was made. The intra-rater and inter-rater reliabilities of the modified-Angoff method were assessed. The passing rate for the absolute standard was 100% (40/40), for the norm-referenced method it was 62.5% (25/40), and for the modified-Angoff method it was 80% (32/40). The modified-Angoff method had good inter-rater reliability of 0.876 and excellent test-retest reliability of 0.941. There were significant differences in the outcomes of these three standard-setting methods, as shown by the difference in the proportion of candidates who passed and failed the assessment. The modified-Angoff method was found to have good reliability for use with a professional qualifying dental examination.

A Novel Application of the Modified Angoff Method to Rate Case Difficulty in Simulation-Based Research.

Simulation use in research is often limited by controlling for scenario difficulty when using repeated measures. Our study assesses the feasibility of the Modified Angoff Method to reach expert consensus regarding difficulty of medical simulations. We compared scores with participant physiologic stress. Emergency medicine physicians with expertise in simulation education were asked to review 8 scenarios and estimate the percentage of resident physicians who would perform all critical actions using the modified Angoff method. A standard deviation (SD) of less than 10% of estimated percentage correct signified consensus. Twenty-five residents then performed the 6 scenarios that met consensus and heart rate variability (HRV) was measured. During round 1, experts rated 4/8 scenarios within a 10% SD for postgraduate year 3 (PGY3) and 3/8 for PGY4 residents. In round 2, 6/8 simulation scenarios were within an SD of 10% points for both years. Intraclass correlation coefficient was 0.84 for PGY3 ratings and 0.89 for PGY4 ratings. A mixed effects analysis of variance showed no significant difference in HRV change from rest to simulation between teams or scenarios. Modified Angoff Score was not a predictor of HRV (multiple R2 = 0.0176). Modified Angoff ratings demonstrated consensus in quantifying the estimated percentage of participants who would complete all critical actions for most scenarios. Although participant HRV did decrease during the scenarios, we were unable to significantly correlate this with ratings. This modified Angoff method is a feasible approach to evaluate simulation difficulty for educational and research purposes and may decrease the time and resources necessary for scenario piloting.

It's Not Just Angoff: Misperceptions of Hard and Easy Items in Bookmark‐Type Ratings

AbstractA common belief is that the Bookmark method is a cognitively simpler standard‐setting method than the modified Angoff method. However, a limited amount of research has investigated panelist's ability to perform well the Bookmark method, and whether some of the challenges panelists face with the Angoff method may also be present in the Bookmark method. This article presents results from three experiments where panelists were asked to give Bookmark‐type ratings to separate items into groups based on item difficulty data. Results of the experiments showed, consistent with results often observed with the Angoff method, that panelists typically and paradoxically perceived hard items to be too easy and easy items to be too hard. These perceptions were reflected in panelists often placing their Bookmarks too early for hard items and often placing their Bookmarks too late for easy items. The article concludes with a discussion of what these results imply for educators and policymakers using the Bookmark standard‐setting method.

Comparing Standard Setting Methods for Objective Structured Clinical Examinations in a Caribbean Medical School.

Background:OSCE are widely used for assessing clinical skills training in medical schools. Use of traditional pass fail cut off yields wide variations in the results of different cohorts of students. This has led to a growing emphasis on the application of standard setting procedures in OSCEs.Purpose/aim:The purpose of the study was comparing the utility, feasibility and appropriateness of 4 different standard setting methods with OSCEs at XUSOM.Methods:A 15-station OSCE was administered to 173 students over 6 months. Five stations were conducted for each organ system (Respiratory, Gastrointestinal and Cardiovascular). Students were assessed for their clinical skills in 15 stations. Four different standard setting methods were applied and compared with a control (Traditional method) to establish cut off scores for pass/fail decisions.Results:OSCE checklist scores revealed a Cronbach’s alpha of 0.711, demonstrating acceptable level of internal consistency. About 13 of 15 OSCE stations performed well with “Alpha if deleted values” lower that 0.711 emphasizing the reliability of OSCE stations. The traditional standard setting method (cut off score of 70) resulted in highest failure rate. The Modified Angoff Method and Relative methods yielded the lowest failure rates, which were typically less than 10% for each system. Failure rates for the Borderline methods ranged from 28% to 57% across systems.Conclusions:In our study, Modified Angoff method and Borderline regression method have shown to be consistently reliable and practically suitable to provide acceptable cut-off score across different organ system. Therefore, an average of Modified Angoff Method and Borderline Regression Method appeared to provide an acceptable cutoff score in OSCE. Further studies, in high-stake clinical examinations, utilizing larger number of judges and OSCE stations are recommended to reinforce the validity of combining multiple methods for standard setting.

Validation of the SYdney Shared and Evidence-based decision-Making test (SYStEM) among healthcare professionals

PurposeWhilst competence in evidence-based practice (EBP) is a critical skill for healthcare professionals, there is a gap in the literature in the assessment of EBP. Current frameworks in Evidence-Based Practice education cite a five-step process (Ask, Acquire, Appraise, Integrate and Evaluate) yet many existing measures of competence focus on the first three, relying on self-report and cognitive testing, falling short of assessing skills. The Fresno Test has been validated for assessing the first three steps and the “Assessing Communication about Patient Preferences” tool is reliable in capturing the quality of information given by physicians in integrating evidence. The Evidence-Based Practice Implementation Scale is useful for the ‘evaluate’ step in the EBP process. We adapted components from all three of these tools to create the SYdney Shared and Evidence-based decision-Making test (SYStEM) in order to measure EBP and Shared Decision Making (SDM) competencies in the context of patient care. The aims of this study were to examine the internal structure, criterion and consequential validity of the SYStEM. MethodData were available from 37 healthcare practitioners who completed the SYStEM online. We conducted a confirmatory factor analysis to determine the theoretical structure and a modified Angoff method to determine a minimum competence level for newly graduated health professionals. ResultsThe final 10-item SYStEM consisted of two subscales: ‘EBP knowledge’ and ‘SDM skills’ demonstrating good construct validity and reliability (α > 0.74). The factorial validity of SYStEM was shown by all items loading significantly on their expected factors and a good fit to the data. Although the benchmark for passing the SYStEM was low, only 13/37 practitioners (35%) reached the minimum standard for EBP competence. DiscussionThis study provides preliminary evidence for the validity the SYStEM tool for measuring EBP competence amongst health graduates. Further large-scale validation of SYStEM is required.

การเทียบผลคะแนนแบบทดสอบสมิทธิภาพภาษาอังกฤษ แห่งมหาวิทยาลัยศรีนครินทรวิโรฒกับกรอบมาตรฐานอ้างอิงสากลของยุโรป

This study used standard setting to map SWU-SET scores onto the CEFR by employing Modified Angoff method. Fourteen experts judged each test item in the Srinakharinwirot University - Standardized English Test (SWU-SET) against the Common European Framework of Reference for Languages (CEFR) in order to recommend cut-off scores covering A2, B1, and B2 levels. The results showed that, based on 100 test items in the SWU-SET, test takers should achieve at A2 level if the scores range from 22-49, at B1 level from 50-77, and at B2 from 78 or above.

Ensuring basic competency in chest tube insertion using a simulated scenario: an international validation study

IntroductionChest tube insertion can be associated with serious complications. A structured training programme is needed to minimise complications and enhance patient safety. Novices should pass a reliable test with solid...

Competency-based Standard Setting for a High-stakes Objective Structured Clinical Examination (OSCE): Validity Evidence.

This article was migrated. The article was marked as recommended. Introduction Medical educators need to demonstrate that their trainees meet expected competency levels when progressing through medical education. This study aimed to develop competency-based pass/fail cut-scores for a graduation required Objective Structured Clinical Examination (OSCE), and examine validity evidence for new standards. Methods Six clinicians used the modified Angoff method to determine the cut-scores for an 8-station OSCE. The clinicians estimated the percentage of minimally competent students who would answer each checklist item correctly. Inter-rater reliability, differences in other academic achievements between pass/fail groups, educational impact, and response process were examined. Results One hundred seventy-four rising 4th-year medical students participated in the OSCE. The cut-scores determined for the OSCE resulted in a substantially lower failure rate (5% vs. 29% of the previous year). The inter-rater reliability across domains and cases was .98 (95% CI = .97 - .99). The pass/fail groups significantly differed in six of the eight measures of academic achievements included in the study. Discussion The impact of the standards setting was substantial as it significantly reduced the failure rate and burdens of remediation for both students and faculty. The very high inter-rater reliability indicates that the modified Angoff method produced reliable cut-scores. The significant differences between the pass/fail groups in other measures support external validity of the standards and ensure no false passes. The study also supports response process validity by including discussion among judges and check of previous student performances, as well as recruiting and training multiple clinician educators experienced in medical student teaching. Conclusion Findings of the study provide strong evidence supporting validity of the new cut-scores from a wide spectrum of validity metrics, including response process, internal structure, relations to other variables, and consequences. The study also added to the literature the value of the modified Angoff method in determining competency-based standards for OSCEs.

Optimizing the amount of simulation training used to teach vaginal delivery skills to medical students.

To determine the amount of simulation training required for students to attain minimal competence and mastery of a vaginal delivery. An observational study was conducted at a US medical school between May 11, 2015, and May 8, 2016. Using a modified Angoff method, 10 members of the Obstetrics and Gynecology faculty evaluated a vaginal delivery procedural checklist and established cutoff scores for minimal competence and mastery. During a 5-week period, all third-year students received between two and five 45-minute vaginal delivery simulation sessions; performance was assessed during week 6. Performance according to the checklist was compared. The cutoff score was 20 and 26 out of 30 for minimal competence and mastery, respectively. Among 115 students, mean checklist scores in final assessment rose with increasing number of simulations: 23.6, 25.1, 27.5, and 27.6 points for two, three, four, and five training sessions, respectively (P<0.001). The proportion of patients achieving mastery also increased with number of simulations: 34%, 59%, 73%, and 93% for two, three, four, and five training sessions, respectively (P<0.001). Two or three training sessions were sufficient to attain minimal competence in most students; however, no significant between-group difference was found. Simulation training exerts an increasing effect on performance with each additional session that students receive.

Analysis of 2014 Post UTME Score of Candidates in the University of Ibadan with Two Methods of Standard Setting to Set Cut Off Points

The aim of the current study is to analyse the 2014 Post UTME scores of candidates in the university of Ibadan towards the establishment of cut off using two methods of standard settings. Prospective candidates who seek admission to higher institution are often denied admission through the Post UTME exercise. There is no single recommended approach to standard setting and many methods exist. These include norm-referenced methods and the criterion- referenced methods. The Angoff method is the most widely used and researched criterion-referenced method of standard setting. Two different methods of standard setting were applied on the raw scores of 648 Post-UTME candidates in the University of Ibadan. One of these was the modified Angoff method and the other the norm-reference method of standard setting (mean minus 1 SD). The pass rates derived from the two methods were compared. The result revealed the pass rate with the norm-reference method was 85% (551/648) and that by the Angoff method was 50% (340/648). The percentage agreement between Angoff method and norm-reference was found to be 54% with the Angolf modified method, whilst by the norm-reference method was 85%. Thus, the two standard-setting methods used, produced significantly different outcomes. This was demonstrated by the different pass rates.

Diagnostic Profiles: A Standard Setting Method for Use With a Cognitive Diagnostic Model

This article introduces the Diagnostic Profiles (DP) standard setting method for setting a performance standard on a test developed from a cognitive diagnostic model (CDM), the outcome of which is a profile of mastered and not‐mastered skills or attributes rather than a single test score. In the DP method, the key judgment task for panelists is a decision on whether or not individual cognitive skill profiles meet the performance standard. A randomized experiment was carried out in which secondary mathematics teachers were randomly assigned to either the DP method or the modified Angoff method. The standard setting methods were applied to a test of student readiness to enter high school algebra (Algebra I). While the DP profile judgments were perceived to be more difficult than the Angoff item judgments, there was a high degree of agreement among the panelists for most of the profiles. In order to compare the methods, cut scores were generated from the DP method. The results of the DP group were comparable to the Angoff group, with less cut score variability in the DP group. The DP method shows promise for testing situations in which diagnostic information is needed about examinees and where that information needs to be linked to a performance standard.

Benchmarks for the interpretation of esophageal high-resolution manometry.

Competent interpretation of esophageal high-resolution manometry (HRM) is integral to a quality study. Currently, methods to assess physician competency for the interpretation of esophageal HRM do not exist. The aim of this study was to use formal techniques to (i) develop an HRM interpretation exam, and (ii) establish minimum competence benchmarks for HRM interpretation skills at the trainee, physician interpreter, and master level. A total of 29 physicians from 8 academic centers participated in the study: 9 content experts separated into 2 study groups-expert test-takers (n=7) and judges (n=2), and 20 HRM inexperienced trainees ("trainee test-taker"; n=20). We designed the HRM interpretation exam based on expert consensus. Expert and trainee test-takers (n=27) completed the exam. According to the modified Angoff method, the judges reviewed the test-taker performance and established minimum competency cut scores for HRM interpretation skills. The HRM interpretation exam consists of 22 HRM cases with 8 HRM interpretation skills per case: identification of pressure inversion point, hiatal hernia >3cm, integrated relaxation pressure, distal contractile integral, distal latency, peristaltic integrity, pressurization pattern, and diagnosis. Based on the modified Angoff method, minimum cut scores for HRM interpretation skills at the trainee, physician interpreter, and master level ranged from 65-80%, 85-90% (with the exception of peristaltic integrity), and 90-95%, respectively. Using a formal standard setting technique, we established minimum cut scores for eight HRM interpretation skills across interpreter levels. This examination and associated cut scores can be applied in clinical practice to judge competency.