Comprehensive Osteopathic Medical Licensing Examination Research Articles

Reliability and validity of conversion formulas between comprehensive osteopathic medical licensing examination of the United States level 1 and United States medical licensing examination step 1.

The Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) Level 1 and United States Medical Licensing Examination (USMLE) Step 1 scores are important factors in the selection process of medical students into US residency programs. The goals of this study were to investigate the correlation between the COMLEX-USA Level 1 and the USMLE Step 1 and to assess the accuracy of the existing formulas in predicting USMLE scores from COMLEX-USA scores. A retrospective study of 1016 paired COMLEX-USA Level 1 and USMLE Step 1 scores was conducted. Formulas by Sarko et al and by Slocum and Louder were used to estimate USMLE Step 1 scores from COMLEX-USA Level 1 scores, and a paired t test between calculated USMLE Step 1 scores and actual USMLE Step 1 scores was performed. During 2006-2012, 1016 of 1440 students (71%) took both the USMLE Step 1 and the COMLEX-USA Level 1 tests in the College of Osteopathic Medicine of the Pacific. The USMLE Step 1 scores were higher than those predicted by Slocum and Louder and by Sarko et al by an average of 14.16 ± 11.69 (P < .001) and 7.80 ± 12.48 (P < .001), respectively. A Pearson coefficient of 0.83 was observed. Regression analysis yielded the following formula: USMLE Step 1 = 0.2392 × COMLEX-USA Level 1 + 82.563 (R (2) = 0.69577). The USMLE Step 1 scores, on average, were higher than those predicted by the formulas derived by Slocum and Louder and by Sarko et al. Residency program directors should use caution when using formulas to derive USMLE Step 1 scores from COMLEX-USA Level 1 scores.

The Allopathic Program Director's Dilemma.

The Allopathic Program Director's Dilemma.

Preliminary Outcomes of the Lake Erie College of Osteopathic Medicine's 3-Year Primary Care Scholar Pathway in Osteopathic Predoctoral Education

In 2007, the Lake Erie College of Osteopathic Medicine initiated its Primary Care Scholar Pathway (PCSP), a 3-year osteopathic predoctoral education curriculum. To assess preliminary outcomes of the PCSP curriculum. Scores for the Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) Levels 1 and 2-Cognitive Evaluation (CE) and pass rates for Level 2-Performance Evaluation (PE) were obtained for individuals who graduated from the PCSP program in 2010, 2011, and 2012. Scores for Levels 1 and 2-CE were compared with national mean scores. Acceptance rates for residency programs were also recorded. Nineteen PCSP graduates were included in the study: 3 graduated in 2010, 6 graduated in 2011, and 10 graduated in 2012. Scores for PCSP students were not significantly different than national average scores for COMLEX-USA Levels 1 and 2-CE (P>.05). All 19 PCSP graduates passed the COMLEX-USA Level 2-PE on the first attempt, and all graduates were accepted into primary care residency programs. The COMLEX-USA scores of PCSP graduates were similar to national mean scores, suggesting that it is possible for osteopathic medical students to attain the same level of education as students of 4-year programs in less time. A 3-year osteopathic predoctoral education curriculum would allow students to complete their education at a reduced cost. This potential reduction in debt burden could encourage more students to pursue a primary care career and thus could help address the shortage of primary care physicians in the United States.

Should Osteopathic Students Applying to Allopathic Emergency Medicine Programs Take the USMLE Exam?

IntroductionBoard scores are an important aspect of an emergency medicine (EM) residency application. Residency directors use these standardized tests to objectively evaluate an applicant’s potential and help decide whether to interview a candidate. While allopathic (MD) students take the United States Medical Licensing Examination (USMLE), osteopathic (DO) students take the Comprehensive Osteopathic Medical Licensing Examination (COMLEX). It is difficult to compare these scores. Previous literature proposed an equation to predict USMLE based on COMLEX. Recent analyses suggested this may no longer be accurate. DO students applying to allopathic programs frequently ask whether they should take USMLE to overcome this potential disadvantage. The objective of the study is to compare the likelihood to match of DO applicants who reported USMLE to those who did not, and to clarify how important program directors consider it is whether or not an osteopathic applicant reported a USMLE score.MethodsWe conducted a review of Electronic Residency Application Service (ERAS) and National Resident Matching Program (NRMP) data for 2010–2011 in conjunction with a survey of EM residency programs. We reviewed the number of allopathic and osteopathic applicants, the number of osteopathic applicants who reported a USMLE score, and the percentage of successful match. We compared the percentage of osteopathic applicants who reported a USMLE score who matched compared to those who did not report USMLE. We also surveyed allopathic EM residency programs to understand how important it is that osteopathic (DO) students take USMLE.ResultsThere were 1,482 MD students ranked EM programs; 1,277 (86%, 95% CI 84.3–87.9) matched. There were 350 DO students ranked EM programs; 181 (52%, 95% CI 46.4–57.0) matched (difference=34%, 95% CI 29.8–39.0, p<0.0001). There were 208 DO students reported USMLE; 126 (61%, 95% CI 53.6–67.2) matched. 142 did not report USMLE; 55 (39%, 95% CI 30.7–47.3) matched (difference=22%, 95% CI 11.2–32.5, p<0.0001). Survey results: 39% of program directors reported that it is extremely important that osteopathic students take USMLE, 38% stated it is somewhat important, and 22% responded not at all important.ConclusionDO students who reported USMLE were more likely to match. DO students applying to allopathic EM programs should consider taking USMLE to improve their chances of a successful match.

Board 415 - Research Abstract Build a Better Checklist

Introduction/Background The National Board of Osteopathic Medicine administers the Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) Level 2-Performance Evaluation (PE) to all graduating osteopathic medical students as part of the licensure pathway in the United States. Candidates rotate through 12 standardized patient (SP) cases. A committee of osteopathic physicians creates a checklist (CL) for the SP to document the history and physical after the encounter. Although performance based assessments have become common in both education and high-stakes assessment, case and checklist development remain inadequately described.1-4 Checklists are often based on expert consensus, as there have been few evidence based performance criteria available for consideration. There is emerging literature in patient care that may help to improve checklist development procedures. The purpose of this study is to compare checklists developed through expert consensus to ones developed based on review of the literature and evidence based practices. We hypothesize that checklist items based on evidence will contribute to a more reliable checklist. Methods Two independent groups of case developers worked to develop a new case and checklist for the COMLEX-USA Level 2-PE. Each group contained five to six osteopathic physicians, a standardized patient trainer and a standardized patient. Each checklist item was classified by category or system. For history items the categories were: History of Present Illness (HPI), associated symptoms related to the chief complaint; Review of systems unrelated to the chief complaint; Patient History, including past medical history, surgical history, medications, family history and allergies; Lifestyle factors, including substance use, occupation, etc. Physical examination items were classified as systems: HEENT (Head/Eyes/Ears/Nose/Throat); Cardiovascular; Pulmonary; Gastrointestinal (GI); Genito-urinary (GU); Neurological (Neuro) and Musculoskeletal. This classification helps balance the checklist for content. Each group was then asked to identify a rationale for selecting each item for the checklist. This system for rationalization was loosely based on the principles of SORT taxonomy: A = CL item based on evidence based information (e.g. positive predictive values and likelihood ratios of symptoms and signs with clear patient outcomes); B = CL item based on guidelines for patient care by a respected body (United States Preventive Services Task Force, American College of Osteopathic Family Physicians, American Osteopathic Association, etc.); C = CL based on panel majority consensus. The final checklist (all items from both groups) was placed into a pretesting station in the COMLEX-USA Level 2-PE. Results A total of 25 items (18 History (HX) and 7 physical examination (PE) items) were suggested for the CL (CL “1”). Eleven items were common to both groups (9 HX and 2 PE, CL “2”). Seven items were classified as items with clear patient outcomes based on the review of literature (Category A above, all HX items, CL “3”). Three hundred seventy five students were exposed to the case during the 2012-2013 test cycle. Each student received a checklist score for each of the three developed checklists and the reliability of the checklists was examined. Cronbach alpha reliability analysis demonstrated that CL 1 and CL 3 performed similarly (0.38 and 0.35). CL 2 performed the worst (0.12). Performance on the three CL was also compared with the students’ performance on the post encounter SOAP note for this case using Pearson correlation coefficients (N = 375 Prob > r under H0: Rho=0). None of the correlations were significantly different from zero. Conclusion Evidence based review may result in a checklist that is significantly shorter than one developed by a small cohort of physicians without loss of reliability. The longer checklists did not correlate better with student performance on a post encounter note. More research on development of examination materials is warranted.

Assessing impact of not attending lectures on osteopathic medical student performance: Brief survey of the literature and proposed future research

It has been anecdotally reported that osteopathic medical students often decrease their attendance of didactic lectures to increase time to study for the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA). There is no strong evidence supporting the impact of attending vs. not attending didactic lectures on osteopathic medical student performance in courses during the first two years of osteopathic medical education. This paper provides a brief survey of the literature. The paper also presents a proposal for future research to assess the impact of not attending didactic lectures on student performance. The proposed study would provide needed evidence toward answering this question in the osteopathic medical school environment.

Relationship Between Standardized Patient Checklist Item Accuracy and Performing Arts Experience

: It is not known whether a Standardized Patient's (SP's) performing arts background could affect his or her accuracy in recording candidate performance on a high-stakes clinical skills examination, such as the Comprehensive Osteopathic Medical Licensing Examination Level 2 Performance Evaluation. The purpose of this study is to investigate the differences in recording accuracy of history and physical checklist items between SPs who identify themselves as performing artists and SPs with no performance arts experience. : Forty SPs identified themselves as being performing artists or nonperforming artists. A sample of SP live examination ratings were compared with a second set of ratings obtained after video review (N = 1972 SP encounters) over 40 cases from the 2008-2009 testing cycle. Differences in SP checklist recording accuracy were tested as a function of performing arts experience. : Mean overall agreement rates, both uncorrected and corrected for chance agreement, were very high (0.94 and 0.79, respectively, at the overall examination level). There was no statistically significant difference between the two groups with respect to any of the mean accuracy measures: history taking (z = -0.422, P = 0.678), physical examination (z = -1.453, P = 0.072), and overall data gathering (z = -0.812, P = 0.417) checklist items. : Results suggest that SPs with or without a performing arts background complete history taking and physical examination checklist items with high levels of precision. Therefore, SPs with and without performing arts experience can be recruited for high-stakes SP-based clinical skills examinations without sacrificing examination integrity or scoring accuracy.

Relationships between high-stakes clinical skills exam scores and program director global competency ratings of first-year pediatric residents

BackgroundResponding to mandates from the Accreditation Council for Graduate Medical Education (ACGME) and American Osteopathic Association (AOA), residency programs have developed competency-based assessment tools. One such tool is the American College of Osteopathic Pediatricians (ACOP) program directors’ annual report. High-stakes clinical skills licensing examinations, such as the Comprehensive Osteopathic Medical Licensing Examination Level 2-Performance Evaluation (COMLEX-USA Level 2-PE), also assess competency in several clinical domains.ObjectiveThe purpose of this study is to investigate the relationships between program director competency ratings of first-year osteopathic residents in pediatrics and COMLEX-USA Level 2-PE scores from 2005 to 2009.MethodsThe sample included all 94 pediatric first-year residents who took COMLEX-USA Level 2-PE and whose training was reviewed by the ACOP for approval of training between 2005 and 2009. Program director competency ratings and COMLEX-USA Level 2-PE scores (domain and component) were merged and analyzed for relationships.ResultsBiomedical/biomechanical domain scores were positively correlated with overall program director competency ratings. Humanistic domain scores were not significantly correlated with overall program director competency ratings, but did show moderate correlation with ratings for interpersonal and communication skills. The six ACGME or seven AOA competencies assessed empirically by the ACOP program directors’ annual report could not be recovered by principal component analysis; instead, three factors were identified, accounting for 86% of the variance between competency ratings.DiscussionA few significant correlations were noted between COMLEX-USA Level 2-PE scores and program director competency ratings. Exploring relationships between different clinical skills assessments is inherently difficult because of the heterogeneity of tools used and overlap of constructs within the AOA and ACGME core competencies.

The Promise and Challenge of Including Multimedia Items in Medical Licensure Examinations: Some Insights From an Empirical Trial

The Comprehensive Osteopathic Medical Licensing Examination conducted a trial of multimedia items in the 2008-2009 Level 3 testing cycle to determine (1) if multimedia items were able to test additional elements of medical knowledge and skills and (2) how to develop effective multimedia items. Forty-four content-matched multimedia and text multiple-choice items were randomly delivered to Level 3 candidates. Logistic regression and paired-samples t tests were used for pairwise and group-level comparisons, respectively. Nine pairs showed significant differences in either difficulty or/and discrimination. Content analysis found that, if text narrations were less direct, multimedia materials could make items easier. When textbook terminologies were replaced by multimedia presentations, multimedia items could become more difficult. Moreover, a multimedia item was found not uniformly difficult for candidates at different ability levels, possibly because multimedia and text items tested different elements of a same concept. Multimedia items may be capable of measuring some constructs different from what text items can measure. Effective multimedia items with reasonable psychometric properties can be intentionally developed.

COMLEX-1 and USMLE-1 Are Not Interchangeable Examinations

Osteopathic medical students must take the Comprehensive Osteopathic Medical Licensing Examination (COMLEX)-USA series of examinations, but not the United States Medical Licensing Examination (USMLE-1) series. Few data are available describing the comparability of the two tests. This study sought to determine if COMLEX-1 scores could predict USMLE-1 scores among osteopathic medical students applying to an emergency medicine (EM) residency and to determine if the scores are interchangeable. This was a retrospective analysis of osteopathic medical students applying to an EM residency program in the 2005-2006 and 2006-2007 application seasons. Students were included if they took both the COMLEX-1 and the USMLE-1 examinations. Linear regression was performed and a Bland-Altman plot of the standardized mean scores of each test was created. Ninety students were included. The mean (+/- standard deviation [SD]) COMLEX-1 score was 559.5 (+/-68.6), and the mean (+/-SD) USMLE-1 score was 207.6 (+/-15.5). The correlation was 0.79, with an R(2) of 62.3%. The Bland-Altman plot showed a mean difference between the standardized scores of 0, with 95% confidence intervals (CIs) of )1.28 to +1.28 standard normal units. Limitations include that this was a single center study, and only students who took both tests could be studied. COMLEX-1 scores predict only 62.3% of the variance in USMLE-1 scores, and the scores are not interchangeable.

Using Standardized Patients to Assess the Communication Skills of Graduating Physicians for the Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 2-Performance Evaluation (Level 2-PE)

Background: Standardized patients can be trained to assess the communication and interpersonal skills of medical students and graduates. Purpose: The purpose of this study is to present data to support the psychometric adequacy of the communication ratings provided by standardized patients. Methods: Using the data from testing of 3, 450 examinees over a 1-year period, a number of psychometric analyses were undertaken. These included a variance component analysis, the calculation of various validity coefficients, the comparison of communication ratings for select examinee cohorts and case characteristics, and the investigation of some potential sources of score invalidity. Results: Communication skills scores are moderately correlated to other competencies (knowledge, skills) and may be influenced by candidate characteristics such as gender and English language proficiency. They are not dependant on the age of the examinees, the clinical case content, or the gender of the standardized patients. Conclusions: For a multistation assessment, a reasonably precise and valid estimate of a candidate's communication ability can be obtained from trained standardized patients.

Patient Note Fabrication and Consequences of Unprofessional Behavior in a High-Stakes Clinical Skills Licensing Examination

The National Board of Osteopathic Medical Examiners (NBOME) administers the Comprehensive Osteopathic Medical Licensing Examination USA Level 2-PE (COMLEX-USA Level 2-PE) and has developed a process that links the competencies of written communication and professionalism by tracking fabrication in the postencounter SOAP (Subjective, Objective, Assessment, Plan) note exercise. A process used to identify potential SOAP note fabrication was implemented in the 2007-2008 test cycle for the COMLEX-USA Level 2-PE. A total of 3,753 candidates took the COMLEX-USA Level 2-PE in the 2007-2008 test cycle. Forty-eight candidates were screened, and the NBOME's Subcommittee on SOAP Note Fabrication made failure decisions on eight, leading to a failure rate of 0.2% based on fabrication review. The NBOME has adopted the stance that postencounter note fabrication represents unprofessional behavior. Screening for and failing candidates who exhibit unprofessional behavior enhances the validity of the examination.

Survey on the clinical skills of osteopathic medical students.

As part of the standard-setting methods used by the National Board of Osteopathic Medical Examiners for its Comprehensive Osteopathic Medical Licensing Examination clinical skills performance evaluation (COMLEX-USA Level 2-PE), a self-administered survey was distributed electronically and by mail to deans of colleges of osteopathic medicine, directors of graduate medical education programs, osteopathic medical students, and experts chosen demographically to represent osteopathic physicians in the United States. Groups were asked to rate fourth-year osteopathic medical students and interns on their clinical skills and acceptable pass rates and expected pass rates on the COMLEX-USA Level 2-PE. The surveys were not used systematically to compute the passing standards but to provide additional support for their validity. The viewpoints of the deans differed from those of the students, osteopathic graduate medical education program directors, and experts regarding clinical skills proficiencies and acceptable pass rates. However, all of the groups agreed that, on average, some students and interns do not have adequate clinical skills. These results provide additional support for requiring acceptable performance on a comprehensive clinical skills examination before admission to osteopathic graduate medical education programs.

Relationships between scores on the COMLEX-USA Level 2-Performance Evaluation and selected school-based performance measures.

The Comprehensive Osteopathic Medical Licensing Examination USA Level 2 Performance Evaluation (COMLEX-USA Level 2-PE) is a national multistation performance examination designed to examine students' osteopathic clinical skills. The current study examines the relationship between achievement levels on the COMLEX-USA Level 2-PE and selected school-related variables for the class of 2005 at the West Virginia School of Osteopathic Medicine in Lewisburg, WVa (N=70). Significant (P<.01) correlations between the COMLEX-USA Level 2-PE summary performance and selected academic achievement measures include: weighted Physical Diagnosis grade, 0.41; weighted year 1 and year 2 Osteopathic Principles and Practice grade, 0.37: overall year 2 grade point average, 0.42; the objective structured clinical evaluation (OSCE) Physical Examination score, 0.40; and the OSCE Total Station score, 0.33. While further research is needed, the current study found modest but notable relationships between school-generated academic variables and performance on the COMLEX-USA Level 2-PE, and therefore supports the validity of the COMLEX-USA Level 2-PE examination for assessing the clinical skills of future osteopathic physicians.

State of the art in standardized patient programs: a survey of osteopathic medical schools.

In 2001, the authors sent a survey to all 19 schools of osteopathic medicine requesting information about how standardized patients are used for clinical teaching and skills assessment. The purpose of this survey was to identify key people involved in standardized patient programs (SPPs), to gather information on the resources allocated by osteopathic medical schools, to determine the "state of the art" of standardized patient teaching and assessment in osteopathic medical schools, and to assess which SPPs would be available to assist in the development of the new component of the National Board of Osteopathic Medical Examiners licensing examination, the Comprehensive Osteopathic Medical Licensing Examination Performance Evaluation. The survey results revealed that 12 osteopathic medical schools have SPPs that teach and assess clinical skills, including history taking; physical examination (complete and focused); genital, rectal, and breast examinations; physician-patient communication, osteopathic manipulative medicine or osteopathic manipulative treatment. Details of the 12 schools' SPPs are highlighted.

Health Professions Scholarship Program: Are the Armed Forces Getting Quality Osteopathic Physicians?

To compare the medical knowledge and reasoning of osteopathic medical students in the Health Professions Scholarship Program (HPSP) with corresponding civilian students, we analyzed their performance on the Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) levels 1 and 2. The results from this study showed no significant difference between that HPSP medical students (COMLEX-USA level 1, N = 37; COMLEX-USA level 2, N = 34) and civilian students (COMLEX-USA level 1, N = 507; COMLEX-USA level 2, N = 492) on COMLEX-USA level 1 (p = 0.24) and COMLEX-USA level 2 (p = 0.50). Moreover, no significant difference was observed between Air Force (COMLEX-USA level 1, N = 8; COMLEX-USA level 2, N = 6), Army (COMLEX-USA level 1, N = 13; COMLEX-USA level 2, N = 13), and Navy (COMLEX-USA level 1, N = 16; COMLEX-USA level 2, N = 15) HPSP students for COMLEX-USA level 1 (p = 0.42) and COMLEX-USA level 2 (p = 0.75). Therefore, we conclude that, upon graduation from medical school, the medical knowledge and reasoning of HPSP osteopathic graduates as determined by COMLEX-USA are equivalent to those of their civilian counterparts.