Multiple-choice Question Examination Research Articles

Assessment Score Comparison of Knowledge and Clinical Reasoning of Medical Student Educated by Bedside or Conference Room Teaching in Internal Medicine Ward

Background: Evidence shows that bedside teaching is declining in recent years. One reason is the expectation for students to earn credit. Here, the impact of education method on undergraduate assessment score will be tested. Methods: This is a quasi-experimental study conducted with 66 undergraduate medical students who were allocated into two matched groups on the basis of a pulmonary pathophysiology course score in the internal medicine department of the Birjand University of Medical Sciences (Iran). Chronic obstructive pulmonary disease (COPD) was chosen as the theme of this education. After groups were allocated, training was conducted with 36 students using the Traditional Patient’s Bedside Teaching (Tr-BT) method and for 30 students using the Conference Room Case Presentations (Cr-CP) method. Evaluation were conducted at the end of the three-month course period using the Multiple-Choice Questions (MCQ) and Key Feature (KF) examination. Data were analyzed using independent t tests, Mann-Whitney U, and the chi-square test. Results: Among the collected student the scores of KF, inward ratings and MCQ examinations in intern vs extern were (8.58 ± 1.46 vs 8.31 ± 1.58; P = 0.65), (14.75±1.54 vs 15.51±1.88; P = 0.08),( 10.77 ± 2.20 vs 10.47 ± 1.61; P = 0.55) respectively. The mean of the pulmonary pathophysiology score was 13.03 ± 1.66 in the Tr-BT group compared with 12.67 ± 1.92 in the Cr-CP group (P = 0.23). The means of the MCQ scores were 10.07 ± 2.13 and 11.37 ± 1.56 in the Tr-BT and Cr-CP groups, respectively (P = 0.007). Conclusion: The score of the MCQ exam was significantly higher in the group taught using the Cr-CP method. Students and teachers may prefer Cr-CP. To increase interest in bedside teaching, the assessment methods of medical students should be based on practical and clinical judgment evaluation.

Exploring examinee behaviours as validity evidence for multiple-choice question examinations.

Clinical-vignette multiple choice question (MCQ) examinations are used widely in medical education. Standardised MCQ examinations are used by licensure and certification bodies to award credentials that are meant to assure stakeholders as to the quality of physicians. Such uses are based on the interpretation of MCQ examination performance as giving meaningful information about the quality of clinical reasoning. There are several assumptions foundational to these interpretations and uses of standardised MCQ examinations. This study explores the implicit assumption that cognitive processes elicited by clinical-vignette MCQ items are like the processes thought to occur with 'real-world' clinical reasoning as theorised by dual-process theory. Fourteen participants (three medical students, five residents and six staff physicians) completed three sets of five timed MCQ items (total 15) from the Medical Knowledge Self-Assessment Program (MKSAP). Upon answering a set of MCQs, each participant completed a retrospective think aloud (TA) protocol. Using constant comparative analysis (CCA) methods sensitised by dual-process theory, we performed a qualitative thematic analysis. Examinee behaviours fell into three categories: clinical reasoning behaviours, test-taking behaviours and reactions to the MCQ. Consistent with dual-process theory, statements about clinical reasoning behaviours were divided into two sub-categories: analytical reasoning and non-analytical reasoning. Each of these categories included several themes. Our study provides some validity evidence that test-takers' descriptions of their cognitive processes during completion of high-quality clinical-vignette MCQs align with processes expected in real-world clinical reasoning. This supports one of the assumptions important for interpretations of MCQ examination scores as meaningful measures of clinical reasoning. Our observations also suggest that MCQs elicit other cognitive processes, including certain test-taking behaviours, that seem 'inauthentic' to real-world clinical reasoning. Further research is needed to explore if similar themes arise in other contexts (e.g. simulated patient encounters) and how observed behaviours relate to performance on MCQ-based assessments.

Effects of additional team-based learning on students\u2019 clinical reasoning skills: a pilot study

BackgroundIn the field of Neurology good clinical reasoning skills are essential for successful diagnosing and treatment. Team-based learning (TBL), an active learning and small group instructional strategy, is a promising method for fostering these skills. The aim of this pilot study was to examine the effects of a supplementary TBL-class on students’ clinical decision-making skills.MethodsFourth- and fifth-year medical students participated in this pilot study (static-group comparison design). The non-treatment group (n = 15) did not receive any additional training beyond regular teaching in the neurology course. The treatment group (n = 11) took part in a supplementary TBL-class optimized for teaching clinical reasoning in addition to the regular teaching in the neurology course. Clinical decision making skills were assessed using a key-feature problem examination. Factual and conceptual knowledge was assessed by a multiple-choice question examination.ResultsThe TBL-group performed significantly better than the non-TBL-group (p = 0.026) in the key-feature problem examination. No significant differences between the results of the multiple-choice question examination of both groups were found.ConclusionsIn this pilot study participants of a supplementary TBL-class significantly improved clinical decision-making skills, indicating that TBL may be an appropriate method for teaching clinical decision making in neurology. Further research is needed for replication in larger groups and other clinical fields.

Teaching neurology to medical students with a simplified version of team-based learning.

To compare the effect of a simplified version of team-based learning (sTBL), an active learning/small group instructional strategy, with that of the traditionally used small group interactive seminars on the acquisition of knowledge and clinical reasoning (CR) skills. Third- and fourth-year medical students (n = 122) were randomly distributed into 2 groups. A crossover design was used in which 2 neurologic topics were taught by sTBL and 2 by small group interactive seminars. Knowledge was assessed with a multiple-choice question examination (MCQE), CR skills with a key feature problem examination (KFPE). Questionnaires were used for further methodologic evaluation. No group differences were found in the MCQE results. sTBL instruction of the topic "acute altered mental status" was associated with a significantly better student performance in the KFPE (p = 0.008), with no differences in the other 3 topics covered. Although both teaching methods were highly rated by the students, a clear majority voted for sTBL as their preferred future teaching method. sTBL served as an equivalent alternative to small group interactive seminars for imparting knowledge and teaching CR skills, and was particularly advantageous for teaching CR in the setting of a complex neurologic topic. Furthermore, students reported a strong preference for the sTBL approach, making it a promising tool for effectively teaching neurology.

Certainty and safe consequence responses provide additional information from multiple choice question assessments

BackgroundClinicians making decisions require the ability to self-monitor and evaluate their certainty of being correct while being mindful of the potential consequences of alternative actions. For clinical students, this ability could be inferred from their responses to multiple-choice questions (MCQ) by recording their certainty in correctness and avoidance of options that are potentially unsafe.MethodsResponse certainty was assessed for fifth year medical students (n = 330) during a summative MCQ examination by having students indicate their certainty in each response they gave on the exam. Incorrect responses were classified as to their inherent level of safeness by an expert panel (response consequence). Analyses compared response certainty, response consequence across student performance groupings.ResultsAs students’ certainty in responses increased, the odds they answered correctly increased and the odds of giving unsafe answers decreased. However, from some ability groups the odds of an incorrect response being unsafe increased with high certainty.ConclusionsCertainty in, and safeness of, MCQ responses can provide additional information to the traditional measure of a number correct. In this sample, even students below standard demonstrated appropriate certainty. However, apart from those scoring lowest, student’s incorrect responses were more likely to be unsafe when they expressed high certainty. These findings suggest that measures of certainty and consequence are somewhat independent of the number of correct responses to MCQs and could provide useful extra information particularly for those close to the pass-fail threshold.

Standardization of Clinical Assessment and Sample Collection Across All PERCH Study Sites.

Background. Variable adherence to standardized case definitions, clinical procedures, specimen collection techniques, and laboratory methods has complicated the interpretation of previous multicenter pneumonia etiology studies. To circumvent these problems, a program of clinical standardization was embedded in the Pneumonia Etiology Research for Child Health (PERCH) study.Methods.Between March 2011 and August 2013, standardized training on the PERCH case definition, clinical procedures, and collection of laboratory specimens was delivered to 331 clinical staff at 9 study sites in 7 countries (The Gambia, Kenya, Mali, South Africa, Zambia, Thailand, and Bangladesh), through 32 on-site courses and a training website. Staff competency was assessed throughout 24 months of enrollment with multiple-choice question (MCQ) examinations, a video quiz, and checklist evaluations of practical skills.Results.MCQ evaluation was confined to 158 clinical staff members who enrolled PERCH cases and controls, with scores obtained for >86% of eligible staff at each time-point. Median scores after baseline training were ≥80%, and improved by 10 percentage points with refresher training, with no significant intersite differences. Percentage agreement with the clinical trainer on the presence or absence of clinical signs on video clips was high (≥89%), with interobserver concordance being substantial to high (AC1 statistic, 0.62–0.82) for 5 of 6 signs assessed. Staff attained median scores of >90% in checklist evaluations of practical skills.Conclusions.Satisfactory clinical standardization was achieved within and across all PERCH sites, providing reassurance that any etiological or clinical differences observed across the study sites are true differences, and not attributable to differences in application of the clinical case definition, interpretation of clinical signs, or in techniques used for clinical measurements or specimen collection.

Comparison in the quality of distractors in three and four options type of multiple choice questions.

IntroductionThe number of distractors needed for high quality multiple choice questions (MCQs) will be determined by many factors. These include firstly whether English language is their mother tongue or a foreign language; secondly whether the instructors who construct the questions are experts or not; thirdly the time spent on constructing the options is also an important factor. It has been observed by Tarrant et al that more time is often spent on constructing questions than on tailoring sound, reliable, and valid distractors.ObjectivesFirstly, to investigate the effects of reducing the number of options on psychometric properties of the item. Secondly, to determine the frequency of functioning distractors among three or four options in the MCQs examination of the dermatology course in University of Bahri, College of Medicine.Materials and methodsThis is an experimental study which was performed by means of a dermatology exam, MCQs type. Forty MCQs, with one correct answer for each question were constructed. Two sets of this exam paper were prepared: in the first one, four options were given, including one key answer and three distractors. In the second set, one of the three distractors was deleted randomly, and the sequence of the questions was kept in the same order. Any distracter chosen by less than 5% of the students was regarded as non-functioning. Kuder-Richardson Formula 20 (Kr-20) measures the internal consistency and reliability of an examination with an acceptable range 0.8–1.0. Chi square test was used to compare the distractors in the two exams.ResultsA significant difference was observed in discrimination and difficulty indexes for both sets of MCQs. More distractors were non-functional for set one (of four options), but slightly more reliable. The reliability (Kr-20) was slightly higher for set one (of four options). The average marks in option three and four were 34.163 and 33.140, respectively.ConclusionCompared to set 1 (four options), set 2 (of three options) was more discriminating and associated with low difficulty index but its reliability was low.

Multiple mini-interview predictive validity for performance on a pharmacy licensing examination.

Predictive validity studies on the use of the multiple mini-interview (MMI) have been primarily in medicine. This study sought to determine the predictive validity of the MMI for performance within a pharmacy programme and on the Pharmacy Examining Board of Canada (PEBC) Qualifying Examination for licensure, and to compare the predictive validity of the MMI with that of pre-pharmacy grade point average (GPA) and Pharmacy College Admission Test (PCAT) score. Admissions data for 223 graduates of the pharmacy programme at the University of Toronto were matched to programme and licensure outcome measures. Multiple linear regression assessed the predictive ability of the MMI, pre-pharmacy GPA, PCAT and covariates for performance in final-year experiential rotations, cumulative GPA (cGPA) and PEBC-MCQ (multiple-choice question examination) and PEBC-OSCE (objective structured clinical examination) overall and subcomponent scores. The PCAT, pre-pharmacy GPA and age significantly predicted the PEBC-MCQ overall score. The MMI was the only significant predictor of overall score on the PEBC-OSCE (β = 0.17, p = 0.02); it also predicted communication and performance subscores. Scores on the PCAT and female gender predicted the communication subscore. Pre-pharmacy GPA, age and female gender significantly predicted cGPA. The MMI was the only significant predictor of institutional/ambulatory rotation score (β = 0.26, p = 0.00). The MMI, designed to measure non-academic attributes including communication, motivation and problem-solving skills, was the only admissions tool with significant predictive validity for performance on the PEBC-OSCE national pharmacy certification examination and in an institutional/ambulatory rotation. These findings, from a single cohort of undergraduates, provide the first report of the predictive validity of the MMI for performance on pharmacy licensure examinations and thereby strengthen the evidence for its use in health professions selection. Prior university academic performance significantly predicted cGPA and performance on the PEBC-MCQ. Performance on the PCAT also predicted PEBC-MCQ results.

A comparison of clinical-scenario (case cluster) versus stand-alone multiple choice questions in a problem-based learning environment in undergraduate medicine.

ObjectivesTo compare stand-alone multiple choice questions (MCQs) and integrated clinical-scenario (case cluster) multiple choice questions (CS-MCQs) in a problem-based learning (PBL) environment.MethodsA retrospective descriptive analysis of MCQ examinations was conducted in a course that integrates the subspecialties of anatomical pathology, chemical pathology, hematology, immunology, microbiology and pharmacology. The MCQ items were analyzed for their reliability (Kuder–Richardson-20, KR-20), level of difficulty (Pi), discrimination index (Di), item distractors and student performances. The statistical analysis of the results was extracted from the integrity online item-analysis programme. The results of the standard stand-alone and CS multiple choice questions were compared.ResultsKR-20 for the CS-MCQs and stand-alone MCQs was consistently high. KR-20 and Pi were higher for the CS-MCQs. There was no significant difference between the CS-MCQs and stand-alone MCQs in Pi and Di. A range of difficulty levels was found based on Bloom's taxonomy. The mean scores for the class were higher for the CS-MCQ examination. The compilation of the CS-MCQ examination was more challenging.ConclusionsCS-MCQs compare favorably to stand-alone MCQs and provide opportunities for the integration of sub-specialties and assessment in keeping with PBL. They assess students' cognitive skills and are reliable and practical. Different levels of item difficulty promote multi-logical and critical thinking. Students' scores were higher for the CS-MCQ examination, which may suggest better understanding of the material and/or better question clarity. The scenarios have to flow logically. Increasing the number of scenarios ensures the examination of more course content.

An Assessment of Intubation Skill Training in Novice Anesthesiology Residents of Tehran University of Medical Sciences With the Use of Mannequins.

BackgroundThe goal of this study was to evaluate the impact of intubation skill training involving the use of mannequins on novice anesthesiology residents in a knowledge, attitudes, and practices designed study in which three different types of evaluation were implemented.MethodsAll first-year anesthesiology residents (24) of Sina Hospital, affiliated to the Tehran University of Medical Sciences, were invited to participate in an intubating skills training course. The program comprised two theoretical and three practical sessions, lasting a total of 16 hours over four days. Faculty assessment of residents’ practices was carried out using the questionnaire results, measured using a Likert scale, as the primary outcome. An improvement in the theoretical knowledge of the novice anesthesiology residents (using the Likert scale) and their attitudes towards the educational course in general (via a multiple choice question examination), were also evaluated.ResultsThe mean score following faculty assessment of the residents’ practical skills was 4.6 out of 5.0 (92%) [standard deviation (SD) of 0.13]. The mean score with respect to the attitudes of the residents was 4.8 out of 5.0 (96%) (SD of 0.16). The overall mean theoretical score of the residents improved significantly upon completion of the training program (P = 0.001).ConclusionsOur results suggest that the personnel in the five participating faculties were highly satisfied with the practical performance of the residents, who were found to hold good attitudes towards the program as a whole.

Revisiting 'The effect of assessments and examinations on the learning of medical students'.

Revisiting 'The effect of assessments and examinations on the learning of medical students'.

Can medical students accurately predict their learning? A study comparing perceived and actual performance in neuroanatomy.

It is important that clinicians are able to adequately assess their level of knowledge and competence in order to be safe practitioners of medicine. The medical literature contains numerous examples of poor self-assessment accuracy amongst medical students over a range of subjects however this ability in neuroanatomy has yet to be observed. Second year medical students attending neuroanatomy revision sessions at the University of Southampton and the competitors of the National Undergraduate Neuroanatomy Competition were asked to rate their level of knowledge in neuroanatomy. The responses from the former group were compared to performance on a ten item multiple choice question examination and the latter group were compared to their performance within the competition. In both cohorts, self-assessments of perceived level of knowledge correlated weakly to their performance in their respective objective knowledge assessments (r=0.30 and r=0.44). Within the NUNC, this correlation improved when students were instead asked to rate their performance on a specific examination within the competition (spotter, rS=0.68; MCQ, rS=0.58). Despite its inherent difficulty, medical student self-assessment accuracy in neuroanatomy is comparable to other subjects within the medical curriculum. Anat Sci Educ 9: 488-495. © 2016 American Association of Anatomists.

Value of General Medical Knowledge Examinations in Performance Assessment of Practicing Physicians With Potential Competence and Performance Deficiencies.

Problems with a physician's performance may arise at any point during their career. As such, there is a need for effective, valid tools and processes to accurately assess and identify deficiencies in competence or performance. Although scores on multiple-choice questions have been shown to be predictive of some aspects of physician performance in practicing physicians, their relationship to overall clinical competence is somewhat uncertain particularly after the first 10 years of practice. As such, the purpose of this study was to examine how a general medical knowledge multiple-choice question examination is associated with a comprehensive assessment of competence and performance in experienced practicing physicians with potential competence and performance deficiencies. The study included 233 physicians, of varying specialties, assessed by the University of California, San Diego Physician Assessment and Clinical Education Program (PACE), between 2008 and 2012, who completed the Post-Licensure Assessment System Mechanisms of Disease (MoD) examination. Logistic regression determined if the examination score significantly predicted passing assessment outcome after correcting for gender, international medical graduate status, certification status, and age. Most physicians (89.7%) received an overall passing assessment outcome on the PACE assessment. The mean MoD score was 66.9% correct, with a median of 68.0%. Logistic regression (P = .038) was significant in indicating that physicians with higher MoD examination scores had an increased likelihood of achieving a passing assessment outcome (odds ratio = 1.057). Physician MoD scores are significant predictors of overall physician competence and performance as evaluated by PACE assessment.

Negative Marking and the Student Physician–-A Descriptive Study of Nigerian Medical Schools

BackgroundThere is considerable debate about the two most commonly used scoring methods, namely, the formula scoring (popularly referred to as negative marking method in our environment) and number right scoring methods. Although the negative marking scoring system attempts to discourage students from guessing in order to increase test reliability and validity, there is the view that it is an excessive and unfair penalty that also increases anxiety. Feedback from students is part of the education process; thus, this study assessed the perception of medical students about negative marking method for multiple choice question (MCQ) examination formats and also the effect of gender and risk-taking behavior on scores obtained with this assessment method.MethodsThis was a prospective multicenter survey carried out among fifth year medical students in Enugu State University and the University of Nigeria. A structured questionnaire was administered to 175 medical students from the two schools, while a class test was administered to medical students from Enugu State University. Qualitative statistical methods including frequencies, percentages, and chi square were used to analyze categorical variables. Quantitative statistics using analysis of variance was used to analyze continuous variables.ResultsInquiry into assessment format revealed that most of the respondents preferred MCQs (65.9%). One hundred and thirty students (74.3%) had an unfavorable perception of negative marking. Thirty-nine students (22.3%) agreed that negative marking reduces the tendency to guess and increases the validity of MCQs examination format in testing knowledge content of a subject compared to 108 (61.3%) who disagreed with this assertion (χ2 = 23.0, df = 1, P = 0.000). The median score of the students who were not graded with negative marking was significantly higher than the score of the students graded with negative marking (P = 0.001). There was no statistically significant difference in the risk-taking behavior between male and female students in their MCQ answering patterns with negative marking method (P = 0.618).ConclusionsIn the assessment of students, it is more desirable to adopt fair penalties for discouraging guessing rather than excessive penalties for incorrect answers, which could intimidate students in negative marking schemes. There is no consensus on the penalty for an incorrect answer. Thus, there is a need for continued research into an effective and objective assessment tool that will ensure that the students’ final score in a test truly represents their level of knowledge.

Quizzes: Quiz Application Development Using Android-Based MIT APP Inventor Platform

This work deals with the development of Android-based multiple-choice question examination system, namely: Quizzes. This application is developed for educational purposes, allowing the users to prepare the multiple choice questions for different examinations conducted on provincial and national level. The main goal of the application is to enable users to practice for subjective tests conducted for admissions and recruitment, with the focus on Computer Science field. This quiz application includes three main modules, namely (i) computer science, (ii) verbal, and (iii) analytical. The computer science and verbal modules contain various sub-categories. This quiz includes three functions: (i) Hint, (ii) Skip, and (iii) Pause/life-lines. These functions can be used only once by a user. It shows progress feedback during quiz play, and at the end, the app also shows the result.

Utiliy of CPX as a Way to Assess Communication Skill

Purpose: The communication skills of doctors become important as recent changes in health care environment. The clinical practice examination (CPX) has been introduced in many countries as a way to assess communication skill. The purpose of this paper was to determine whether CPX is useful to assess communication skills through an evaluation the correlation between the CPX score and the written multiple-choice question (MCQ) examination on the In-training examination for family medicine. Methods: The subjects of this study were 31 residents in Seoul and Gyunggi-do. CPX consists of 5 station (low back pain, rectal bleeding, blood tinged sputum, bad news delivery, alcoholic) using standardized patient (SP) was performed on April, 2007. We determined the correlation between the total scores and 4 domain scores (history taking, physical examination, patient education, and patient-physician interaction) on the CPX with the MCQ score of the In-training examination for family medicine 2007. Results: The total score of CPX and the CPX scores on history taking, patient education and patient-physician interaction did not correlate with MCQ score of In-training examination for family medicine, The CPX score on physical examination correlated slightly with MCQ score. Conclusion: The lower correlation with other existing examination tool is, the higher the value independently. Our experiences suggest that CPX has specific evaluation region and is useful to assess the communication skills of family physician.

Curricular Change and the Learning Approaches of Medical Students

We read with great interest the recent article in your journal by Mogre and Amalba about the learning approaches of Ghanian students who followed a problem-based learning, Communitybased education and service-based (PBLCOBES) medical curriculum [1]. Determining students’ preferred, ongoing and contextual approaches to learning can provide a good idea about the presage, process and product levels of Bigg’s “Presage-Process-Product (3P) model” of the learning process [2]. Learning approaches should therefore not be viewed as stable characteristics of students; rather their value in reflecting the learning environment should be recognized and utilized. The authors have rightly determined their students’ learning approaches to judge the impact of the new curriculum. We had studied the learning approaches of first year medical students belonging to medical colleges in South India following a discipline based, non-PBL curriculum [3, 4]. The majority of our medical students had a deep learning approach and there was a positive correlation between deep approach scores and the marks obtained in a Multiple Choice Question (MCQ) examination containing questions that tested understanding, application and recall [3] and between surface learning approach scores and the Perceived Stress Scale scores [4]. To us, this reiterated the importance of students adopting a deep learning approach that focuses on understanding and relating new information to prior knowledge. © Medical Education Department, School of Medical Sciences, Universiti Sains Malaysia. All rights reserved. Curricular Change and the Learning Approaches of Medical Students

The Effect of Senior Medical Student Tutors Compared to Faculty Tutors on Examination Scores of First- and Second-Year Medical Students in Two Problem-Based Learning Courses

At the University of Hawaii John A. Burns School of Medicine, senior medical student volunteers are used as tutors for some problem-based learning groups in both the first and second years. Previous studies on the advantages and disadvantages of student tutors compared to faculty tutors have been equivocal. This study expected to answer the following question: Are there differences in examination scores for learners in their first or second year tutored by fourth-year medical students compared to those tutored by faculty members on two different types of examinations? Students were assessed using more clinically relevant, modified essay question examinations and multiple-choice question examinations. Student grades for eight consecutive years were sorted for year and type of examination into those tutored by a faculty member and those tutored primarily by a senior medical student. The only difference favored faculty tutors on second-year examinations that contained more clinically relevant questions. This phenomenon may be explained by the clinical expertise of faculty tutors making a difference in the second year but not the first year.

A Randomized Educational Intervention Trial to Determine the Effect of Online Education on the Quality of Resident-Delivered Care.

There is limited research on whether online formative self-assessment and learning can change the behavior of medical professionals. We sought to determine if an adaptive longitudinal online curriculum in bone health would improve resident physicians' knowledge, and change their behavior regarding prevention of fragility fractures in women. We used a randomized control trial design in which 50 internal medicine resident physicians at a large academic practice were randomized to either receive a standard curriculum in bone health care alone, or to receive it augmented with an adaptive, longitudinal, online formative self-assessment curriculum delivered via multiple-choice questions. Outcomes were assessed 10 months after the start of the intervention. Knowledge outcomes were measured by a multiple-choice question examination. Clinical outcomes were measured by chart review, including bone density screening rate, calculation of the fracture risk assessment tool (FRAX) score, and rate of appropriate bisphosphonate prescription. Compared to the control group, residents participating in the intervention had higher scores on the knowledge test at the end of the study. Bone density screening rates and appropriate use of bisphosphonates were significantly higher in the intervention group compared with the control group. FRAX score reporting did not differ between the groups. Residents participating in a novel adaptive online curriculum outperformed peers in knowledge of fragility fracture prevention and care practices to prevent fracture. Online adaptive education can change behavior to improve patient care.

Considerations in the use of reflective writing for student assessment: issues of reliability and validity.

Reflective writing is a popular tool to support the growth of reflective capacity in undergraduate medical learners. Its popularity stems from research suggesting that reflective capacity may lead to improvements in skills such as empathy, communication, collaboration and professionalism. This has led to assumptions that reflective writing can also serve as a tool for student assessment. However, evidence to support the reliability and validity of reflective writing as a meaningful assessment strategy is lacking. Using a published instrument for measuring 'reflective capacity' (the Reflection Evaluation for Learners' Enhanced Competencies Tool [REFLECT]), four trained raters independently scored four samples of writing from each of 107 undergraduate medical students to determine the reliability of reflective writing scores. REFLECT scores were then correlated with scores on a Year 4 objective structured clinical examination (OSCE) and Year 2 multiple-choice question (MCQ) examinations to examine, respectively, convergent and divergent validity. Across four writing samples, four-rater Cronbach's α-values ranged from 0.72 to 0.82, demonstrating reasonable inter-rater reliability with four raters using the REFLECT rubric. However, inter-sample reliability was fairly low (four-sample Cronbach's α = 0.54, single-sample intraclass correlation coefficient: 0.23), which suggests that performance on one reflective writing sample was not strongly indicative of performance on the next. Approximately 14 writing samples are required to achieve reasonable inter-sample reliability. The study found weak, non-significant correlations between reflective writing scores and both OSCE global scores (r = 0.13) and MCQ examination scores (r = 0.10), demonstrating a lack of relationship between reflective writing and these measures of performance. Our findings suggest that to draw meaningful conclusions about reflective capacity as a stable construct in individuals requires 14 writing samples per student, each assessed by four or five raters. This calls into question the feasibility and utility of using reflective writing rigorously as an assessment tool in undergraduate medical education.