Computer-based Case Simulations Research Articles

Using Process Data to Evaluate the Impact of Shortening Allotted Case Time in a Simulation‐Based Assessment

AbstractThe Computer‐based Case Simulations (CCS) component of the United States Medical Licensing Examination (USMLE) Step 3 was developed to assess the decision‐making and patient‐management skills of physicians. Process data can provide deep insights into examinees’ behavioral processes related to completing the CCS assessment task. In this paper, we utilized process data to evaluate the impact of shortening allotted time limit by rescoring the CCS cases based on process data extracted at various timestamps that represented different percentages of the original allotted case time. It was found that examinees’ performance as well as the correlation between original and newly generated scores both tended to decrease as the timestamp condition became stricter. The impact of shortening allotted time limit was found marginally associated with case difficulties, but strongly dependent on the case time intensity under the original time setting.

An Examination of the Associations Among USMLE Step 3 Scores and the Likelihood of Disciplinary Action in Practice.

As the last examination in the United States Medical Licensing Examination (USMLE) sequence, Step 3 provides a safeguard before physicians enter into unsupervised practice. There is, however, little validity research focusing on Step 3 scores beyond examining its associations with other educational and professional assessments thought to cover similar content. This study examines the associations between Step 3 scores and subsequent receipt of disciplinary action taken by state medical boards for problematic behavior in practice. It analyzes Step 3 total, Step 3 computer-based case simulation (CCS), and Step 3 multiple-choice question (MCQ) scores. The final sample included 275,392 board-certified physicians who graduated from MD-granting medical schools and who passed Step 3 between 2000 and 2017. Cross-classified multilevel logistic regression models were used to examine the effects of Step 3 scores on the likelihood of receiving a disciplinary action, controlling for other USMLE scores and accounting for jurisdiction and specialty. Results showed that physicians with higher Step 3 total, CCS, and MCQ scores tended to have lower chances of receiving a disciplinary action, after accounting for other USMLE scores. Specifically, a 1-standard-deviation increase in Step 3 total, CCS, and MCQ score was associated with a 23%, 11%, and 17% decrease in the odds of receiving a disciplinary action, respectively. The effect of Step 2 CK score on the likelihood of receiving a disciplinary action was statistically significant, while the effect of Step 1 score became statistically nonsignificant when other Step scores were included in the analysis. Physicians who perform better on Step 3 are less likely to receive a disciplinary action from a state medical board for problematic behavior in practice. These findings provide some validity evidence for the use of Step 3 scores when making medical licensure decisions in the United States.

Effects of bedside team-based learning on pediatric clinical practice in Chinese medical students

BackgroundBedside teaching is a primary educational tool to improve the clinical practice of medical students. As a new medical pedagogical approach, team-based learning (TBL) is gradually being integrated into Chinese medical education programmes to promote clinical reasoning, knowledge application, teamwork and collaboration. The aim of this controlled study is to investigate the effects of TBL on pediatric bedside teaching in medical students.MethodsThirty medical students in pediatric clinical practice were randomly assigned to an intervention and a control group. Students in the intervention group exposed bedside teaching activity with TBL while students in the control group received traditional bedside teaching. Teaching for the two groups was conducted biweekly, and the same clinical cases were selected for both groups with the same instructors. After six months of clinical practice, the differences of learning outcomes between the two groups were compared through assessments by computer-based case simulations (CCS) and mini-Clinical Evaluation Exercise (mini-CEX). Student feedback following completion of bedside teaching was collected by questionnaire.ResultsThe CCS scores in the intervention group were significantly higher than that in the control group (p < 0.05). The mini-CEX results showed that clinical judgment and counseling skills of the intervention group were higher than those in the control group (p < 0.01). Medical interviewing skills and overall clinical competence in the intervention group were better than those in the control group (p < 0.05). In the questionnaire survey, students in the intervention group believed that bedside teaching activity with TBL could promote active learning ability, improve counseling skills and strengthen teamwork.ConclusionsApplication of TBL in bedside teaching not only enhanced clinical practice skills among medical students but also improved their clinical reasoning and counseling skills.

Examining Audiology Students' Clinical Collaboration Skills When Using Virtual Audiology Cases Aided With No Collaboration, Live Collaboration, and Virtual Collaboration.

The purpose of this study was to examine students' ability to use effective clinical collaboration online in a designed scaffolded environment. Three groups were formed to achieve this goal: two control groups (one using no collaboration and one using live, face-to-face collaboration) and one treatment group using virtual collaboration. A quasi-experimental design was conducted at two U.S. universities to examine whether there is a significant difference in clinical reasoning skills between three treatment groups using IUP Audiosim software. Two computer-based audiology case simulations were developed, and participants were randomly placed into the three groups. The clinical reasoning data were analyzed using one-way analysis of variance and Tukey's post hoc analyses. The results indicated that there was a significant difference in clinical reasoning skills between the three treatment groups. The score obtained by the no-collaboration group was significantly less than the scores obtained by the virtual and live collaboration groups. The results imply that lower scores were associated with students receiving more instructor-designed content and higher scores with students receiving less instructor-designed content. Students who received more scaffolds with the collaborations may have demonstrated better decision-making outside the training exercise than those who did not receive scaffolds. However, lower scores on the exercise did not necessarily imply lower skill. Lower scores simply implied a different path toward mastery.

Changes in the Objective Structured Clinical Examination (OSCE) of University Schools of Medicine during COVID-19. Experience with a computer-based case simulation OSCE (CCS-OSCE)

Background and objectivesThe COVID-19 pandemic has forced universities to move the completion of university studies online. Spain’s National Conference of Medical School Deans coordinates an objective, structured clinical competency assessment called the Objective Structured Clinical Examination (OSCE), which consists of 20 face-to-face test sections for students in their sixth year of study. As a result of the pandemic, a computer-based case simulation OSCE (CCS-OSCE) has been designed. The objective of this article is to describe the creation, administration, and development of the test.Materials and methodsThis work is a descriptive study of the CCS-OSCE from its planning stages in April 2020 to its administration in June 2020.ResultsThe CCS-OSCE evaluated the competences of anamnesis, exploration, clinical judgment, ethical aspects, interprofessional relations, prevention, and health promotion. No technical or communication skills were evaluated. The CCS-OSCE consisted of ten test sections, each of which had a 12-min time limit and ranged from six to 21 questions (mean: 1.1 min/question). The CCS-OSCE used the virtual campus platform of each of the 16 participating medical schools, which had a total of 2829 students in their sixth year of study. It was jointly held on two dates in June 2020.ConclusionsThe CCS-OSCE made it possible to bring together the various medical schools and carry out interdisciplinary work. The CCS-OSCE conducted may be similar to Step 3 of the United States Medical Licensing Examination.

Cambios en el examen clínico objetivo y estructurado (ECOE) de las facultades de Medicina durante la COVID-19. Experiencia de una ECOE de casos-clínicos computarizados simulados (ECOE-CCS) conjunta

Antecedentes y objetivosLa pandemia de la COVID-19 ha obligado a completar los estudios universitarios online. La Conferencia Nacional de Decanos de Facultades de Medicina coordina una prueba de evaluación de competencias clínicas objetiva y estructurada (ECOE) de 20 estaciones presenciales a los estudiantes de sexto del grado. Como consecuencia de la pandemia se ha diseñado una ECOE sustitutoria con casos-clínicos computarizados simulados (ECOE-CCS). El objetivo del artículo es describir la elaboración, la ejecución y el desarrollo de la prueba.Materiales y métodosEstudio descriptivo de la ECOE-CCS conjunta desde su gestación en abril 2020 hasta su ejecución en junio 2020.ResultadosLa ECOE-CCS evaluó las competencias de anamnesis, exploración, juicio clínico, aspectos éticos, relaciones interprofesionales, prevención y promoción de la salud. No se evaluaron habilidades técnicas ni de comunicación. La ECOE-CCS consistió en 10 estaciones de 12 minutos de duración, con un número de preguntas de seis a 21 (media: 1,1 minutos/pregunta). En la ECOE-CCS se utilizó la plataforma virtual del campus de cada una de las 16 facultades de Medicina que participaron, con un total de 2.829 estudiantes de sexto curso. Se realizó de una forma conjunta en dos fechas de junio del 2020.ConclusionesLa experiencia de la ECOE-CCS permitió llevar a cabo una integración y el trabajo interdisciplinar de las diferentes facultades de Medicina. La ECOE-CCS realizada podría asemejarse al Step 3 CCS de la United States Medical Licensing Examination.

The evaluation and comparison on different types of resident doctors in training through DxR Clinician system

Compared with the traditional problem-based learning (PBL) mode, computer-based case simulations (CCS) are more advanced in aspects of clinical reasoning, critical thinking and clinical thinking. CCS program, Diagnosis & Reasoning (DxR) Clinician, has been used for assessment of clinical diagnosis and treatment skills in more than 300 medical colleges and universities around the world. However, the evaluating effectiveness and practicability of this software continue to be uncertain and ambiguity in China. Moreover, the influence factors of the DxR evaluation score are still remain indefinite. The purpose of this study was to investigate the impact of training years, professional background and educational background on DxR scores in order to better improve the clinical skills of residents. Eighty one resident doctors were selected as the experimental subjects to conduct the clinical skills assessment with DxR Clinician. The relationships between residents’ scores in DxR Clinician components (overall score, diagnostic score, and management score) and their training years, professional background and educational background were analysed using simultaneous regression. The results showed that years of training, professional background and educational background are potential influence factors of DxR evaluation score. The average DxR Clinician scores in this study are generally lower than foreign research, so we should strengthen resident standardized training to improve our clinical skills.

Computer-Based Case Simulations for Assessment in Health Care: A Literature Review of Validity Evidence.

Simulation for education and training in health-care professions has been widely applied. However, its value as an assessment tool for competence is not fully known. Logistical barriers of simulation-based assessments have led some health-care organizations to utilize computer-based case simulations (CCSs) for assessment. This article provides a review of the literature on the identification of psychometrically sound, CCS instruments designed to measure decision-making competence in health-care professionals. CINAHL, MEDLINE, and Ovid databases identified 84 potentially relevant articles published between January 2000 and May 2017. A total of 12 articles met criteria for inclusion in this review. Findings of these 12 articles indicate that summative assessment in health care using CCSs in the form of clinical scenarios is utilized to assess higher order performance aspects of competence in the form of decision-making. Psychometric strength was validated in eight articles and supported by four replication studies. Two of the eight articles reported evidence of construct validity and support the need for evidence based on a theoretical framework. This literature review offers implications for further research on the use of CCS tools as a method for assessment of competence in health-care professionals and the need for psychometric evidence to support it.

Do medical students' scores using different assessment instruments predict their scores in clinical reasoning using a computer-based simulation?

PurposeThe development of clinical problem-solving skills evolves over time and requires structured training and background knowledge. Computer-based case simulations (CCS) have been used for teaching and assessment of clinical reasoning skills. However, previous studies examining the psychometric properties of CCS as an assessment tool have been controversial. Furthermore, studies reporting the integration of CCS into problem-based medical curricula have been limited.MethodsThis study examined the psychometric properties of using CCS software (DxR Clinician) for assessment of medical students (n=130) studying in a problem-based, integrated multisystem module (Unit IX) during the academic year 2011–2012. Internal consistency reliability of CCS scores was calculated using Cronbach’s alpha statistics. The relationships between students’ scores in CCS components (clinical reasoning, diagnostic performance, and patient management) and their scores in other examination tools at the end of the unit including multiple-choice questions, short-answer questions, objective structured clinical examination (OSCE), and real patient encounters were analyzed using stepwise hierarchical linear regression.ResultsInternal consistency reliability of CCS scores was high (α=0.862). Inter-item correlations between students’ scores in different CCS components and their scores in CCS and other test items were statistically significant. Regression analysis indicated that OSCE scores predicted 32.7% and 35.1% of the variance in clinical reasoning and patient management scores, respectively (P<0.01). Multiple-choice question scores, however, predicted only 15.4% of the variance in diagnostic performance scores (P<0.01), while students’ scores in real patient encounters did not predict any of the CCS scores.ConclusionStudents’ scores in OSCE are the most important predictors of their scores in clinical reasoning and patient management using CCS. However, real patient encounter assessment does not appear to test a construct similar to what is tested in CCS.

Constructs Emerging from Students’ Scores Using Five Assessment Instruments at the End of Pre-Clerkship Phase in a PBL Curriculum

Introduction : Developing a framework of educational outcomes which is aligned with student assessment helps in driving the student learning to achieve these outcomes. The validity of this framework is essential for ensuring fairness to learners and to society. Objective : The goal of this study is to investigate the validity of the underlying domains of the assessment framework emerging from the students’ scores using five instruments used at the end of a pre-clerkship, problem-based medical curriculum. Method : Medical students (n=245, Year 4) were enrolled in the study during two consecutive academic years (2011 and 2012). We examined the construct validity of students’ scores from the following assessment instruments: multiple choice questions (MCQs), integrated short answer questions (SAQs), objective structured clinical examinations (OSCE), real patient clinical examinations (PCE) and computer-based clinical simulations (CCS). An exploratory factor analysis (EFA) was carried out followed by confirmatory analysis (CFA) using structural equation modelling (SEM) to evaluate the constructs emerging from the students’ scores of the five instruments. Result : The analysis yielded four inter-related constructs, called medical knowledge, clinical skills, procedural skills, and reasoning skills. In addition, the four constructs loaded on a higher order construct (called competence) with high regression weights. Although the overall fitness of the separate assessment instruments was poor, the fitness indices of the three factor model after cross-loadings between variables of different constructs were improved ( ( χ 2 [94] = 149.2, p < 0.01 (Bollen-Stine p value = 0.104), CFI = 0.96, TLI = 0.94 and RMSEA = 0.04 (95% CI = 0.03-0.07). Conclusion : The study indicates better validity evidence of the structure from the combined scores of the five instruments in explaining ‘clinical competence” than the individual assessment instruments. The study also demonstrates that computer-based case simulations measure a separate construct which is different from measuring clinical skills in real or standardized patients.

An Explanatory Item Response Theory Approach for a Computer-Based Case Simulation Test

Problem: Practitioners working with multiple-choice tests have long utilized Item Response Theory (IRT) models to evaluate the performance of test items for quality assurance. The use of similar applications for performance tests, however, is often encumbered due to the challenges encountered in working with complicated data sets in which local calibrations alone provide a poor model fit. Purpose: The purpose of this study was to investigate whether the item calibration process for a performance test, computer-based case simulations (CCS), taken from the United States Medical Licensing Examination® (USMLE®) Step 3® examination may be improved through explanatory IRT models. It was hypothesized that explanatory IRT may help improve data modeling for performance assessment tests by allowing important predictors to be added to a conventional IRT model, which are limited to item predictors alone. Methods: The responses of 767 examinees from a six-item CCS test were modeled using the Partial Credit Model (PCM) and four explanatory model extensions, each incorporating one predictor variable of interest. Predictor variables were the examinees’ gender, the order in which examinees encountered an individual item (item sequence), the time it took each examinee to respond to each item (response time), and examinees’ ability score on the multiple-choice part of the examination. Results: Results demonstrate a superior model fit for the explanatory PCM with examinee ability score from the multiple-choice portion of Step 3. Explanatory IRT model extensions might prove useful in complex performance assessment test settings where item calibrations are often problematic due to short tests and small samples. Recommendations: Findings of this study have great value in practice and implications for researchers working with small or complicated response data. Explanatory IRT methodology not only provides a way to improve data modeling for performance assessment tests but also enhances the

Study and exploration of stage assessment of standardized resident training

Objective To explore and establish the effective model of assessment for residen- cy training. Methods Totally 181 residents completed general specialist stage of resident training of Liaoning province in 2010. Assessment process was designed and implemented for general specialist training stage, including theory exam （basic knowledge, professional knowledge and public knowledge） and objective structured clinical examination （OSCE） （standardized patient（SP）, computer-based case simulations（CCS） and clinical skills operation）. T test and variance analysis were used and P〈~O.05 stands for statistically significant difference. Results The total results consisted of two parts, theory examination resuhs （40.00%） and OSCE results （60.00%） and two students failed the examination. The mean score of OSCE was up to standard （66.58 ± 5.50）. The mean score of SP and CCS was close to standard. The mean score of clinical skills operation reached a good level. In terms of the OSCE, SP examination and CCS examination, the performance of residents from the two universities had sig- nificant difference （t=3.818, P=0.000; t=5.581, P=0.000; t=5.419, P=0.013） and the performance of residents from the five hospitals also had significant differences. Conclusions The assessment of standardized resident training is improving gradually. Through the widespread application of OSCE in the assessment; it could be helpful to the clinical training of residents. Key words: Residency training; Objective structured clinical examination; Stage assessment

Comparison of Automated Scoring Methods for a Computerized Performance Assessment of Clinical Judgment

Growing reliance on complex constructed response items has generated considerable interest in automated scoring solutions. Many of these solutions are described in the literature; however, relatively few studies have been published that compare automated scoring strategies. Here, comparisons are made among five strategies for machine-scoring examinee performances of computer-based case simulations, a complex item format used to assess physicians’ patient-management skills as part of the Step 3 United States Medical Licensing Examination. These strategies utilize expert judgments to obtain various (a) case-specific or (b) generic scoring algorithms. The various compromises between efficiency, validity, and reliability that characterize each scoring approach are described and compared.

Computer-based Case Simulations in China: 2001-2011

Computer-based Case Simulations in China: 2001-2011

The design and application of CCS examination system based on ASP.NET technology

To examine the actual clinical diagnosis capability,the import of computer-based case simulations ( CCS ) technology in medical education is very important.The CCS examination system that integrated the ASP.NET and CCS technology can simulate doctor's clinical course more realistically and realize the students' practical skills evaluation.The system has a good prospect in medical education and clinical skills training. Key words: Computer-based case simulations; Network technology; Examination system; Clinical capability

The Impact of Postgraduate Training on USMLE® Step 3® and its Computer-Based Case Simulation Component

The United States Medical Licensing Examination® (USMLE®) Step 3® examination is a computer-based examination composed of multiple choice questions (MCQ) and computer-based case simulations (CCS). The CCS portion of Step 3 is unique in that examinees are exposed to interactive patient-care simulations. The purpose of the following study is to investigate whether the type and length of examinees' postgraduate training impacts performance on the CCS component of Step 3, consistent with previous research on overall Step 3 performance. Retrospective cohort study Medical school graduates from U.S. and Canadian institutions completing Step 3 for the first time between March 2007 and December 2009 (n = 40,588). Post-graduate training was classified as either broadly focused for general areas of medicine (e.g. pediatrics) or narrowly focused for specific areas of medicine (e.g. radiology). A three-way between-subjects MANOVA was utilized to test for main and interaction effects on Step 3 and CCS scores between the demographic characteristics of the sample and type of residency. Additionally, to examine the impact of postgraduate training, CCS scores were regressed on Step 1 and Step 2 Clinical Knowledge (CK) scores. Residuals from the resulting regressions were plotted. There was a significant difference in CCS scores between broadly focused (μ = 216, σ = 17) and narrowly focused (μ=211, σ = 16) residencies (p < 0.001). Examinees in broadly focused residencies performed better overall and as length of training increased, compared to examinees in narrowly focused residencies. Predictors of Step 1 and Step 2 CK explained 55% of overall Step 3 variability and 9% of CCS score variability. Factors influencing performance on the CCS component may be similar to those affecting Step 3 overall. Findings are supportive of the validity of the Step 3 program and may be useful to program directors and residents in considering readiness to take this examination.

The Development of Computer-based Case Simulations for Psychological Counseling

Computer-based Case Simulations (CCS) is the world-class medical clinical simulation system. Computer-based Case Simulations for Psychological Counseling (CCSPC) is suited with psychological counseling settings. CCSPC is developed in this study in order to measure the subjects’ practical skills and decision-making styles during the psychological counseling progress, as well as the related knowledge of concepts, principles and methods, to get a whole assessment of the subjects’ counseling skills. C#2003 is used to realize the visualized interfaces, and SQL2005 is in charge of the database. The stability and reliability of CCSPC is considerably acceptable.

Pharmacist Licensure: Time to Step It Up?

Pharmacist Licensure: Time to Step It Up?

Assessing Potentially Dangerous Medical Actions With the Computer-Based Case Simulation Portion of the USMLE Step 3 Examination

The 2000 Institute of Medicine report on patient safety brought renewed attention to the issue of preventable medical errors, and subsequently specialty boards and the National Board of Medical Examiners were encouraged to play a role in setting expectations around safety education. This paper examines potentially dangerous actions taken by examinees during the portion of the United States Medical Licensing Examination Step 3 that is particularly well suited to evaluating lapses in physician decision making, the Computer-based Case Simulation (CCS). Descriptive statistics and a general linear modeling approach were used to analyze dangerous actions ordered by 25,283 examinees that completed CCS for the first time between November 2006 and January 2008. More than 20% of examinees ordered at least one dangerous action with the potential to cause significant patient harm. The propensity to order dangerous actions may vary across clinical cases. The CCS format may provide a means of collecting important information about patient-care situations in which examinees may be more likely to commit dangerous actions and the propensity of examinees to order dangerous tests and treatments.

Computer-Delivered Patient Simulations in the United States Medical Licensing Examination (USMLE)

To obtain a full and unrestricted license to practice medicine in the United States, students and graduates of the MD-granting US medical schools and of medical schools located outside of the United States must take and pass the United States Medical Licensing Examination. United States Medical Licensing Examination began as a series of paper-and-pencil examinations in the early 1990s and converted to computer-delivery in 1999. With this change to the computerized format came the opportunity to introduce computer-simulated patients, which had been under development at the National Board of Medical Examiners for a number of years. This testing format, called a computer-based case simulation, requires the examinee to manage a simulated patient in simulated time. The examinee can select options for history-taking and physical examination. Diagnostic studies and treatment are ordered via free-text entry, and the examinee controls the advance of simulated time and the location of the patient in the health care setting. Although the inclusion of this format has brought a number of practical, psychometric, and security challenges, its addition has allowed a significant expansion in ways to assess examinees on their diagnostic decision making and therapeutic intervention skills and on developing and implementing a reasonable patient management plan.