Script Concordance Test Research Articles

Examining the impact of sleep deprivation on medical reasoning’s performance among anaesthesiology residents and doctors: a prospective study

BackgroundWorking long consecutive hours’ is common for anaesthesia and critical care physicians. It is associated with impaired medical reasoning’s performance of anaesthesiology and serious medical errors. However, no study has yet investigated the impact of working long consecutive hours’ on medical reasoning.ObjectiveThe present study evaluated the impact of working long consecutive hours’ on the medical reasoning’s performance of anaesthesiology and intensive care physicians (residents and seniors).MethodsThis multicentric, prospective, cross-over study was conducted in 5 public hospitals of Normandy region. Two groups of anaesthesia and critical care physicians were formed. One was in a rest group, RG (after a 48-hours weekend without hospital work) and the other in Sleep Deprivation Group (SDG) after a 24 h-consecutives-shift. Changes in medical reasoning’s performance were measured by 69-items script concordance tests (SCT) through to the two tests. Group A completed the first part of the assessment (Set A) after a weekend without work and the second part (Set B) after a 24 h-shift; group B did the same in reverse order. The primary outcome was medical reasoning’s performance as measured by SCT in RG and SDG. The secondary outcomes included association between the performance with the demographic data, variation of the KSS (Karolinska sleepiness scale) daytime alertness score, the number of 24 h-shift during the previous 30 days, the vacations during the previous 30 days, the presence of more or less than 4 h consecutives hours slept, the management of a stressful event during the shift, the different resident years, the place where the shift took place (University hospital or general hospitals) and the type of shift (anaesthesia or intensive care).Results84 physicians (26 physicians and 58 residents) were included. RG exhibited significantly higher performance scores than SDG (68 ± 8 vs. 65 ± 9, respectively; p = 0.008). We found a negative correlation between the number of 24 h-shifts performed during the previous month and the variation of medical reasoning’s performance and no significant variation between professionals who slept 4 h or less and those who slept more than 4 h consecutively during the shift (-4 ± 11 vs. -2 ± 11; p = 0.42).ConclusionOur study suggests that medical reasoning’ performance of anaesthesiologists, measured by the SCT, is reduced after 24 h-shift than after rest period. Working long consecutive hours’ and many shifts should be avoided to prevent the occurrence of medical errors.

Script Concordance Test in Physiology: Preparation, Scoring, and Student Perceptions – A Mixed Method Study

Background: Although the curriculum has changed, assessment tools are not in alignment with the new types of teaching such as early clinical exposure (ECE) and self-directed learning. Both in summative and formative assessment most commonly used tools for assessment of cognitive domain are written formats including MCQ. However, these assessment tools such as MCQ and written essays cannot assess the higher order thinking skills and clinical reasoning skills. Materials and Methods: The present study was conducted in the department of physiology, in collaboration with medical simulation center, as a part of formative assessment for topics on pathophysiology of shock and obstructive and restrictive lung disorders taught during ECE using case scenarios. Two script concordance tests (SCTs) each on the topic of pathophysiological mechanism of different types of shock and obstructive and restrictive lung disorders each were prepared and administered to the students. Student perception to this assessment tools was obtained. Results: The Cronbach’s alpha of the 6-item SCT for respiratory and cardiovascular topic administered to the students (n = 107, 98) was 0.83 and 0.82, respectively. The inter-item correlation was respiratory and cardiovascular topic for 0.71, 0.69 suggesting that they are close and repetitive. Students were of the opinion that SCT are difficult to understand (75.8%), challenging to answer (47.2%), and it tested the clinical content better (71.4%). Conclusions: In our study, students felt though SCT tested the clinical content better when compared to conventionally used MCQ, they still find it challenging to understand and are not in favor of its use in summative and formative assessments.

Knowledge, confidence and clinical reasoning of dental students for diagnosing pediatric periodontal diseases

ObjectiveGiven the importance of diagnosing periodontal diseases in pediatric patients, the objective of this study was to examine knowledge, confidence, and clinical reasoning of third- and fourth-year dental students regarding periodontal diagnosis and treatment planning in pediatric patients. MethodsAn explanatory sequential mixed-method study was conducted in two phases. In the first phase, the survey questions were administered online to the third-and fourth-year dental students. Surveys tested for knowledge, confidence and clinical decision-making using questions and case-studies. Answers from periodontists and the pediatric dentists were used as the gold standard. For the next phase, individual interviews were recorded, transcribed, and coded followed by verbal analysis to assess their clinical reasoning for the specific diagnosis and treatment planning. ResultsWhile there was no overall difference in the knowledge and the confidence level between the students, they differed from the experts for some questions in confidence levels. As measured using the Script Concordance Test (SCT), no significant differences were observed between the participants in their clinical reasoning. The verbal analysis of the interview data showed that the students used three different patterns for searching through the problem space. Most of the students used the forward pattern of search and reported low confidence in diagnosing and treating pediatric periodontal diseases. ConclusionDue to low self-reported confidence in diagnosing periodontal disease in pediatric patients. Providing education solutions will improve the confidence of undergraduate dental students in diagnosing and treating periodontal disease.

Development of script concordance test for assessment of clinical reasoning in nursing: Lessons learned regarding construct validity

BackgroundThe script concordance test (SCT) has been shown to be an effective tool to assess the clinical reasoning skills of nursing students. Various nursing studies have demonstrated the construct validity of this test. However, studies on the barriers that may impede construct validity during the development process are limited. ObjectiveThis evaluation describes the barriers to the development of SCT for Bachelor's nursing students and the lessons learned regarding construct validity. MethodsWe conducted a descriptive evaluation of the SCT development and a validation process was performed. The evaluation was based on written comments during the assessment (N = 327), a Student's Perspective Questionnaire (N = 100), and student feedback during three live review sessions (N = 27). ResultsDespite consideration of the guidelines during SCT development, we encountered three main barriers that may impede construct validity. We undertook the necessary efforts to recruit an appropriate expert panel. We overestimated the experts' and students' understanding of the SCT methodology. Additionally, four potential causes of invalid item construction were identified. These possible causes were ‘questionable intervention, hypothesis, or investigation’, ‘blurred data in new information’, ‘regression to the middle’, and ‘misinterpretation of the midpoint’. ConclusionThe three lessons learned are as follows: 1) The recruitment of an appropriate expert panel must not be underestimated. Besides clinical expertise, experts need training in SCT methodology, including awareness of possible pitfalls; 2) SCT training is a prerequisite for SCT as an assessment; and 3) student feedback may offer a deeper understanding of potential hidden script errors and causes for misinterpretation of SCT. Further studies are necessary to identify additional causes which may impede the construct validity of SCT in nursing education.

Le panel d’experts pour les tests de concordance de script : une référence réellement adéquate ?

The Script Concordance Tests (SCTs) are an examination modality introduced by decree in the French National Ranking Exam for medical students in 2024. Their objective is to evaluate clinical reasoning in situations of uncertainty. In practice, SCTs assess the impact of new information on the probability of a hypothesis formulated a priori based on an authentic clinical scenario. This approach resembles probabilistic (or Bayesian) reasoning. Due to the uncertainty associated with the explored clinical situation, SCTs do not compare the student's response to an expected one in a theoretical knowledge reference. Instead, the distribution of responses from a panel of experienced physicians is used to establish the question's scoring scale. Literature data suggest that physicians, even experienced ones, like most humans, often exhibit biased intuitive probabilistic reasoning. These biases raise questions about the relevance of using expert panel responses as scoring scales for SCTs.

A Prompt for Generating Script Concordance Test Using ChatGPT, Claude, and Llama Large Language Model Chatbots

Medical education always evolves to incorporate more tools for specific needs in assessing clinical reasoning skills. Among these tools, Script Concordance Test (SCT) has a particular importance due to its focus on assessing decision-making in uncertain clinical situations. However, development of SCT items is effortful. Artificial intelligence tools, such as large language models, offer significant benefits. These models are already used for generating multiple-choice questions, and their use in generating SCTs offers great promise. However, this requires well-designed prompts to generate SCTs. This article proposes a generic prompt for the ChatGPT-4, Claude 3, Llama 3, and ChatGPT-4o large language model chatbots to generate SCTs, which can be tailored to various fields of medicine and different stages of medical education. It can help to streamline the development process of SCTs. Initial findings are promising, and there is a need for generating SCTs using large language models and conducting research to assess the quality of SCTs.

Digital Health Education for the Future: The SaNuRN (Santé Numérique Rouen-Nice) Consortium's Journey.

SaNuRN is a five-year project by the University of Rouen Normandy (URN) and the Côte d’Azur University (CAU) consortium to optimize digital health education for medical and paramedical students, professionals, and administrators. The project includes a skills framework, training modules, and teaching resources. In 2027, SaNuRN is expected to train a significant portion of the 400,000 health and paramedical professions students at the French national level. Our purpose is to give a synopsis of the SaNuRN initiative, emphasizing its novel educational methods and how they will enhance the delivery of digital health education. Our goals include showcasing SaNuRN as a comprehensive program consisting of a proficiency framework, instructional modules, and educational materials and explaining how SaNuRN is implemented in the participating academic institutions. SaNuRN is a project aimed at educating and training health-related and paramedics students in digital health. The project results from a cooperative effort between URN and CAU, covering four French departments. The project is based on the French National Referential on Digital Health (FNRDH), which defines the skills and competencies to be acquired and validated by every student in the health, paramedical, and social professions curricula. The SaNuRN team is currently adapting the existing URN and CAU syllabi to FNRDH and developing short-duration video capsules of 20 to 30 minutes to teach all the relevant material. The project aims to ensure that the largest student population earns the necessary skills, and it has developed a two-tier system involving facilitators who will enable the efficient expansion of the project’s educational outreach and support the students in learning the needed material efficiently. With a focus on real-world scenarios and innovative teaching activities integrating telemedicine devices and virtual professionals, SaNuRN is committed to enabling continuous learning for healthcare professionals in clinical practice. The SaNuRN team introduced new ways of evaluating healthcare professionals by shifting from a knowledge-based to a competencies-based evaluation, aligning with the Miller teaching pyramid and using the Objective Structured Clinical Examination and Script Concordance Test in digital health education. Drawing on the expertise of URN, CAU, and their public health and digital research laboratories and partners, the SaNuRN project represents a platform for continuous innovation, including telemedicine training and living labs with virtual and interactive professional activities. The SaNuRN project provides a comprehensive, personalized 30-hour training package for health and paramedical students, addressing all 70 FNRDH competencies. The program is enhanced using AI and NLP to create virtual patients and professionals for digital healthcare simulation. SaNuRN teaching materials are open-access. The project collaborates with academic institutions worldwide to develop educational material in digital health in English and multilingual formats. SaNuRN offers a practical and persuasive training approach to meet the current digital health education requirements.

Using ChatGPT in Psychiatry to Design Script Concordance Tests in Undergraduate Medical Education: Mixed Methods Study.

Undergraduate medical studies represent a wide range of learning opportunities served in the form of various teaching-learning modalities for medical learners. A clinical scenario is frequently used as a modality, followed by multiple-choice and open-ended questions among other learning and teaching methods. As such, script concordance tests (SCTs) can be used to promote a higher level of clinical reasoning. Recent technological developments have made generative artificial intelligence (AI)-based systems such as ChatGPT (OpenAI) available to assist clinician-educators in creating instructional materials. The main objective of this project is to explore how SCTs generated by ChatGPT compared to SCTs produced by clinical experts on 3 major elements: the scenario (stem), clinical questions, and expert opinion. This mixed method study evaluated 3 ChatGPT-generated SCTs with 3 expert-created SCTs using a predefined framework. Clinician-educators as well as resident doctors in psychiatry involved in undergraduate medical education in Quebec, Canada, evaluated via a web-based survey the 6 SCTs on 3 criteria: the scenario, clinical questions, and expert opinion. They were also asked to describe the strengths and weaknesses of the SCTs. A total of 102 respondents assessed the SCTs. There were no significant distinctions between the 2 types of SCTs concerning the scenario (P=.84), clinical questions (P=.99), and expert opinion (P=.07), as interpretated by the respondents. Indeed, respondents struggled to differentiate between ChatGPT- and expert-generated SCTs. ChatGPT showcased promise in expediting SCT design, aligning well with Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria, albeit with a tendency toward caricatured scenarios and simplistic content. This study is the first to concentrate on the design of SCTs supported by AI in a period where medicine is changing swiftly and where technologies generated from AI are expanding much faster. This study suggests that ChatGPT can be a valuable tool in creating educational materials, and further validation is essential to ensure educational efficacy and accuracy.

Evaluating clinical reasoning in first year DPT students using a script concordance test

BackgroundA script concordance test (SCT) provides a series of clinical vignettes to assess clinical reasoning in uncertainty. Appraised throughout health education literature, SCTs are cognitive assessments of clinical reasoning, though their use in Doctor of Physical Therapy (DPT) entry-level education has not been investigated. The purpose of this study was to develop and explore the reliability and validity of a SCT for first year DPT students.MethodsThe SCT was developed and implemented over four phases. During phases one and two, DPT program faculty consulted on course content from the first-year curriculum. Thirty clinical vignettes with three follow-up questions each were constructed. The SCT was pilot tested with five clinicians in phase three to assess question clarity. During phase four, the SCT was administered to students and a reference panel via Qualtrics. First year DPT students (n = 44) and reference panel physical therapists with at least two years of experience and advanced certification (n = 15) completed the SCT. Internal consistency was analyzed using Cronbach’s Alpha. Differences between student and reference panel percent-correct scores were analyzed with a t-test. Relationships between student SCT scores and academic records were explored with Spearman’s Rho.ResultsThe SCT had an internal consistency of 0.74. A significant difference in scores was found between the students [mean 58.5 (+/-5.31)] and reference panel [65.8 (+/-4.88), p < .01]. No significant correlations between student SCT scores and academic records were found.ConclusionsThe developed SCT was reliable and demonstrated satisfactory internal consistency among test items. The SCT successfully differentiated between groups, with the reference panel demonstrating statistically significant higher percent-correct scores compared to students. SCTs may provide means to measure clinical reasoning in DPT students and lead to novel pedagogical approaches to enhance clinical reasoning.

Comparison between MCQ, Extended matching questions (EMQ) and Script concordance test (SCT) for assessment among first-year medical students - A pilot study.

Single-best response types of MCQs are commonly used tools in medical assessment. However, these are not particularly apt for the assessment of higher-order thinking skills (HOTS) among students. Assessment of HOTS and clinical reasoning skills requires unexplored tools like extended matching questions (EMQs) and SCTs. The aim is to assess HOTS and clinical reasoning skills during formative assessment among medical students post a simulation-based education (SBE) using EMQ and SCT on the topic of shock and collect student perceptions regarding new assessment tools. The research is an observational descriptive study. Fifty-two first-year medical students were asked to take a formative assessment consisting of 20 MCQs, 6 EMQs, and 2 SCT post a SBE during July 2022 on the topic of shock. MCQs were categorized into themes of aetiology, pathophysiology, and management of shock. These categorized MCQs were compared and analyzed with EMQs and SCTs prepared on the same themes. The data analysis by a one-way analysis of variance (ANOVA) was conducted to compare the difference in per cent mean scores of MCQ with EMQ and MCQ with SCT. The overall mean scores and also under the theme of aetiology and management, the student's score in MCQ were higher than EMQ and SCT and the difference was statistically significant with the P value (≤0.001). Students scored better in familiar assessment tools like MCQ, but majority of the students were of the opinion that EMQ tested the delivered content better and SCT tested the clinical application better.

The Impact of Script Concordance Testing on Clinical Decision-Making in Paramedic Education.

This study investigates the effectiveness of the Script Concordance Test (SCT) in enhancing clinical reasoning skills within paramedic education. Focusing on the Medical University of Lublin, we evaluated the SCT's application across two cohorts of paramedic students, aiming to understand its potential to improve decision-making skills in emergency scenarios. Our approach, informed by Van der Vleuten's assessment framework, revealed that while the SCT's correlation with traditional methods like multiple-choice questions (MCQs) was limited, its formative nature significantly contributed to improved performance in summative assessments. These findings suggest that the SCT can be an effective tool in paramedic training, particularly in strengthening cognitive abilities critical for emergency responses. The study underscores the importance of incorporating innovative assessment tools like SCTs in paramedic curricula, not only to enhance clinical reasoning but also to prepare students for effective emergency responses. Our research contributes to the ongoing efforts in refining paramedic education and highlights the need for versatile assessment strategies in preparing future healthcare professionals for diverse clinical challenges.

Outcome measures in clinical education research

. The Journal of Clinical Education in Physical Therapy (JCEPT) is an open-access journal that disseminates peer-reviewed quantitative and qualitative scholarship to increase the quality evidence available related to clinical education in physical therapy. Since 2021, the number of article downloads has increased by more than 10-fold, with annual increases exceeding 200%. Open-access allows JCEPT readers free access to both read online and download full-text articles. Last year, one article was viewed over 900 times while another was downloaded nearly 1,200 times. The Directory of Open Access Journals (DOAJ) began indexing JCEPT last year and now accounts for 14% of our referral sources. Next year, we expect to obtain indexing in PubMed Central (PMC), which will further increase the discoverability of JCEPT articles. The nature of clinical education raises various challenges to research, with one being how to assess outcomes. Articles published this year in JCEPT reflect the thoughtful application of different approaches to outcome measures. Some questions were assessed with quantitative measures while others used qualitative research methods. Self-efficacy was measured in two different articles exploring the domains of cross-cultural psychological capital and clinical competence. Clinical reasoning was assessed using the physical therapy Clinical Performance Instrument in one article and script concordance tests in another. A more recently established outcome such as inter-professional socialization was assessed with a quantitative scale while emotional and educational adaptability themes were explored with qualitative methods. In past volumes, a SNAPPS model (summarize, narrow, analyse, probe, plan, select) of clinical mentoring was used to help students process initial evaluation findings and longitudinal narrative analyses were used to assess the development of resident clinical reasoning skills. Outcome measures assessing the effectiveness of clinical education are essential.1 Efforts may start with the thoughtful choice of psychometrically supported quantitative measures as well as rigorous description of qualitative measures.2 However, recent JCEPT articles suggest many factors can influence the quality of a clinician,1 including their educational curriculum, pre- clinical training, the clinical environment of their patient exposures, the individual characteristics and stresses of students and their clinical instructors, as well as environmental forces such as the COVID pandemic. Translating the effectiveness of clinical education to improved clinical care and patient outcomes remains a challenge for future researchers. JCEPT publishes scholarly work affecting any aspect of clinical education in physical therapy addressing academic or clinical faculty, entry-level students or residents, curricula or teaching and learning methods. In addition to original research, systematic reviews, and innovative teaching methods, this year JCEPT published our first ‘clinical education case report’ describing remediation methods during clinical training and related outcomes. We look forward to publishing more papers and developing the evidence underpinning physical therapy clinical education.

Validation of a French questionnaire assessing knowledge of suicide

ObjectivesThe objective of this study was to develop and validate the Knowledge of Suicide Scale (KSS), elaborated to assess adherence to myths about suicide. MethodsThe KSS is a self-questionnaire including 22 statements relating to myths about suicide for which the respondent is asked to rate his degree of adherence on a scale ranging from 0 (“strongly disagree”) to 10 (“completely agree”). Using the script concordance test scoring method, the respondents’ scores were compared with those of experts to obtain, for each item, a score between 0 (maximum deviation with the experts) and 1 (minimum deviation with the experts). One thousand and thirty-five individuals (222 psychiatric interns, 332 medical interns in the first semester excluding psychiatry and 481 journalism students) were included. ResultsAccording to the exploratory factor analysis, the KSS is a two-dimensional scale: the first subscale includes 15 items and the second seven items. The tool showed excellent face validity, correct convergent and divergent validities (multi-method multi-feature analyzes), and good internal consistency (Cronbach's alpha coefficient between 0.66 and 0.83 for scales and subscales). The KSS is moderately and negatively correlated with the Stigma of Suicide Scale (r=–0.3). It significantly discriminates groups with different expected levels of knowledge regarding suicide (P<0.001). ConclusionsThe KSS demonstrated good psychometric properties to measure adherence to myths about suicide. This tool could be useful in assessing the effectiveness of suicide prevention literacy improvement programs.

453 Innovating Emergency MedicineEducation: Leveraging Online Trainingand Script Concordance Testing forEnhanced Skill Development in theCOVID-19 Era

453 Innovating Emergency MedicineEducation: Leveraging Online Trainingand Script Concordance Testing forEnhanced Skill Development in theCOVID-19 Era

Test de concordance de script et évaluation du raisonnement clinique pour les cancers bronchiques localement avancés: Script Concordance Test and assessment of clinical reasoning in locally advanced non-small cell lung cancer

Script concordance tests (SCTs) represent a realistic and representative means of assessing healthcare professionals’ ability to solve clinical problems in situations of uncertainty. The advantages of SCTs include their reliability, internal validity, and their application across various medical domains. In the field of medical oncology, their use has remained limited to date. The complexity of therapeutic strategy choices in the specific context of locally advanced NSCLC, as illustrated by several recent studies on the reproducibility of MDT decisions, makes this particular theme an ideal application for SCTs. We propose to apply this method during an intervention at the GOLF 2023 conference, which could help better understand the determinants of clinical reasoning and, consequently, improve therapeutic decision-making in the precise management of locally advanced NSCLC.1877-1203/© 2023 SPLF. Published by Elsevier Masson SAS. All rights reserved.

Scoring Methods in Script Concordance Tests: An Exploratory Psychometric Study.

Despite the increasingly popular role of script concordance test (SCT) scoring methods in the evaluation of clinical reasoning, studies examining these methods in nursing are relatively scarce. This study explored the psychometric properties of five SCT scoring methods. An SCT was administered to 12 experts and 43 learners. Scores were calculated using five methods and descriptive statistics. Differences in scores were assessed with the Mann-Whitney U test, and Spearman correlation coefficients were calculated for the different methods. The median scores of both experts and learners differed substantially according to the scoring method used. Learners' scores were statistically different from experts' scores (p < .01) for each method. Spearman coefficients (range, 0.44 to 0.95) were positive for the different methods. Further research is needed to refine the influence of SCT scoring methods for use in certifying assessment of clinical reasoning in nursing. [J Nurs Educ. 2023;62(10):549-555.].

Modified endocrinology script concordance test: evaluating the reliability and construct validity for assessing clinical reasoning.

Modified endocrinology script concordance test: evaluating the reliability and construct validity for assessing clinical reasoning.

Impact of reference panel composition on scores of script concordance test assessing basic nephrology knowledge in undergraduate medical education

Purpose In the assessment of basic medical knowledge, the composition of the reference panel between specialists and primary care (PC) physicians is a contentious issue. We assessed the effect of panel composition on the scores of undergraduate medical students in a script concordance test (SCT). Methods The scale of an SCT on basic nephrology knowledge was set by a panel of nephrologists or a mixed panel of nephrologists and PC physicians. The results of the SCTs were compared with ANOVA for repeated measurements. Concordance was assessed with Bland and Altman plots. Results Forty-five students completed the SCT. Their scores differed according to panel composition: 65.6 ± 9.73/100 points for nephrologists, and 70.27 ± 8.82 for the mixed panel, p < 0.001. Concordance between the scores was low with a bias of −4.27 ± 2.19 and a 95% limit of agreement of −8.96 to −0.38. Panel composition led to a change in the ranking of 71% of students (mean 3.6 ± 2.6 places). Conclusion The composition of the reference panel, either specialist or mixed, for SCT assessment of basic knowledge has an impact on test results and student rankings.

Script concordance testing in genetic counseling training: Apilot study.

Clinical reasoning is a complex skill that represents a trainee's ability to use their professional knowledge and skills to assess and solve the problems that arise in clinical practice. As an integral tenet of the genetic counseling process, clinical reasoning skills underlie many of the Practice-Based Competencies (2019) across a variety of domains. Despite the long-lasting recognition of the importance of this complex skill in the training of genetic counselors, clinical reasoning has traditionally been difficult to assess in a standardized way in healthcare education. Script concordance testing is a standardized method of assessing clinical reasoning skills in ambiguous clinical situations. The tool has been used to successfully measure the clinical reasoning skills of trainees in various healthcare training programs and has never been used in a genetic counseling training program. We conducted a pilot study to assess the utility of script concordance testing in the field of genetic counseling as an objective measure of clinical reasoning in trainees. The script concordance test was constructed for the field of genetic counseling and administered to 22 second year genetic counseling students in the Joan H. Marks Graduate Program in Human Genetics at Sarah Lawrence College. Twelve genetic counselors served on a panel to provide expert clinical reasoning responses and a scoring grid was developed using the aggregate scores method. The utility of the tool was measured using Cronbach's alpha coefficient, and scores of students and the panel were compared using Hedge's g. Results revealed statistically significant differences between the scores of panelists and students and good reliability. This study shows that script concordance testing can be used to measure clinical reasoning skills in genetic counseling trainees in a way that is reliable, standardized, and easy to use, thereby allowing programs to better assess the clinical reasoning skills of trainees prior to graduation.

A Video-Based Assessment Tool to Measure Intraoperative Laparoscopic Suturing Using a Modified Script Concordance Methodology

Laparoscopic suturing (LS) is a challenging laparoscopic skill to teach. Its complexity and nuances are not modeled or measured in current simulation and assessment platforms.The script concordance test (SCT) is used to assess clinical reasoning.The purpose of this study is to provide evidence for validity of this novel SCT based online assessment for LS skills. We designed a video-based online SCT for LS using a cognitive task analysis and expert panelists.The CTA yielded 4 LS domains: needle handling (NH), tissue handling (TH), knot tying techniques (KT) and operative ergonomics (OE). Five-point scales with anchoring descriptors from -2 to +2 were used. Scoring was based on a modified SCT methodology. The test was administrated to 37 subjects (18 experts and 19 novices). There was no time limit given. A different expert group from the minimal invasive surgery (MIS) panelist were recruited. Experts were defined as surgeons and fellows with LS experience of >25 cases annually. Validity was assessed by comparing SCT scores of experienced and inexperienced surgeons. Cronbach's alpha was used to assess the internal consistency of the test. The survey started off with 47 questions in each of the following domains: 13 NH, 4 TH, 20 KT and 10 OE. Thirty-seven surgeons (18 experts and 19 inexperienced surgeons). Questions that demonstrated a large discrepancy among experts and panelists with a weighted score difference more than 40 were discarded (n = 20). One question was discarded because it received a 100% score from all participants. This yielded 26 remaining questions in the following domains: 8 NH, 2 TH, 11 KT and 5 OE. The test reliability level (Cronbach a) was 0.80. The mean score was 72 ± 9% and 63 ± 15% (p = 0.02) for experts and inexperienced surgeons, respectively. The mean time to complete the test was 21 minutes. This study provides validity evidence for a novel intraoperative LS assessment. The variability of responses between experts and panelists suggests that SCT may capture the clinical differences/surgeon preferences in performing LS intraoperatively.