Script Concordance Test Research Articles

How do cognitive processes influence script concordance test responses?

The script concordance test (SCT) is a test of clinical decision-making (CDM) that compares the thought process of learners to that of experts to determine to what extent their cognitive 'scripts' align. Without understanding test-takers' cognitive process, however, it is unclear what influences their responses. The objective of this study was to gather response process validity evidence by studying the cognitive process of test-takers to determine whether the SCT tests CDM and what cognitive processes may influence SCT responses. Cases from an SCT used in a national validation study were administered and semi-structured cognitive interviews were conducted with ten residents and five staff surgeons. A retrospective verbal probing technique was used. Data was independently analysed and coded by two analysts. Themes were identified as factors that influence SCT responses during the cognitive interview. Cognitive interviews demonstrated variability in CDM among test-takers. Consistent with dual process theory, test-takers relied on scripts formed through past experiences, when available, to make decisions and used conscious deliberation in the absence of experience. However, test-takers' response process was also influenced by their comprehension of specific terms, desire for additional information, disagreement with the planned management, underlying knowledge gaps and desire to demonstrate confidence or humility. The rationale behind SCT answers may be influenced by comprehension, underlying knowledge and social desirability in addition to formed scripts and/or conscious deliberation. Having test-takers verbalise their rationale for responses provides a depth of assessment that is otherwise lost in the SCT's current format. With the improved ability to standardise CDM assessment using the SCT, consideration of test-makers improving the SCT construction process and combining the SCT question format with verbal responses may improve the use of the SCT for CDM assessment.

Resident competencies before and after short intensive care unit rotations: a multicentre pilot observational study.

Residency programs need to understand the competencies developed by residents during an intensive care unit (ICU) rotation, so that curricula and assessments maximize residents' learning. The primary study objective was to evaluate the feasibility for training programs and acceptability by residents of conducting a multi-competency assessment during a four-week ICU rotation. We conducted a prospective, multicentre observational pilot study in three ICUs. During weeks 1 and 4 of an ICU rotation, we conducted repeated standardized assessments of non-critical care specialty residents' competencies in cognitive reasoning (script concordance test [SCT]), procedural skills (objective structured assessment of technical skills [OSATS]-global rating scale], and communication skills through a written test, two procedural simulations, and a simulated encounter with a "family member". The feasibility outcomes included program costs, the proportion of enrolled residents able to complete at least one three-station assessment during their four-week ICU rotation, and acceptability of the assessment for the trainees. We enrolled 63 (69%) of 91 eligible residents, with 58 (92%) completing at least one assessment. The total cost to conduct 90 assessments was CAD 33,800. The majority of participants agreed that the assessment was fair and that it measured important clinical abilities. For the 32 residents who completed two assessments, the mean (standard deviation) cognitive reasoning and procedural skill scores increased between weeks 1 and 4 [SCT difference, 3.1 (6.5), P = 0.01; OSATS difference for bag-mask ventilation and central line insertion, 0.4 (0.5) and 0.6 (0.8), respectively; both P ≤ 0.001]. Nevertheless, the communication scores did not change significantly. A monthly multi-competency assessment for specialty residents rotating in the ICU is likely feasible for most programs with appropriate resources, and generally acceptable for residents. Specialty residents' cognitive reasoning and procedural skills may improve during a four-week ICU rotation, whereas communication skills may not.

Pediatric Hospitalists' Performance and Perceptions of Script Concordance Testing for Self-Assessment.

The cognitive expertise of Pediatric Hospitalists (PH) lies not in standard knowledge but in making decisions under conditions of uncertainty. To maintain expertise, PH should engage in deliberate practice via self-assessments that promote higher-level cognitive processes necessary to address problems with missing or ambiguous information. Higher levels of cognition are purported with Script Concordance Test (SCT) questions compared to Multiple Choice Questions (MCQ). To determine if PH use higher levels of cognition when answering SCT versus MCQ questions and to analyze participants' perceptions of the utility of using SCT self-assessment for deliberate practice in addressing clinical problems encountered in daily practice. This is a mixed methods study comparing the cognitive level expressed according to Bloom's Taxonomy by PH answering MCQ versus SCT questions using a "think aloud" (TA) exercise, followed by qualitative analysis of interviews conducted afterward. A significantly greater percentage of comments were coded as higher cognitive processes (apply, analyze, evaluate, and create) for SCT versus MCQ (74% vs 19%) compared with lower order (remember, understand); chi-square P < .00001. Analysis of interviews revealed 6 themes. SCT questions elicited higher level cognition essential to clinical reasoning compared to MCQ questions. PH-indicated MCQ questions measure standard knowledge, while SCT questions better measure decision-making under conditions of uncertainty. PH-perceived SCT could be useful for deliberate practice in Pediatric Hospital Medicine decision-making if they could compare their rationale in answering questions with that of experts.

Learning strategies used by undergraduate nursing students in the context of a digitial educational strategy based on script concordance: A descriptive study

Backgroundthe digital educational strategy based on script concordance is an educational method that has been attracting increasing attention in healthcare education programs to fostering the development of clinical reasoning. It includes a digitized Script Concordance Test with incorporated expert feedback. However, the learning strategies required of students in the context of its use remain unknown. ObjectiveThis study aimed to identify the learning strategies that undergraduate nursing students need to use in the context of the digital educational strategy based on script concordance. MethodA qualitative descriptive design was used to identify student learning strategies. Data was collected using an online questionnaire and semi-directed focus group interviews. Bégin's taxonomy provided the framework for linking the data collected to learning strategies required of students. ResultsForty-four students participated in the study. Results show that when using a digital educational strategy based on script concordance, students are called to rely on their nascent scripts in order to select the data in short ill-defined clinical vignettes, evaluate new information repeatedly, anticipate microjudgments, and thus, gradually increase their knowledge and refine their scripts. Viewing the experts' feedback and consulting the referencing tools helped students self-monitor their knowledge, a key metacognitive strategy to learning clinical reasoning. Completed individually or with peers, the digital educational strategy could be used to learn a particular concept or as an integrative activity before an evaluation. ConclusionThis original study has allowed us to link nursing clinical reasoning teaching conditions to the learning strategies used to develop this competency. Study results inform instructors about digital educational strategy based on script concordance to make it complementary with other educational strategies to better support complex learning of nursing clinical reasoning.

Impact of panelists\u2019 experience on script concordance test scores of medical students

BackgroundThe evaluation process of French medical students will evolve in the next few years in order to improve assessment validity. Script concordance testing (SCT) offers the possibility to assess medical knowledge alongside clinical reasoning under conditions of uncertainty. In this study, we aimed at comparing the SCT scores of a large cohort of undergraduate medical students, according to the experience level of the reference panel.MethodsIn 2019, the authors developed a 30-item SCT and sent it to experts with varying levels of experience. Data analysis included score comparisons with paired Wilcoxon rank sum tests and concordance analysis with Bland & Altman plots.ResultsA panel of 75 experts was divided into three groups: 31 residents, 21 non-experienced physicians (NEP) and 23 experienced physicians (EP). Among each group, random samples of N = 20, 15 and 10 were selected. A total of 985 students from nine different medical schools participated in the SCT examination. No matter the size of the panel (N = 20, 15 or 10), students’ SCT scores were lower with the NEP group when compared to the resident panel (median score 67.1 vs 69.1, p < 0.0001 if N = 20; 67.2 vs 70.1, p < 0.0001 if N = 15 and 67.7 vs 68.4, p < 0.0001 if N = 10) and with EP compared to NEP (65.4 vs 67.1, p < 0.0001 if N = 20; 66.0 vs 67.2, p < 0.0001 if N = 15 and 62.5 vs 67.7, p < 0.0001 if N = 10). Bland & Altman plots showed good concordances between students’ SCT scores, whatever the experience level of the expert panel.ConclusionsEven though student SCT scores differed statistically according to the expert panels, these differences were rather weak. These results open the possibility of including less-experienced experts in panels for the evaluation of medical students.

Predictors of clinical reasoning in neurological localisation: A study in internal medicine residents

Introduction: Clinical reasoning is the cognitive process of weighing clinical information together with past experience to evaluate diagnostic and management dilemmas. There is a paucity of literature regarding predictors of clinical reasoning at the postgraduate level. We performed a retrospective study on internal medicine residents to determine the sociodemographic and experiential correlates of clinical reasoning in neurological localisation, measured using validated tests. Methods: We recruited 162 internal medicine residents undergoing a three-month attachment in neurology at the National Neuroscience Institute, Singapore, over a 2.5 year period. Clinical reasoning was assessed on the second month of their attachment via two validated tests of neurological localisation–Extended Matching Questions (EMQ) and Script Concordance Test (SCT). Data on gender, undergraduate medical education (local vs overseas graduates), graduate medical education, and amount of clinical experience were collected, and their association with EMQ and SCT scores evaluated via multivariate analysis. Results: Multivariate analysis indicated that local graduates scored higher than overseas graduates in the SCT (adjusted R2 = 0.101, f2 = 0.112). Being a local graduate and having more local experience positively predicted EMQ scores (adjusted R2 = 0.049, f2 = 0.112). Conclusion: Clinical reasoning in neurological localisation can be predicted via a two-factor model–undergraduate medical education and the amount of local experience. Context specificity likely underpins the process.

Factors Influencing Inpatient Insulin Management of Adults With Type 1 and Type 2 Diabetes by Residents and Medical Students

ObjectivesThe day-to-day management of diabetes is often the responsibility of resident physicians in teaching hospitals, yet studies have consistently shown that learners lack confidence and knowledge of inpatient insulin titration. We sought to elucidate factors that influence inpatient management of diabetes by medical learners. We predicted that there would be conscious and unconscious variables at play, and we sought to determine the process by which learners arrived at insulin titration decisions. MethodsWe administered a survey using a hypothetical case based on script concordance testing in 88 medical students and residents at the University of Calgary. We used multilevel, mixed-effects linear regression and random-effects meta-analysis to compare clerks’ and residents’ insulin prescribing, confidence in prescribing decisions and concordance with endocrinologists. ResultsResidents had higher confidence than clerks, yet were less concordant with the insulin decisions of staff endocrinologists. Both residents and clerks prescribed more insulin when the reported blood glucose was higher, or the patient had positive ketones and prescribed less insulin when the patient had hypoglycemic unawareness. Residents prescribed a higher dose of insulin when the charge nurse reported the result compared with a student nurse, but this variable did not significantly impact prescribing by clerks. ConclusionsWe propose a model in which learner factors (knowledge, confidence, experience), patient variables and perceived expectations influence insulin titration decisions via analytical and heuristic processes. We propose that educational interventions designed to improve inpatient glycemic control must, therefore, address both knowledge and systems factors.

Script Concordance Test in Pharmacology: Maiden experience from a Medical School in India.

Introduction: Script concordance test (SCT) is an innovative tool to teach and assess the clinical reasoning skills of medical students. It is the key aspect of clinical competency that enables the medical graduates to progress from novice to practicing general practitioner. SCT was used the first time in pharmacology to inculcate clinic reasoning skills in medical students by focusing on the topic of pharmacotherapy. Methods: A SCT with a total of 18 questions, with 15 questions having 3 items each, one having four items and two questions having two items each was administered to 170 second year undergraduate medical students in the subject of pharmacology to assess the clinical reasoning skills. It was an interventional study conducted using convenience sampling technique with a sample size of 170. Aggregate scoring method was used to do the scoring obtained from the answers given by 10 expert-panel members in the field of pharmacology, which were used as an answer key to do the final scoring of the students. Descriptive statistics were computed using Students t test. SCT conduct included a small-group feedback session to the students post-SCT. A feedback questionnaire was administered to the students one week after the feedback session. Reliability of the SCT and feedback questionnaire was checked by calculating Cronbach alpha through Siegle reliability calculator. Content validity of the test as well as feedback questionnaire was done by the panel of experts included in the study.Results: Though the mean score of the students (27.68±4.59) was significantly lower than the expert panel (40.91±3.52), students were highly satisfied as they expressed that SCT enhanced their perceived clinical reasoning skills (median value=5) and critical thinking (median value=4). The Cronbach alpha for the test was 0.76. The students were also highly satisfied with the feedback given by the teachers after the SCT (median value=4).Conclusion: SCT enhances critical thinking and clinical reasoning skills of the students, as reported by them. With the conduct of feedback session post-SCT, it can be used as assessment for learning tool and can be well used in a para-clinical subject of pharmacology.

Examining response process validity of script concordance testing: a think-aloud approach.

ObjectivesThis study investigated whether medical student responses to Script Concordance Testing (SCT) items represent valid clinical reasoning. Using a think-aloud approach students provided written explanations of the reasoning that underpinned their responses, and these were reviewed for concordance with an expert reference panel. MethodsA set of 12, 11 and 15 SCT items were administered online to Year 3 (2018), Year 4 (2018) and Year 3 (2019) medical students respectively. Students' free-text descriptions of the reasoning supporting each item response were analysed, and compared with those of the expert panel. Response process validity was quantified as the rate of true positives (percentage of full and partial credit responses derived through correct clinical reasoning); and true negatives (percentage of responses with no credit derived through faulty clinical reasoning). ResultsTwo hundred and nine students completed the online tests (response rate = 68.3%). The majority of students who had chosen the response which attracted full or partial credit also provided justifications which were concordant with the experts (true positive rate of 99.6% for full credit; 99.4% for partial credit responses). Most responses that attracted no credit were based on faulty clinical reasoning (true negative of 99.0%). ConclusionsThe findings provide support for the response process validity of SCT scores in the setting of undergraduate medicine. The additional written think-aloud component, to assess clinical reasoning, provided useful information to inform student learning. However, SCT scores should be validated on each testing occasion, and in other contexts.

Assessment of Emergency Medicine Residents' Clinical Reasoning: Validation of a Script Concordance Test.

IntroductionA primary aim of residency training is to develop competence in clinical reasoning. However, there are few instruments that can accurately, reliably, and efficiently assess residents’ clinical decision-making ability. This study aimed to externally validate the script concordance test in emergency medicine (SCT-EM), an assessment tool designed for this purpose.MethodsUsing established methodology for the SCT-EM, we compared EM residents’ performance on the SCT-EM to an expert panel of emergency physicians at three urban academic centers. We performed adjusted pairwise t-tests to compare differences between all residents and attending physicians, as well as among resident postgraduate year (PGY) levels. We tested correlation between SCT-EM and Accreditation Council for Graduate Medical Education Milestone scores using Pearson’s correlation coefficients. Inter-item covariances for SCT items were calculated using Cronbach’s alpha statistic.ResultsThe SCT-EM was administered to 68 residents and 13 attendings. There was a significant difference in mean scores among all groups (mean + standard deviation: PGY-1 59 + 7; PGY-2 62 + 6; PGY-3 60 + 8; PGY-4 61 + 8; 73 + 8 for attendings, p < 0.01). Post hoc pairwise comparisons demonstrated that significant difference in mean scores only occurred between each PGY level and the attendings (p < 0.01 for PGY-1 to PGY-4 vs attending group). Performance on the SCT-EM and EM Milestones was not significantly correlated (r = 0.12, p = 0.35). Internal reliability of the exam was determined using Cronbach’s alpha, which was 0.67 for all examinees, and 0.89 in the expert-only group.ConclusionThe SCT-EM has limited utility in reliably assessing clinical reasoning among EM residents. Although the SCT-EM was able to differentiate clinical reasoning ability between residents and expert faculty, it did not between PGY levels, or correlate with Milestones scores. Furthermore, several limitations threaten the validity of the SCT-EM, suggesting further study is needed in more diverse settings.

The cognitive process of test takers when using the script concordance test rating scale.

Clinical decision making (CDM) skills are important to learn and assess in order to establish competence in trainees. A common tool for assessing CDM is the script concordance test (SCT), which asks test takers to indicate how a new clinical finding influences a proposed plan using a Likert-type scale. Most criticisms of the SCT relate to its rating scale but are largely theoretical. The cognitive process of test takers when selecting their responses using the SCT rating scale remains understudied, but is essential to gathering validity evidence for use of the SCT in CDM assessment. Cases from an SCT used in a national validation study were administered to 29 residents and 14 staff surgeons. Semi-structured cognitive interviews were then conducted with 10 residents and five staff surgeons based on the SCT results. Cognitive interview data were independently coded by two data analysts, who specifically sought to elucidate how participants mapped their internally generated responses to any of the rating scale options. Five major issues were identified with the response matching cognitive process: (a) the meaning of the '0' response option; (b) which response corresponds to agreement with the planned management; (c) the rationale for picking '±1' versus '±2'; (d) which response indicates the desire to undertake the planned management plus an additional procedure, and (e) the influence of time on response selection. Studying how test takers (experts and trainees) interpret the SCT rating scale has revealed several issues related to inconsistent and unintended use. Revising the scale to address the variety of interpretations could help to improve the response process validity of the SCT and therefore improve the SCT's ability to be used in CDM skills assessments.

Case-Based Teaching: Does the Addition of High-Fidelity Simulation Make a Difference in Medical Students' Clinical Reasoning Skills?

Situativity theory posits that learning and the development of clinical reasoning skills are grounded in context. In case-based teaching, this context comes from recreating the clinical environment, through emulation, as with manikins, or description. In this study, we sought to understand the difference in student clinical reasoning abilities after facilitated patient case scenarios with or without a manikin. Fourth-year medical students in an internship readiness course were randomized into patient case scenarios without manikin (control group) and with manikin (intervention group) for a chest pain session. The control and intervention groups had identical student-led case progression and faculty debriefing objectives. Clinical reasoning skills were assessed after the session using a 64-question script concordance test (SCT). The test was developed and piloted prior to administration. Hospitalist and emergency medicine faculty responses on the test items served as the expert standard for scoring. Ninety-six students were randomized to case-based sessions with (n = 48) or without (n = 48) manikin. Ninety students completed the SCT (with manikin n = 45, without manikin n = 45). A statistically significant mean difference on test performance between the two groups was found (t = 3.059, df = 88, p = .003), with the manikin group achieving higher SCT scores. Use of a manikin in simulated patient case discussion significantly improves students' clinical reasoning skills, as measured by SCT. These results suggest that using a manikin to simulate a patient scenario situates learning, thereby enhancing skill development.

Contribution of an Early Internal Medicine Rotation to the Clinical Reasoning Learning for Young Residents.

Clinical reasoning is the cornerstone of medical practice, and achieving this competence depends on a large number of factors. Internal medicine departments provide junior doctors with plentiful and varied patients, offering a comprehensive basis for learning clinical reasoning.In order to evaluate the usefulness of an early rotation at internal medicine departments, we compared, via script concordance tests, the evolution of residents’ clinical reasoning after an initial internal medicine rotation compared to rotations through other medical specialties.Twenty-two residents were tested after six months of their internal medicine rotation and compared to twenty-five residents that had the first rotation in another specialty (control). We showed a significant difference in the improvement of the script concordance tests scores (p=0.015) between the beginning and the end of their first rotation between the internal medicine and the control groups, and this implies the lower improvement of clinical reasoning skills and spontaneous learning slope of the junior doctors in other departments.

Taking CBL to the Lecture Hall: a Comparison of Outcomes Between Traditional Small Group CBL and a Novel Large Group Team-Based CBL Teaching Method.

Case-based learning (CBL), an important component of medical school curricula, is an effective inquiry-based teaching method associated with high levels of student and teacher satisfaction. However, because traditional CBL requires small groups, its feasibility is limited by faculty and resources. We developed and tested a novel team-based CBL (TB-CBL) method to be implemented in the lecture hall. All second-year students at our institution (n = 121) were randomized to either traditional small group CBL or TB-CBL during the Endocrine block and to the other modality during the Renal block. All students were exposed to both methods. Case content was identical, and sessions were run concurrently. This cross-over, non-inferiority study tested the hypothesis that no difference in knowledge acquisition, clinical reasoning, or student satisfaction would be detected between groups. Based on student performance on case-relevant exam questions, no difference in knowledge acquisition was seen between groups for either block (p = 0.62 Endocrine, p = 0.38 Renal). There was also no difference in overall final exam performance between groups (p = 0.56 Endocrine, p = 0.26 Renal). Case-relevant script concordance testing revealed no difference in clinical reasoning skills between groups (p = 0.87 Endocrine, p = 0.17 Renal). Satisfaction was higher for the TB-CBL format (p = 0.005). Cost analysis revealed that each small group CBL session costs $2654, while each TB-CBL session costs approximately $221. TB-CBL, a novel case-based teaching method, appears to produce similar learner outcomes and higher student satisfaction when compared with small group CBL. TB-CBL may be used to supplement case-based curricula while optimizing resource allocation.

Assessment methods in medical specialist assessments in the DACH region - overview, critical examination and recommendations for further development.

Introduction: Specialist medical assessments fulfil the task of ensuring that physicians have the clinical competence to independently represent their field and provide the best possible care to patients, taking into account the current state of knowledge. To date, there are no comprehensive reports on the status of specialist assessments in the German-speaking countries (DACH). For that reason, the assessment methods used in the DACH region are compiled and critically evaluated in this article, and recommendations for further development are described.Methods: The websites of the following institutions were searched for information regarding testing methods used and the organisation of specialist examinations: Homepage of the Swiss Institute for Medical Continuing Education (SIWF), Homepage of the Academy of Physicians (Austria) and Homepage of the German Federal Medical Association (BAEK). Further links were considered and the results were presented in tabular form. The assessment methods used in the specialist assessments are critically examined with regard to established quality criteria and recommendations for the further development of the specialist assessments are derived from these.Results: The following assessment methods are already used in Switzerland and Austria: written examinations with multiple choice and short answer questions, structured oral examinations, the Script Concordance Test (SCT) and the Objective Structured Clinical Examination (OSCE). In some cases, these assessment methods are combined (triangulation). In Germany, on the other hand, the oral examination has so far been conducted in an unstructured manner in the form of a ‘collegial content discussion’. In order to test knowledge, practical and communicative competences equally, it is recommended to implement a triangulation of methods and follow the further recommendations described in this article.Conclusion: While there are already accepted approaches for quality-assured and competence-based specialist assessments in Switzerland and Austria at present, there is still a long way to go in Germany. Following the recommendations presented in this article, a contribution could be made to improving the specialist assessments in the DACH region according to the specialist assessments objectives.

Combination of different clinical reasoning tests in a national exam.

Introduction:Clinical reasoning as a critical and high level of clinical competency should be acquired during medical education, and medical educators should attempt to assess this ability in medical students. Nowadays, there are several ways to evaluate medical students’ clinical reasoning ability in different countries worldwide. There are some well-known clinical reasoning tests such as Key Feature (KF), Clinical Reasoning Problem (CRP), Script Concordance Test (SCT), and Comprehensive Integrative Puzzle (CIP). Each of these tests has its advantages and disadvantages. In this study, we evaluated the reliability of combination of clinical reasoning tests SCT, KF, CIP, and CRP in one national exam and the correlation between the subtest scores of these tests together with the total score of the exam. Methods:In this cross sectional study, a total number of 339 high ranked medical students from 60 medical schools in Iran participated in a national exam named “Medical Olympiad”. The ninth Medical Olympiad was held in Shahid Beheshti University of Medical Sciences, Tehran, Iran, under the direct supervision of the Ministry of Health and Medical Education in summer 2017. The expert group designed a combination of four types of clinical reasoning tests to assess both analytical and non-analytical clinical reasoning. Mean scores of SCT, CRP, KF, and CIP were measured using descriptive statistics. Reliability was calculated for each test and the combination of tests using Cronbach's alpha. Spearman's correlation coefficient was used to evaluate the correlation between the score of each subtest and the total score. SPSS version 21 was used for data analysis and the level of significance was considered <0.05. Results: The reliability of the combination of tests was 0.815. The reliability of KF was 0.81 and 0.76, 0.80, and 0.92 for SCT, CRP, and CIP, respectively. The mean total score was 169.921±41.54 from 240. All correlations between each clinical reasoning test and total score were significant (P<0.001). The highest correlation (0.887) was seen between CIP score and total score. Conclusion:The study showed that combining different clinical reasoning tests can be a reliable way of measuring this ability.

Construct validity of script concordance testing: progression of scores from novices to experienced clinicians.

ObjectivesTo investigate the construct validity of Script Concordance Testing (SCT) scores as a measure of the clinical reasoning ability of medical students and practising General Practitioners with different levels of clinical experience.MethodsPart I involved a cross-sectional study, where 105 medical students, 19 junior registrars and 13 experienced General Practitioners completed the same set of SCT questions, and their mean scores were compared using one-way ANOVA. In Part II, pooled and matched SCT scores for 5 cohorts of students (2012 to 2017) in Year 3 (N=584) and Year 4 (N=598) were retrospectively analysed for evidence of significant progression. ResultsA significant main effect of clinical experience was observed [F(2, 136)=6.215, p=0.003]. The mean SCT score for General Practitioners (M=70.39, SD=4.41, N=13) was significantly higher (p=0.011) than that of students (M = 64.90, SD = 6.30, N=105). Year 4 students (M=68.90, SD= 7.79, N=584) scored a significantly higher mean score [t(552)=12.78, p<0.001] than Year 3 students (M = 64.03, SD=7.98, N=598). ConclusionsThe findings that candidate scores increased with increasing level of clinical experience add to current evidence in the international literature in support of the construct validity of Script Concordance Testing. Prospective longitudinal studies with larger sample sizes are recommended to further test and build confidence in the construct validity of SCT scores.

Evaluation of the use of script concordance test in a multicampus psychiatric pharmacy elective course.

IntroductionEvaluating a student's ability to accept complexity, uncertainty, and ambiguity as part of clinical practice is difficult in a classroom setting using written tests. This study was conducted to explore the feasibility and validation of using a script concordance test (SCT) to evaluate pharmacy student knowledge and clinical competence in a psychiatry elective course.MethodsThis study involved prospective validation of psychiatry-focused SCT questions using a panel of practicing psychiatric pharmacists and retrospective review of student performance on the same SCT questions. The reliability of the SCT was also evaluated using Cronbach alpha coefficient.ResultsA total of 13 practicing psychiatric pharmacists participated in the validation phase of the study of 75 questions. Pharmacy student scores (n = 17) averaged 39.79 (±5.02) points, and psychiatric pharmacist scores averaged 50.11 (±4.51) points, representing mean percentages of 61.2% and 77.1%, respectively, on the adjusted exam. The Cronbach alpha was 0.94.DiscussionThe development of a valid and reliable SCT to test student psychiatric pharmacy knowledge and clinical competence after taking a psychiatry elective course was feasible.

Script concordance test examinations: Student perception and study approaches

Introduction: Script concordance test questions (SCTs) are a style of question that has been developed based upon the script theory of information storage and retrieval. SCTs use script theory to examine a clinical reasoning construct purported to be different from that assessed by multiple-choice questions (MCQs). Evidence regarding the construct validity of SCT is somewhat limited, but generally supportive. Methods: This project involved a content-matched MCQ and SCT practice examination delivered to senior medical students (n = 211) from a single institution with an accompanying survey regarding educational consequences of SCT and MCQ examinations. Results: Students’ responses differed significantly between how they would prepare for an MCQ examination as compared to an SCT examination. These differences were present in a number of areas, including greater focus on textbooks and use of websites for MCQ examination preparation (p < 0.001). Students reported that for an SCT examination they would benefit most from in-person teaching from local consultants involved in the process of SCT generation. Students felt that they would also benefit from further instruction regarding key elements of SCT technique, and from opportunities to provide written explanations of their reasoning during the exam. Conclusions: The results suggest that SCTs have educational consequences different from MCQs. Further research into the educational consequences of SCTs, and how these consequences may differ from that of other methods of assessment, is warranted.

2019 Canadian Surgery Forum Abstracts.

# 01. The future of general surgery training: a Canadian resident nationwide Delphi consensus statement {#article-title-2} Various pedagogical models have been introduced in an attempt to improve and restructure surgical training, but significant obstacles remain. Before implementing national