High-stakes Testing Research Articles

Global assessment of surgical skills (GASS): validation of a new instrument to measure global technical safety in surgical procedures.

Broad implementation of the American Board of Surgery's entrustable professional activities initiative will require assessment instruments that are reliable and easy to use. Existing assessment instruments of general laparoscopic surgical skills have limited reliability, efficiency, and validity across the spectrum of formative (low-stakes) and summative (high-stakes) assessments. A novel six-item global assessment of surgical skills (GASS) instrument was developed and evaluated with a focus upon safe versus unsafe surgical practice scoring rubric. The GASS was developed by iterative engagement with expert laparoscopic surgeons and includes six items (economy of motion, tissue handling, appreciating operative anatomy, bimanual dexterity, achievement of hemostasis, overall performance) with a uniform three-point scoring rubric ("poor-unsafe", "adequate-safe", "good-safe"). To test inter-rater reliability, a cross-sectional study of four bariatric surgeons with experience ranging from 4 to 28years applied the GASS and the global operative assessment of laparoscopic skills (GOALS) to 30 consecutive Roux-en-Y gastric bypass procedure operative videos. Inter-rater reliability was assessed for a simplified dichotomous "safe" versus "unsafe" scoring rubric using Gwet's AC2. The GASS inter-rater reliability was very high across all six domains (0.88-1.00). The GASS performed comparably to the GOALS inter-rater reliability scores (0.96-1.00). The economy of motion and bimanual dexterity items had the highest percentage of unsafe ratings (9.2% and 5.8%, respectively). The GASS, a novel six-item instrument of general laparoscopic surgical skills, was designed with a simple scoring rubric (poor-safe, adequate-safe, good-safe) to minimize rater burden and focus feedback to trainees and promotion evaluations on safe surgical performance. Initial evaluation of the GASS is promising, demonstrating high inter-rater reliability. Future research will seek to assess the GASS against a broader spectrum of laparoscopic procedures.

Exploring Whole-Person Education Reform From Principals Perspective on High-Stakes Testing

Whole-person education is undeniable and indispensable for student benevolence and lifelong building. However, the high test score is the central issue in Chinese education. The purpose of this descriptive qualitative inquiry is to explore the high-test score issue by analyzing principals’ experiences in advocating the whole-person education in leading the school reform in China. A total of 12 participants of principals ( N = 12) were employed in this study. There are six female ( N = 6) and six male ( N = 6) principals from the six private ( N = 6) and public ( N = 6) schools in Beijing through semi-structural and open-ended interview protocols. This study found that principals face three major challenges, including (1) high test score causes distrust, unfairness, and separation between school stakeholders, (2) high test score causes academic pressure, and (3) high test score limits student competency and educators’ academic freedom. This study found that principals use three major strategies to form a school reform and student development, including (1) communication for whole-person education, (2) correlation between the whole-person education and student individual career plan, and (3) pathway from high school to college through cooperation project. Finally, the proposed study of using a basic qualitative research inquiry through semi-structural and open-ended interviewing with a total of 60 participants, including principals ( N = 12), teachers ( N = 12), parents ( N = 12), students ( N = 12), and superintendents ( N = 12), would be employed to investigate what challenges the schools face, how the schools should be, and what strategies should implement for lifelong student learning, suggested.

High-Stakes Online Exams

COVID-19 has affected university assessment procedures on a large scale. This empirical study aims to understand the types of high-stakes exams delivered online at the Lucerne School of Business in Switzerland during the “Corona Semesters” of 2020 and 2021 and the decision-making factors that influenced their implementation. To do so, the authors conducted semi-structured interviews with eight faculty members across a variety of disciplines. Requirements from the exam workflow (preparation, proctoring, grading) were identified and analyzed by course type. Four factors emerged that significantly impacted design and delivery for high-stakes exams online: 1) Digital exam formats significantly impact the nature of exams for procedural subjects such as mathematics; 2) “Group Exams” are not the answer to preventing student collusion on online exams; 3) interrater reliability and low answer variance are considered a central factor for exam quality assurance; 4) second-order effects such as stable wifi and device compatibility will continue to hinder widescale adoption of digital exams. The findings suggest that online exam delivery significantly affects institutional exam practice beyond mere consideration of learning outcomes. The authors conclude by speculating that similar dynamics may have impacted other business schools during their Corona semesters and invite future research on whether the findings from this article can spark discussion and reflection for policy makers in other institutions on post-pandemic legacies.

New ACGME Clinician Educator Milestones as a Roadmap for Faculty Development: a Position Paper from the Society of General Internal Medicine Education Committee.

Traditionally, clinician educators are tasked with the responsibility of training future physician workforce. However, there is limited identification of skills required to fulfill this responsibility and a lack of consensus on effective faculty development for career growth as a clinician educator. The newly released Accreditation Council of Graduate Medical Education (ACGME) Clinician Educator (CE) Milestones framework outlines important skills for clinician educators and provides the opportunity to create robust faculty development. In this paper, members of the Society of General Internal Medicine Education Committee discuss the importance of these CE Milestones, outline the novel themes highlighted in the project, and provide recommendations for proper application on both the individual and institutional levels to optimize faculty development. The paper discusses strategies for how to apply the CE Milestones as a tool to create a culture of professional growth and self-directed learning. Using a reflective approach, CE faculty and mentors can identify areas of proficiency and opportunities for growth, thereby creating individualized professional development plans for career success. Institutions should use aggregate CE Milestones data as a needs assessment of their faculty "population" to create targeted faculty development. Most importantly, institutions should not use CE Milestones for high-stakes assessments but rather encourage reflection by CE faculty and create subsequent robust faculty development programs. The ACGME CE Milestones present an exciting opportunity and lay an important foundation for future CE faculty development.

The Effects of Virtual Reality on Stress in Graduate Students: A Pilot Study

Abstract Date Presented 04/21/2023 This study found that virtual reality significantly reduced stress levels after a high-stakes exam or lab practical. This is meaningful because of the prevalence of stress, anxiety, and other mental health conditions among current graduate students. Primary Author and Speaker: Scott Truskowski Additional Authors and Speakers: Jordan Lanning, Madeline McIntyre, Nicole Novak Contributing Authors: Kristin Everhart

Early prediction of medical students' performance in high-stakes examinations using machine learning approaches

IntroductionSince the advent of medical education systems, managing high-stakes exams has been a top priority and challenge for all policymakers. However, considering machine learning (ML) techniques as a replacement for medical licensing examinations, particularly during crises such as the COVID-19 outbreak, could be an effective solution. This study uses ML models to develop a framework for predicting medical students' performance on high-stakes exams, such as the Comprehensive Medical Basic Sciences Examination (CMBSE). Material and methodsPrediction of students' status and score on high-stakes examinations faces several challenges, including an imbalanced number of failing and passing students, a large number of heterogeneous and complex features, and the need to identify at-risk and top-performing students. In this study, two major categories of ML approaches are compared: first, classic models (logistic regression (LR), support vector machine (SVM), and k-nearest neighbors (KNN)), and second, ensemble models (voting, bagging (BG), random forests (RF), adaptive boosting (ADA), extreme gradient boosting (XGB), and stacking). ResultsTo evaluate the models' discrimination ability, they are assessed using a real dataset containing information on medical students over a five-year period (n = 1005). The findings indicate that ensemble ML models demonstrate optimal performance in predicting CMBSE status (RF and stacking). Similarly, among the classic regressors, LR exhibited the highest root-mean-square deviation (RMSD) (0.134) and coefficient of determination (R2) (0.62), whereas the RF model had the highest RMSD (0.077) and R2 (0.80) overall. Furthermore, Anatomical Sciences, Biochemistry, Parasitology, and Entomology grade point average (GPA) and grades demonstrated the strongest positive correlation with the outcomes. ConclusionComparing classic and ensemble ML models revealed that ensemble models are superior to classic models. Therefore, the presented framework could be considered a suitable alternative for the CMBSE and other comparable medical licensing examinations.

Use of lexical features in high-stakes tests: Evidence from the perspectives of complexity, accuracy and fluency

Use of lexical features in high-stakes tests: Evidence from the perspectives of complexity, accuracy and fluency

Assessing operative competence in core surgical training: A reliability analysis

BackgroundThis study quantifies the number of observations required to reliably assess the operative competence of Core Surgical Trainees (CSTs) in Ireland, using the Supervised Structured Assessment of Operative Performance (SSAOP) tool. MethodsSSAOPs (April 2016–February 2021) were analysed across a mix of undifferentiated procedures, as well as for three commonly performed general surgery procedures in CST: appendicectomy, abdominal wall hernia repair, and skin/subcutaneous lesion excision. Generalizability and Decision studies determined the number of observations required to achieve dependability indices ≥0.8, appropriate for use in high-stakes assessment. ResultsA total of 2,294 SSAOPs were analysed. Four assessors, each observing 10 cases, can generate scores sufficiently reliable for use in high-stakes assessments. Focusing on a selection of core procedures yields more favourable reliability indices. ConclusionTrainers should conduct repeated assessments across a smaller number of procedures to improve reliability. Programs should increase the assessor mix to yield sufficient dependability indices for high-stakes assessment.

Teaching or testing, which matters more? Transition among education levels in Turkey

This study analyses the alignment between educational policy of Turkey and high stakes tests administered for transitioning from secondary to high schools. Research questions focus on the opinions of secondary school teachers about the alignment between the transit exam questions and curricula, course books, and materials as well as their opinions on high stakes testing. Research was conducted based on a mixed methods study model utilizing the triangulation design. A total of 109 teachers from six different majors working in Ankara participated in the study. An online survey consisting of eight questions was used to get teachers’ opinions. The first three questions were analyzed by chi-square tests and percentages, the fourth by arithmetic means and the last four questions by qualitative content analysis method. Results showed that in the current Turkish learning-teaching process, education is serving dominantly for tests emphasizing an evaluation oriented education system which is in contrast with education policy documents targeting the year 2023.

Using SFL-based text analysis to inform instruction for emergent bilingual learners

Emergent Bilingual Learners (EBs) need language-focused instructional support if they are to access content, make meaning, and engage critically with academic texts in English. Text analysis informed by systemic functional linguistics (SFL) helps teachers identify “the language of schooling” (Schleppegrell, 2004) which they will need to make visible to their students. The authors, both teacher educators in an ESOL endorsement program, recognize that both the teacher candidates who are analyzing texts and the students whose instruction will be informed by the analysis engage in literacy practices that may not be recognized and valued by all teachers or aligned with the language demands of mandatory, high-stakes assessments. With that in mind, we envision a text analysis assignment that promotes language as meaning based, rather than rule based, and additive, rather than subtractive (Garcia, 2009) in terms of the language resources of teachers and students in language learning settings. Additionally, we look for ways to support our teacher candidates in interrogating texts critically as part of their analysis to uncover dominant perspectives. This practice-based article models the process of SFL-based text analysis facilitated by a text analysis tool we designed. The goal is that teachers can apply what they have learned from their analysis to create language-focused instruction that both supports academic language learning and promotes critical stances towards the connections between language choices and meaning making in specific academic contexts/disciplines. Additionally, we encourage teachers to use a variation of this text analysis format with students to explore how language is used in the text to develop all languages in the students' linguistic repertoire.

Mapping global research on shadow education: Trends and future agenda

This study aimed to analyze bibliographies of journals, authors, and research topics on shadow education using the Scopus database. Bibliometric analysis focuses on the metadata of journals, authors, and topics, visualized, and analyzed to produce a road map, research trends, and future agenda. The data were obtained from 207 articles published on Scopus downloaded on 29/8/2021 by using “shadow education” or “shadow curriculum” keywords. Furthermore, descriptive statistical methods and bibliometric analysis using Biblioshiny, an R-based application that generates bibliometric maps were used. Shadow education research has not been widely developed. Therefore, this bibliographic study may form the basis for future developments. Shadow education is the highest trend, followed by education and policy, high stakes testing, teacher education, curriculum, academic achievement, and private tutoring. This study provides an overview of trends in journals, authors, and research topics related to shadow education. Specifically, it provides relevant information to develop the potential and related themes in the future.

Exploring aural vocabulary knowledge for TOEIC as a language exit requirement in higher education in Taiwan

Abstract The Test of English for International Communication (TOEIC) is a high-stakes test for students in higher education in Taiwan to fulfill the English-language graduation requirement. However, little is known regarding the vocabulary threshold for the test or the effects of the lexical coverage and profiles of test items on the adequate comprehension of the test. This study used a validated Listening Vocabulary Levels Test (LVLT) and the TOEIC listening subtest to estimate learners’ aural vocabulary knowledge required for an exit TOEIC listening score. The findings showed: (1) aural lexical knowledge accounted for more than half of the variance in comprehension performance; (2) a minimum level of 3,000 word families for a lexical coverage of 98 % considerably affected the comprehension of spoken texts; and (3) lexical profiles varied in the individual parts of the listening subtest, with a range of 3,000–5,000 word families required for achieving a lexical coverage of 98 %. The crucial role of lexical knowledge/coverage in comprehension performance on the exit test was discussed.

Language Assessment Instrument Analysis: Scoping and Investigating the Writing Assessment of College English Test Band 4

The writing evaluation portion of the College English Test band 4 (CET-4) in China is the subject of this study. The goal is to evaluate the language evaluation tool utilized in this test and pinpoint its advantages, disadvantages, and prospective growth areas. The study uses a scoping and investigative strategy to fully comprehend the CET-4 writing assessment's administration, scoring, and design. The study makes a contribution to the field of language assessment by offering insightful information on how writing abilities are assessed in high-stake test and by outlining prospective directions for future study and research in this area.

Defining language goals in EMI: vocabulary demand in a high-stakes assessment in Hong Kong

Despite the rapid growth of English Medium Instruction (EMI) programmes around the globe, the language goals in such programmes have not been clearly defined, making it difficult to evaluate their success. Accordingly, our paper reports on a study that attempted to identify the language goals, operationalised as the vocabulary expected of learners, by investigating the under-explored area of EMI assessment. We sampled a corpus of 16 sets of examination papers in two subject areas, namely, Biology and Geography, spanning eight years in Hong Kong. The examination papers, analysed for their vocabulary profiles, indicated that students were expected to learn beyond the K3 level (the most frequent 3,000 words in English) and master mid-frequency words (K4−K8) as well as academic words. Our findings suggest that the less frequent words were not often shared by the two subjects. Analysing academic words also enabled us to showcase how some words were common across the two subjects while others were more subject-specific. Pedagogical implications are discussed, highlighting the need for teachers and students to attend to target language goals in EMI, and to consider the subject-specific nature of vocabulary items in EMI.

Evaluating the ethics of machines assessing humans

The case focusses on Assessment and Qualification Alliance, one of England’s largest exam boards, and its evaluation of whether Artificial Intelligence (AI) could be deployed, in principle, for marking high-stakes assessments. At a time when generative AI, such as ChatGPT, has gained popularity, the case offers insights into the challenges and risks of algorithmic decision making and algorithmic fairness, such as accuracy and explainability. The case allows students to explore the role of ethics when developing an AI-based tool in an area that they all know very well: Most students will have had to sit high-stakes assessments in the past and are still likely to be assessed on an ongoing basis as a current student. Students will thus be able to relate to the case and have their own stake(s) and opinion(s) about whether they would want to be assessed by AI. The case is also more broadly applicable in raising general awareness of the challenges and potential risks involved when using AI for decision making and it encourages students to consider the wider consequences for all stakeholders that are triggered by the use of digital technology.

Literacy assessment in the early years of schooling in an era of neoliberalism

This paper explores the literature related to literacy assessment in the early years of schooling in an era of neoliberalism and reports on a key aspect of a study which focused on the literacy assessment practices of early years teachers and literacy leaders in Australian Catholic schools within the Melbourne archdiocese. Background: The study was undertaken in a period following the Melbourne Catholic Education system’s devolution of literacy assessment responsibility to schools after a long period of mandated literacy assessment (1998–2012) but also occurred within a neoliberal high-stakes assessment environment, characterised by heightened levels of teacher accountability. Research Objectives: The intention of the research was to explore the literacy assessment beliefs and practices of teachers in early years classrooms (F–2), to explore the impacts of assessment devolution and the implications in terms of the literacy assessment practices in the early years in Australian Catholic primary schools in the Melbourne Archdiocese. Methods: A predominantly qualitative case study approach consisting of two phases was used to explore literacy assessment in the early years of schooling. Phase 1 consisted of an online questionnaire comprising both open and closed questions, which was completed by 76 literacy leaders from 76 Catholic schools in the Melbourne archdiocese. The questionnaire was designed to elicit information on schools’ literacy assessment practices in the early years and if these had changed since a Catholic Education Melbourne (CEM) policy change in 2012 allowing schools greater autonomy over literacy assessment in the early years. Phase 2 involved semistructured interviews with 23 early years teachers and seven literacy leaders from eight diverse schools to investigate their literacy assessment practices in greater detail and identify how they responded to the opportunities that came with devolved decision-making.

Towards accountability-centred practices: governance in OSCEs subordinating patient and practitioner clinical experience

New public management ideals and standards have become increasingly adhered to in health professions education; this is particularly apparent in high-stakes assessment, as a gateway to practice. Using an Institutional Ethnographic approach, we looked at the work involved in running high-stakes Objective Structured Clinical Exams (OSCEs) throughout an academic year including use of observations, interviews and textual analysis. In our results, we describe three types of ‘work’—standardising work, defensibility work and accountability work–summarising these in the discussion as an Accountability Circuit, which shows the organising role of texts on people’s work processes. We show how this form of governance mandates a shift towards accountability-centred practices, away from practices which are person-centred; this lens on accountability-centring during high-stakes assessments invites critique of the often-unquestioned emphasis of new public management in health professions education.

Teacher and headteacher assessment, feedback, and continuing professional development: the Mexican case

ABSTRACT Teacher and headteacher assessments and professional development are deemed critical levers for improving quality education in an increasing number of countries and contexts. However, prioritising improvement is not always evident, given that sometimes those subjected to assessments are not effectively informed about their performance (e.g. from national teacher examinations) or how to improve their work via feedback. This paper explores the bridging role of feedback concerning teacher and headteacher assessments and continuing professional development in the Mexican context, where scarce research exists. This mixed-methods research contributes evaluation evidence of Mexico’s 2013–2018 education reform, including high-stakes staff assessments and individual feedback reports. Survey data from 122 primary school teachers and headteachers and thirteen interviews with teachers, headteachers and policymakers indicate the feedback report made it difficult to distinguish between poor and proficient performance and identify strengths and weaknesses. Challenges regarding suitable continuing professional development following assessment feedback are put forward.

Evaluating the Construct Validity of Competencies: A Retrospective Analysis.

A competency-based framework focuses on alignment between professional standards and assessment design. This alignment implies improved measurement validity, yet it has not been established that competence in one context predicts performance in another context. High-stakes competence assessments offer insights into the relationship between assessment design and competencies. The internationally educated nurses competency assessment program (IENCAP) was developed at Touchstone Institute in collaboration with the College of Nurses of Ontario (CNO) and includes a 12-station OSCE. Each station evaluated the same 10 competencies. We submitted competency scores to a multi-trait multi-method matrix analysis to evaluate the convergent and discriminant validity of competencies. All correlations were significant and positive; however, we did not find evidence of convergent or discriminant validity. Correlations were higher between different competencies evaluated within the same station (mean correlation = 0.60) compared to identical competencies evaluated across different stations (mean correlation = 0.19). The results do not provide evidence of construct validity for competencies. While competency-based approaches emphasize various generalized knowledge, skills, and attitudes, these findings indicate that the clinical context is a major determinant of performance. The context-dependent nature of competencies requires multiple assessments in varied contexts. Performance on a single competency cannot be determined in a single occasion. The online version contains supplementary material available at 10.1007/s40670-023-01794-z.

Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings

Foundation models are a novel type of artificial intelligence algorithms, in which models are pretrained at scale on unannotated data and fine-tuned for a myriad of downstream tasks, such as generating text. This study assessed the accuracy of ChatGPT, a large language model (LLM), in the ophthalmology question-answering space. Evaluation of diagnostic test or technology. ChatGPT is a publicly available LLM. We tested 2 versions of ChatGPT (January 9 "legacy" and ChatGPT Plus) on 2 popular multiple choice question banks commonly used to prepare for the high-stakes Ophthalmic Knowledge Assessment Program (OKAP) examination. We generated two 260-question simulated exams from the Basic and Clinical Science Course (BCSC) Self-Assessment Program and the OphthoQuestions online question bank. We carried out logistic regression to determine the effect of the examination section, cognitive level, and difficulty index on answer accuracy. We also performed a post hoc analysis using Tukey's test to decide if there were meaningful differences between the tested subspecialties. We reported the accuracy of ChatGPT for each examination section in percentage correct by comparing ChatGPT's outputs with the answer key provided by the question banks. We presented logistic regression results with a likelihood ratio (LR) chi-square. We considered differences between examination sections statistically significant at a P value of < 0.05. The legacy model achieved 55.8% accuracy on the BCSC set and 42.7% on the OphthoQuestions set. With ChatGPT Plus, accuracy increased to 59.4% ± 0.6% and 49.2% ± 1.0%, respectively. Accuracy improved with easier questions when controlling for the examination section and cognitive level. Logistic regression analysis of the legacy model showed that the examination section (LR, 27.57; P= 0.006) followed by question difficulty (LR, 24.05; P < 0.001) were most predictive of ChatGPT's answer accuracy. Although the legacy model performed best in general medicine and worst in neuro-ophthalmology (P < 0.001) and ocular pathology (P= 0.029), similar post hoc findings were not seen with ChatGPT Plus, suggesting more consistent results across examination sections. ChatGPT has encouraging performance on a simulated OKAP examination. Specializing LLMs through domain-specific pretraining may be necessary to improve their performance in ophthalmic subspecialties. Proprietary or commercial disclosure may be found after the references.