High-stakes Testing Research Articles

Validity Evidence for Procedure-specific Competence Assessment Tools in Orthopaedic Surgery: A Scoping Review.

Competency-based training requires frequent assessment of residents' skills to determine clinical competence. This study reviews existing literature on procedure-specific competence assessment tools in orthopaedic surgery. A systematic search of eight databases up to May 2023 was conducted. Two reviewers independently assessed validity evidence and educational utility of each assessment tool and evaluated studies' methodological quality. Database searching identified 2,556 unique studies for title and abstract screening. Full texts of 290 studies were reviewed; 17 studies met the inclusion criteria. Bibliography review identified another five studies, totaling 22 studies examining 24 assessment tools included in the analysis. These tools assessed various orthopaedic surgery procedures within trauma, sports medicine, spine, and upper extremity. Overall validity evidence was low across all studies, and was lowest for consequences and highest for content. Methodological quality of studies was moderate. Educational utility assessment was not explicitly done for most tools. The paucity of current procedure-specific assessment tools in orthopaedic surgery lacks the validity evidence required to be used reliably in high-stake summative assessments. Study strengths include robust methodology and use of an evidence-based validity evidence framework. Poor-quality existing evidence is a limitation and highlights the need for evidence-based tools across more subspecialties.

Innocuous Exam Features? The Impact of Answer Placement on High-Stakes Test Performance and College Admissions

Innocuous Exam Features? The Impact of Answer Placement on High-Stakes Test Performance and College Admissions

The use and impact of pre-task planning time in the monologic task of LanguageCert speaking tests

Extensive oral tasks or monologues of different types (e.g., presentations, storytelling) are often used as second language acquisition tasks in the fields of language learning and language testing. Pre-task planning time is a common provision to test-takers who may use different strategies to prepare their response. High-stakes tests, such as the LanguageCert IESOL suite of tests, include planning time prior to monologic tasks and offer test-takers the opportunity for note-making. While the language assessment literature supports planning time for reasons of face validity and fairness, research studies do not consistently support correlations between planning and performance. The current study examined the differences between the scores of test-takers who used note-making as a strategy and those who did not. The research questions investigated: (i) whether test-takers who make notes during planning time in the monologue task of an L2-English B2 speaking task are awarded higher scores on their spoken performances than test-takers who do not and (ii) test-takers’ perceptions of their use of planning time. The findings suggest that making notes did not improve test-takers’ performance against any of the rating criteria used in the assessment. It also revealed that most test-takers use their planning time to generate their main propositions.

Comparing Estimated and Real Item Difficulty Using Multi-Facet Rasch Analysis

This study aimed to compare estimated item difficulty based on expert opinion with real item difficulty based on data. For security reasons, some high-stakes tests are not pre-tested and item difficulty is estimated by teachers in classroom assessments, so it is necessary to examine the extent to which experts make accurate predictions. In this study, we developed a 12-item assessment test like the Turkish teacher certification exam. Item difficulty was estimated and compared separately based on 1165 student responses and the opinions of 12 experts. The study revealed that the experts had a good ability to estimate item difficulty for items of moderate difficulty. However, they tended to underestimate item difficulty for items categorized as medium-easy.

Unveiling the Tapestry of Learning: Exploring the Pedagogical Landscape for Transversal and Academic Competencies through the Lived Experiences of School Teachers and Students

This research investigates the pedagogical approaches employed for the development of transversal and academic competencies based on the lived experiences of students and teachers. Despite the presence of qualified educators in educational activities, schools face numerous challenges in fully realizing the potential of students. In the context of developing nations like Nepal, schools and teachers encounter unpreparedness in facilitating activities geared towards enhancing students' academic and transversal competencies. Despite teachers' expressed commitment in workshops to adopt student-centered methods, conventional lecture-based approaches persist due to the impact of high-stakes testing. The absence of clear guidelines and motivation further contributes to the prioritization of traditional testing over competency-based education. Furthermore, inadequate facilities and an unsupportive environment hinder the implementation of practice-based education in schools. Recognizing that students excel when engaged in enjoyable activities, it is imperative to align educational activities with learners' needs and interests while progressing towards predefined objectives. This study advocates for a blended management model, incorporating both top-down and bottom-up approaches, to encourage teachers in adopting student-centered methods, thereby playing a pivotal role in nurturing both academic and transversal competencies. Given that both teachers and students tend to prioritize aspects that align with assessment criteria, evaluating assessments covering both competencies become crucial for a comprehensive understanding of their educational progress.

‘Cracking the teacher’s code’ – an ethnographic study of students preparing for oral exams

ABSTRACT This paper explores students’ preparatory processes for high-stakes exams using Danish oral exams as an example. To graduate, students must convince two teacher-examiners as the state’s representatives that they deserve to pass. Average grades determine students’ admission into tertiary education. Fieldwork data following students transitioning from teaching to exams show students ‘cracking teachers’ codes’ instead of delivering the curriculum requirement of independent critical interpretations of text material. norms of subjects called ‘super-addressees’ (Bakhtin sss1986 [“The Problem of Speech Genres.” In Speech Genres and Other Late Essays, edited by Carol Emerson and Michael Holquist, 60–102. Austin: University of Texas Press]) are not transparent to students despite an almost symbiotic interrelationship between teaching and exams. Students identify ‘Fitting responses’ (Bitzer 1968 [“The Rhetorical Situation.” Philosophy & Rhetoric 1 (1): 1–14]) for exams by navigations named 1: Disciplinary seduction in a transparent exam system, 2: Physically observing the teacher’s reactions in an opaque exam system, or 3: Jotting down an encyclopaedia of teacher’s keywords. Researching students anticipating exams illuminates their notions of what it takes to succeed in education.

Investigation of the factors affecting the curriculum fidelity of teachers working at different types of schools


 
 
 
 
 The purpose of this study is to examine the factors affecting the curriculum fidelity of teachers working in different types of school. In the study, case study design from qualitative research designs was used. The participants of the study werre teachers working in primary, secondary and high school types in publice school affiliated to the Ministry of National Education in rural district in a province located in the Northers Anatolia region of Türkiye in the fall semester of the 2022-2023 academic year. The participants was determined by convenient sampling method. The data of the study were obtained through a semi-structured interview form. The research data were analyzed using the thematic analysis method in accordance with qualitative research and findings were determined. While determining the findings, themes and categories were obtained and direct quotations were included to reflect the teachers' opinions. As a result of the research; It was determined that they expressed the opinion that curriculum characteristics, institutional characteristics, student characteristics, geographical conditions, family structure, social development of students, centralized education system, high-stakes tests, and pre-service and in-service training that teachers receive affect teachers' curriculum fidelity.
 
 
 

Students’ Understandings Related to Their Upcoming High-Stakes Tests: A Phenomenographic Study

The aim of the present study is to uncover the understanding of the students who struggle with high-stakes tests and to investigate the variations in their perceptions, taking into account the dimensions of variation. Phenomenography was chosen as the research method for this purpose. The study involved 18 students who prepared for three different high-stakes tests (LGS, YKS, and KPSS) during the 2021-2022 school year. Data were collected through semi-structured interviews. The researchers examined what students experienced while preparing for their high-stakes tests, how they felt, and they understood their tests. After phenomenographic analysis, the researchers identified six different ways of understanding related high-stakes tests. These are, parting of the ways which determine the direction of their lives; frightening; a time of strains; at an inconvenient time; an experience that improves me; an opportunity to succeed through striving. It is clear that age, experience, and number of retakes of the test are critical in forming these understandings.

Effects of errors on ratings of writing performances – Evidence from a high-stakes exam

The study reported in the paper starts with a hypothesis that errors observable in writing performances can account for much of the variability of the ratings awarded to them. The assertion is that this may be the case even when prescribed rating criteria explicitly direct rater focus towards successfully performed aspects of a writing performance rather than towards errors. The hypothesis is tested on a sample of texts rated independently of the study, using a five-point analytic rating scale involving ‘Can do’-like descriptors. The correlation between errors and ratings is ascertained using ordinal logistic regression, with Pseudo R2 of 0.51 discerned overall. Thus, with roughly 50% of score variability explainable by error occurrences, the stated hypothesis is considered confirmed. The study goes on to discuss the consequences of the findings and their potential employ in assessment of writing beyond the local assessment context.

English High-stakes Testing in Indonesia: Friend or Foe?

The Education Ministry has eliminated standardised high-stakes testing called Ujian Nasional (UN) since 2021 throughout schools all around Indonesia. However, it might be practical to look backwards at the UN and its implementation with the impact it gives on the teachers. This article has a concern about the washback consequences of a high-stakes exam on teachers. The effects were investigated within the specific context of the 2016 UN for students in Indonesia at the junior secondary level. A semi-structured interview with eight English teachers was set up to explore the teachers' perceptions towards the UN. Regarding feelings, all participants shared divergent feelings ranging from disinterested to excessive levels of fear or anxiety and mixed feelings. The data also showed mixed opinions, both positive and negative, regarding the examination. The students were thought to be motivated to study because the examination was used for selection purposes. The results did, however, also imply that the students' performance on the national exam was thought to have been assessed unfairly. The instructors demonstrated how the exam affected their teaching roles and methods. Making the national exam a low-stakes test was one of the suggestions made by attendees to enhance it.

Bridging Validity Frameworks in Assessment: Beyond Traditional Approaches in Health Professions Education

Construct: High-stakes assessments measure several constructs, such as knowledge, competencies, and skills. In this case, validity evidence for test scores’ uses and interpretations is of utmost importance, because of the consequences for everyone involved in their development and implementation. Background: Educational assessment requires an appropriate understanding and use of validity frameworks; however, health professions educators still struggle with the conceptual challenges of validity, and frequently validity analyses have a narrow focus. Important obstacles are the plurality of validity frameworks and the difficulty of grounding these abstract concepts in practice. Approach: We reviewed the validity frameworks literature to identify the main elements of frequently used models (Messick and Kane’s) and proposed linking frameworks including Russell’s recent overarching proposal. Examples are provided with commonly used assessment instruments in health professions education. Findings: Several elements in these frameworks can be integrated into a common approach, matching and aligning Messick’s sources of validity with Kane’s four inference types. Conclusions: This proposal to contribute evidence for assessment inferences may provide guidance to understanding the use of validity evidence in applied settings. The evolving field of validity research provides opportunities for its integration and practical use in health professions education.

Predicting the difficulty of EFL reading comprehension tests based on linguistic indices

Estimating the difficulty of reading tests is critical in second language education and assessment. This study was aimed at examining various text features that might influence the difficulty level of a high-stakes reading comprehension test and predict test takers’ scores. To this end, the responses provided by 17,900 test takers on the reading comprehension subsection of a major high-stakes test, the Iranian National University Entrance Exam for the Master’s Program were examined. Overall, 63 reading passages in different versions of the test from 2017 to 2019 were studied with a focus on 16 indices that might help explain the reading difficulty and test takers’ scores. The results showed that the content word overlap index and the Flesch-Kincaid Reading Ease formula had significant correlations with the observed difficulty and could therefore be considered better predictors of test difficulty compared to other variables. The findings suggest the use of various indices to estimate the reading difficulty before administering tests to ensure the equivalency and validity of tests.

Curriculum Standpoint for Social Justice: Understanding the Politics of School Knowledge

This conceptual paper argues for the need for us to understand the relationship between curriculum and social justice, largely through the lens of what the author calls “curricular standpoint.” This framing helps define the connective tissue of how social, cultural, institutional, economic, and political context manifest within the struggle over knowledge, including which knowledge gets framed as commonsense and hegemonic through things like textbooks, state standards, and high-stakes exams – thereby also implicated student consciousness about the world. Drawing on Feminist Standpoint Theory, this paper ultimately argues that, if we can fully recognize the extent to which social relations manifest in school knowledge, we also create the conditions for a critical examination of that knowledge as well as the possibilities for the development of increased critical consciousness about the curriculum and society – thus detailing the relationship between curriculum and social justice.

Objective Structured Clinical Examinations (OSCEs) for High-Stakes Assessment of Certified Registered Nurse Anesthesiologists (CRNAs)

BackgroundCertified registered nurse anesthesiologists (CRNAs) undergoing high stakes assessment must demonstrate competency. Written competency is the current method of high-stakes assessment, and this paper uses Miller's Pyramid of competency to explore use of the objective structured clinical examination (OSCE) for high-stakes assessment. MethodsThe modified Delphi method was used with 10 experts to design and execute an anaphylaxis OSCE simulation scenario of 3 proficiency levels. OSCE performance assessment tools included a checklist and global rating scale (O-GRS). A system usability survey was used to assess OSCE utility. ResultsThe overall system usability survey score was below our goal of 75. Assessment tool results did not demonstrate high correlation among panelists, although mean scores for each performance showed statistically significant differences in scores (p < .001). ConclusionOur paper showed the process of OSCE design, execution, and utility. Although results did not allow us to strongly recommend use of OSCEs for high-stakes assessment of CRNAs, we are encouraged that our results will provide direction for future papers related to this topic.

‘You are the hero’: teaching great literature in a London Jesuit school

This article argues that Ignatian pedagogy in the Jesuit tradition allows great literature to become transformative. It takes as a case study an extracurricular course – ‘Becoming a Hero: From Homer to Harry Potter’ – taught to a socially diverse group of key stage three ‘gifted and talented’ children in a London Jesuit comprehensive school. While secondary teaching is normally geared towards high-stakes summative assessment, this course aims instead at personal growth. Young people discover themselves as the heroes in a perennial plotline which appears throughout both classic texts and young adult fiction. The teacher (himself a Jesuit) had previously taught the course to undergraduates at Harvard University. Adapting it for younger children, he makes the case for ‘gifted and talented’ provision and narrates his use of assessment for prior learning and creative teaching techniques in order to ensure holistic – spiritual, intellectual, and emotional – learning beyond the curriculum.

Unveiling Teachers’ Perception: Cultural Influences on Educators' Decision-Making in High-Stakes Exams

This study analyses the diverse motives of educators entering the teaching profession and their impact on pedagogical decisions during high-stakes exams. Semi-structured interviews with eight participants (six male and two female teachers) from secondary schools in Kabupaten Bandung, Jawa Barat, Indonesia, lasted up to forty minutes. These interviews aimed to elicit insights into the participants' beliefs on their professional calling.Three key themes emerged from the interviews' thematic analysis: the trajectory of a teacher's personal history, the influential role of degree choice, and the weight of accountability. Furthermore, the findings highlighted how gender identity not only influenced the teachers but also echoed across their familial spheres, guiding them towards a profession in education.This investigation underlined the critical significance of cultural beliefs in developing educators' motivations, which serve as the foundation of their professional identity. As a result, this identification has a substantial impact on their pedagogical practises in the classroom, particularly when confronted with the demands imposed by high-stakes exams. This study sheds insight into the intricacies underpinning educators' decision-making processes, particularly in circumstances typified by rigorous assessments, by providing a comprehensive view of the delicate interplay between cultural factors, individual motives, and professional practice.

Digital Education, Mental Resilience and Emotional Well-being of the Students of HEIs: Addressing Mental Health in the Light of NEP 2020

The present paper probes the interplay between digital education, students’ mental resilience, and their emotional well-being within the context of Higher Education Institutions (HEIs). In particular, the paper places a focal point on addressing mental health issues among students and assesses the impact of the National Education Policy (NEP) of 2020 in creating an educational atmosphere and environment facilitative to the cognitive growth and well-being of students. It is evident that the widespread circulation of technology among students of HEIs has detrimental effects on their mental resilience and emotional well-being. Excessive digital screen time leads to deteriorated mental health, including increased anxiety, depression, compulsive behaviours etc., intensified by nomophobia (no mobile phobia). Again, information overload in the digital age hampers decision-making ability and deepens anxiety among students. The deleterious effects of cyberbullying and phubbing on social skills, well-being, interaction and relationships capture the essence for educators, researchers and policymakers to address these concerns without much delay. Nevertheless, NEP 2020 already prioritizes both academic excellence and the mental health of students of HEIs through facilitating counselling services, a flexible curriculum, diverse extracurricular activities etc. The policy aims to reduce anxiety and stress by shifting away focus from rote memorization, high-stakes assessments and a rigid academic atmosphere while cultivating a vibrant campus life. It also recognizes the unique needs of vulnerable student populations, emphasizing targeted support for their academic success and mental well-being. While NEP 2020 acknowledges these concerns, it may fall short of fully addressing these complex mental health challenges, necessitating a more adaptable, holistic approach.

The role of teacher support in students' academic performance in low- and high-stakes assessments

Teachers' impact on learning outcomes is well supported, but more robust evidence is needed on the complex indirect pathways through which teacher support impacts performance in low-versus high-stakes examinations. This study sought to understand the divergent mechanisms through which teachers' support for students affects their performance on low- and high-stakes assessments. Bayesian structural equation modeling with the Markov Chain Monte Carlo algorithm and data from Ghana were used to test mediated, unmediated, and moderated-mediated relationships. The results show that a typical student's high-stakes exam performance increases by approximately 13 % for each one-unit increase in teacher support. A chain mediation effect also exists through student homework behavior and low-stakes everyday performance. Student gender is also a moderator. These results could aid in identifying the malleable leverage points associated with low- and high-stakes assessments. Such empirical clarity would help education administrators to develop appropriate professional development programs that enhance teachers' support roles, enabling them to respond better to learning disparities and related challenges.

Modification of Fundamentals of Laparoscopic Surgery Manual Tasks Simulation Curriculum With the Addition of the Vaginal Cuff Closure Training

Fundamentals of Laparoscopic Surgery (FLS) is a multiple-choice test and a manual skills exam using simulation that Obstetrics and Gynecology (OBGYN) residents must pass to qualify for board certification. There is insufficient validity evidence supporting the use of FLS as a high-stakes exam. This study examines the correlation between OBGYN residents' performance on the FLS manual tasks and simulated vaginal cuff closure. We compared residents' performance on FLS tasks with simulated vaginal cuff suturing on a model. During the first coached simulation session, after completion of training on the standard 5 FLS tasks, residents were coached on vaginal cuff closure using a simulated model placed inside the standard FLS box trainer. At a subsequent session, their performance was scored using the Global Operative Assessment of Laparoscopic Skills Scale (GOALS) and a second task-specific metric, and these scores were compared to their official FLS score. Twenty-nine residents completed the vaginal cuff simulation training between June 2019 and November 2021. Nineteen of the 29 were able to complete the cuff closure with the mean time to completion being 14.5 minutes. We found no correlation between official manual skills FLS scores and vaginal cuff GOALS scores (rho = -0.02, p = 0.90) or cuff closure assessment tool score (rho = -0.015, p = 0.048). There was also no correlation between time to completion for any FLS task and vaginal cuff closure OSAT scores. All residents reported that they found the cuff to be a useful addition to the FLS curriculum. Our study demonstrated that trainee performance on a simulated vaginal cuff closure model did not correlate with official FLS manual tasks skills. This finding adds to the body of evidence disputing the use of FLS as a high-stakes exam to assess laparoscopic skills in gynecology in the relationship with other variables category.

EFL teachers’ cognition of social and psychological consequences of high-stake national language tests: role of teacher training workshops

Consequential validity, a facet of construct validity, has been extensively explored by educational psychologists and scholars focused on English language assessment. It is widely recognized that English language instructors must possess a thorough understanding of high-stake language tests. However, the body of research concerning EFL teachers’ perceptions of high-stake tests is quite limited. This study aims to delve into the cognitions of Iranian EFL teachers regarding the social and psychological ramifications of high-stake English language tests. Additionally, the study investigates the influence of a teacher training workshop on EFL teachers’ perceptions of test consequences. The research employs a two-phase quantitative research design. A total of 210 EFL teachers participated in the survey, completing a scale comprising 25 items that delineated their perceptions of test consequences. Furthermore, thirty teachers engaged in a two-session workshop focused on test consequences. Data analysis encompassed both one-sample and paired-sample t tests.The results reveal that EFL teachers possess knowledge regarding certain social and psychological consequences of language tests; however, their awareness of some adverse consequences remains limited. Notably, the outcomes indicate that teacher training workshops have a positive impact on teachers’ cognitions of both positive and negative test consequences. The implications of these findings extend to teacher trainers and English language educators, offering practical insights to enhance the effectiveness of their teaching practices.