High-stakes Testing Research Articles

A Multidimensional Analysis of a High-Stakes English Listening Test: A Corpus-Based Approach

The Gaokao, also known as China’s national college entrance exam, is a high-stakes exam for nearly all Chinese students. English has been one of the three most important subjects for a long time, and listening plays an important role in the Gaokao English test. However, relatively little research has been conducted on local versions of Gaokao’s English listening tests. This study analyzed the linguistic features and corresponding functional dimensions of the three different text types in the Gaokao’s listening test, investigating whether the papers used in three major regions of China were differentiated in terms of the co-occurrence patterns of lexicogrammatical features and dimensions of the transcripts. A corpus consisting of 170 sets of test papers (134,913 words) covering 31 provinces and cities from 2000 to 2022 was analyzed using a multidimensional analysis wherein six exclusive dimensions were extracted. The results showed that there were meaningful differences across short conversations, long conversations, and monologues with regard to the six dimensions’ scores, and regions further had significant differences in three dimensions: Syntactic and Clausal Complexity, Oral versus Literate Discourse, and Procedural Discourse, while Time Period was not associated with any differences. Implications for language teaching and assessment are discussed.

Teachers of action: when teachers take care of their own professional development

ABSTRACT This narrative inquiry explores the identities of five teachers from Turkey who engage in volunteering and activism besides their everyday teaching. In a teaching environment where the continuing professional development of teachers is neglected and teachers’ autonomy is limited, alongside several structural issues, some of which are a century old, it would be quite normal for teachers to feel restricted. In Turkey, performance pressures due to the nationwide, high-stakes tests limit the participant teachers’ autonomy, while the educational bodies do not provide opportunities for them to become active agents of transformative change. Nevertheless, the participants in this study manifest with their narratives that they take action against the odds that would normally hinder taking action or as a coping mechanism against adversaries in the workplace. Their unique engagements are performed in various forms, taken up early in their lives or later in their careers, and not limited to educational activities or the sphere of the school. Hence, they become examples of informal continuing professional development, which are more likely to create transformative learning opportunities, bringing authentic and long-lasting benefits.

Development and evaluation of three-dimensional transfers to depict skin conditions in simulation-based education.

Modern medical moulages are becoming increasingly important in simulation-based health professions education. Their lifelikeness is important so that simulation engagement is not disrupted while their standardization is crucial in high-stakes exams. This report describes in detail how three-dimensional transfers are developed and produced so that educators will be able to develop their own. In addition, evaluation findings and lessons learnt from deploying transfers in summative assessments are shared. Step-by-step instructions are given for the creation and application of transfers, including materials and photographic visualizations. We also examined feedback on 10 exam stations (out of a total of 81) with self-developed three-dimensional transfers and complement this with additional lessons learnt. By the time of submission, the authors successfully developed and deployed over 40 different three-dimensional transfers representing different clinical findings in high-stakes exams using the techniques explained in this article or variations thereof. Feedback from students and examiners after completing the OSCE is predominantly positive, with lifelikeness being the quality most often commented upon. Caveats derived from feedback and own experiences are included. The step-by-step approach reported can be adapted and replicated by healthcare educators to build their own three-dimensional transfers. This should widen the scope and the lifelikeness of their simulations. At the same time we propose that this level of lifelikeness should be expected by learners as not to disrupt simulation engagement. Our evaluation of their use in high-stakes assessments suggests they are both useful and accepted.

IRTree modeling of a large scale, high stakes exam in India: Candidates' latent traits, and a policy implication

IRTree modeling of a large scale, high stakes exam in India: Candidates' latent traits, and a policy implication

Using Constructed Response Questions on High-Stakes Assessments in Large Classes With Limited Support

Using Constructed Response Questions on High-Stakes Assessments in Large Classes With Limited Support

Validity Evidence for Procedure-specific Competence Assessment Tools in Orthopaedic Surgery: A Scoping Review.

Competency-based training requires frequent assessment of residents' skills to determine clinical competence. This study reviews existing literature on procedure-specific competence assessment tools in orthopaedic surgery. A systematic search of eight databases up to May 2023 was conducted. Two reviewers independently assessed validity evidence and educational utility of each assessment tool and evaluated studies' methodological quality. Database searching identified 2,556 unique studies for title and abstract screening. Full texts of 290 studies were reviewed; 17 studies met the inclusion criteria. Bibliography review identified another five studies, totaling 22 studies examining 24 assessment tools included in the analysis. These tools assessed various orthopaedic surgery procedures within trauma, sports medicine, spine, and upper extremity. Overall validity evidence was low across all studies, and was lowest for consequences and highest for content. Methodological quality of studies was moderate. Educational utility assessment was not explicitly done for most tools. The paucity of current procedure-specific assessment tools in orthopaedic surgery lacks the validity evidence required to be used reliably in high-stake summative assessments. Study strengths include robust methodology and use of an evidence-based validity evidence framework. Poor-quality existing evidence is a limitation and highlights the need for evidence-based tools across more subspecialties.

Innocuous Exam Features? The Impact of Answer Placement on High-Stakes Test Performance and College Admissions

Innocuous Exam Features? The Impact of Answer Placement on High-Stakes Test Performance and College Admissions

The use and impact of pre-task planning time in the monologic task of LanguageCert speaking tests

Extensive oral tasks or monologues of different types (e.g., presentations, storytelling) are often used as second language acquisition tasks in the fields of language learning and language testing. Pre-task planning time is a common provision to test-takers who may use different strategies to prepare their response. High-stakes tests, such as the LanguageCert IESOL suite of tests, include planning time prior to monologic tasks and offer test-takers the opportunity for note-making. While the language assessment literature supports planning time for reasons of face validity and fairness, research studies do not consistently support correlations between planning and performance. The current study examined the differences between the scores of test-takers who used note-making as a strategy and those who did not. The research questions investigated: (i) whether test-takers who make notes during planning time in the monologue task of an L2-English B2 speaking task are awarded higher scores on their spoken performances than test-takers who do not and (ii) test-takers’ perceptions of their use of planning time. The findings suggest that making notes did not improve test-takers’ performance against any of the rating criteria used in the assessment. It also revealed that most test-takers use their planning time to generate their main propositions.

Comparing Estimated and Real Item Difficulty Using Multi-Facet Rasch Analysis

This study aimed to compare estimated item difficulty based on expert opinion with real item difficulty based on data. For security reasons, some high-stakes tests are not pre-tested and item difficulty is estimated by teachers in classroom assessments, so it is necessary to examine the extent to which experts make accurate predictions. In this study, we developed a 12-item assessment test like the Turkish teacher certification exam. Item difficulty was estimated and compared separately based on 1165 student responses and the opinions of 12 experts. The study revealed that the experts had a good ability to estimate item difficulty for items of moderate difficulty. However, they tended to underestimate item difficulty for items categorized as medium-easy.

Unveiling the Tapestry of Learning: Exploring the Pedagogical Landscape for Transversal and Academic Competencies through the Lived Experiences of School Teachers and Students

This research investigates the pedagogical approaches employed for the development of transversal and academic competencies based on the lived experiences of students and teachers. Despite the presence of qualified educators in educational activities, schools face numerous challenges in fully realizing the potential of students. In the context of developing nations like Nepal, schools and teachers encounter unpreparedness in facilitating activities geared towards enhancing students' academic and transversal competencies. Despite teachers' expressed commitment in workshops to adopt student-centered methods, conventional lecture-based approaches persist due to the impact of high-stakes testing. The absence of clear guidelines and motivation further contributes to the prioritization of traditional testing over competency-based education. Furthermore, inadequate facilities and an unsupportive environment hinder the implementation of practice-based education in schools. Recognizing that students excel when engaged in enjoyable activities, it is imperative to align educational activities with learners' needs and interests while progressing towards predefined objectives. This study advocates for a blended management model, incorporating both top-down and bottom-up approaches, to encourage teachers in adopting student-centered methods, thereby playing a pivotal role in nurturing both academic and transversal competencies. Given that both teachers and students tend to prioritize aspects that align with assessment criteria, evaluating assessments covering both competencies become crucial for a comprehensive understanding of their educational progress.

‘Cracking the teacher’s code’ – an ethnographic study of students preparing for oral exams

ABSTRACT This paper explores students’ preparatory processes for high-stakes exams using Danish oral exams as an example. To graduate, students must convince two teacher-examiners as the state’s representatives that they deserve to pass. Average grades determine students’ admission into tertiary education. Fieldwork data following students transitioning from teaching to exams show students ‘cracking teachers’ codes’ instead of delivering the curriculum requirement of independent critical interpretations of text material. norms of subjects called ‘super-addressees’ (Bakhtin sss1986 [“The Problem of Speech Genres.” In Speech Genres and Other Late Essays, edited by Carol Emerson and Michael Holquist, 60–102. Austin: University of Texas Press]) are not transparent to students despite an almost symbiotic interrelationship between teaching and exams. Students identify ‘Fitting responses’ (Bitzer 1968 [“The Rhetorical Situation.” Philosophy & Rhetoric 1 (1): 1–14]) for exams by navigations named 1: Disciplinary seduction in a transparent exam system, 2: Physically observing the teacher’s reactions in an opaque exam system, or 3: Jotting down an encyclopaedia of teacher’s keywords. Researching students anticipating exams illuminates their notions of what it takes to succeed in education.

Investigation of the factors affecting the curriculum fidelity of teachers working at different types of schools


 
 
 
 
 The purpose of this study is to examine the factors affecting the curriculum fidelity of teachers working in different types of school. In the study, case study design from qualitative research designs was used. The participants of the study werre teachers working in primary, secondary and high school types in publice school affiliated to the Ministry of National Education in rural district in a province located in the Northers Anatolia region of Türkiye in the fall semester of the 2022-2023 academic year. The participants was determined by convenient sampling method. The data of the study were obtained through a semi-structured interview form. The research data were analyzed using the thematic analysis method in accordance with qualitative research and findings were determined. While determining the findings, themes and categories were obtained and direct quotations were included to reflect the teachers' opinions. As a result of the research; It was determined that they expressed the opinion that curriculum characteristics, institutional characteristics, student characteristics, geographical conditions, family structure, social development of students, centralized education system, high-stakes tests, and pre-service and in-service training that teachers receive affect teachers' curriculum fidelity.
 
 
 

Students’ Understandings Related to Their Upcoming High-Stakes Tests: A Phenomenographic Study

The aim of the present study is to uncover the understanding of the students who struggle with high-stakes tests and to investigate the variations in their perceptions, taking into account the dimensions of variation. Phenomenography was chosen as the research method for this purpose. The study involved 18 students who prepared for three different high-stakes tests (LGS, YKS, and KPSS) during the 2021-2022 school year. Data were collected through semi-structured interviews. The researchers examined what students experienced while preparing for their high-stakes tests, how they felt, and they understood their tests. After phenomenographic analysis, the researchers identified six different ways of understanding related high-stakes tests. These are, parting of the ways which determine the direction of their lives; frightening; a time of strains; at an inconvenient time; an experience that improves me; an opportunity to succeed through striving. It is clear that age, experience, and number of retakes of the test are critical in forming these understandings.

Effects of errors on ratings of writing performances – Evidence from a high-stakes exam

The study reported in the paper starts with a hypothesis that errors observable in writing performances can account for much of the variability of the ratings awarded to them. The assertion is that this may be the case even when prescribed rating criteria explicitly direct rater focus towards successfully performed aspects of a writing performance rather than towards errors. The hypothesis is tested on a sample of texts rated independently of the study, using a five-point analytic rating scale involving ‘Can do’-like descriptors. The correlation between errors and ratings is ascertained using ordinal logistic regression, with Pseudo R2 of 0.51 discerned overall. Thus, with roughly 50% of score variability explainable by error occurrences, the stated hypothesis is considered confirmed. The study goes on to discuss the consequences of the findings and their potential employ in assessment of writing beyond the local assessment context.

English High-stakes Testing in Indonesia: Friend or Foe?

The Education Ministry has eliminated standardised high-stakes testing called Ujian Nasional (UN) since 2021 throughout schools all around Indonesia. However, it might be practical to look backwards at the UN and its implementation with the impact it gives on the teachers. This article has a concern about the washback consequences of a high-stakes exam on teachers. The effects were investigated within the specific context of the 2016 UN for students in Indonesia at the junior secondary level. A semi-structured interview with eight English teachers was set up to explore the teachers' perceptions towards the UN. Regarding feelings, all participants shared divergent feelings ranging from disinterested to excessive levels of fear or anxiety and mixed feelings. The data also showed mixed opinions, both positive and negative, regarding the examination. The students were thought to be motivated to study because the examination was used for selection purposes. The results did, however, also imply that the students' performance on the national exam was thought to have been assessed unfairly. The instructors demonstrated how the exam affected their teaching roles and methods. Making the national exam a low-stakes test was one of the suggestions made by attendees to enhance it.

Bridging Validity Frameworks in Assessment: Beyond Traditional Approaches in Health Professions Education

Construct: High-stakes assessments measure several constructs, such as knowledge, competencies, and skills. In this case, validity evidence for test scores’ uses and interpretations is of utmost importance, because of the consequences for everyone involved in their development and implementation. Background: Educational assessment requires an appropriate understanding and use of validity frameworks; however, health professions educators still struggle with the conceptual challenges of validity, and frequently validity analyses have a narrow focus. Important obstacles are the plurality of validity frameworks and the difficulty of grounding these abstract concepts in practice. Approach: We reviewed the validity frameworks literature to identify the main elements of frequently used models (Messick and Kane’s) and proposed linking frameworks including Russell’s recent overarching proposal. Examples are provided with commonly used assessment instruments in health professions education. Findings: Several elements in these frameworks can be integrated into a common approach, matching and aligning Messick’s sources of validity with Kane’s four inference types. Conclusions: This proposal to contribute evidence for assessment inferences may provide guidance to understanding the use of validity evidence in applied settings. The evolving field of validity research provides opportunities for its integration and practical use in health professions education.

Predicting the difficulty of EFL reading comprehension tests based on linguistic indices

Estimating the difficulty of reading tests is critical in second language education and assessment. This study was aimed at examining various text features that might influence the difficulty level of a high-stakes reading comprehension test and predict test takers’ scores. To this end, the responses provided by 17,900 test takers on the reading comprehension subsection of a major high-stakes test, the Iranian National University Entrance Exam for the Master’s Program were examined. Overall, 63 reading passages in different versions of the test from 2017 to 2019 were studied with a focus on 16 indices that might help explain the reading difficulty and test takers’ scores. The results showed that the content word overlap index and the Flesch-Kincaid Reading Ease formula had significant correlations with the observed difficulty and could therefore be considered better predictors of test difficulty compared to other variables. The findings suggest the use of various indices to estimate the reading difficulty before administering tests to ensure the equivalency and validity of tests.

Curriculum Standpoint for Social Justice: Understanding the Politics of School Knowledge

This conceptual paper argues for the need for us to understand the relationship between curriculum and social justice, largely through the lens of what the author calls “curricular standpoint.” This framing helps define the connective tissue of how social, cultural, institutional, economic, and political context manifest within the struggle over knowledge, including which knowledge gets framed as commonsense and hegemonic through things like textbooks, state standards, and high-stakes exams – thereby also implicated student consciousness about the world. Drawing on Feminist Standpoint Theory, this paper ultimately argues that, if we can fully recognize the extent to which social relations manifest in school knowledge, we also create the conditions for a critical examination of that knowledge as well as the possibilities for the development of increased critical consciousness about the curriculum and society – thus detailing the relationship between curriculum and social justice.

Objective Structured Clinical Examinations (OSCEs) for High-Stakes Assessment of Certified Registered Nurse Anesthesiologists (CRNAs)

BackgroundCertified registered nurse anesthesiologists (CRNAs) undergoing high stakes assessment must demonstrate competency. Written competency is the current method of high-stakes assessment, and this paper uses Miller's Pyramid of competency to explore use of the objective structured clinical examination (OSCE) for high-stakes assessment. MethodsThe modified Delphi method was used with 10 experts to design and execute an anaphylaxis OSCE simulation scenario of 3 proficiency levels. OSCE performance assessment tools included a checklist and global rating scale (O-GRS). A system usability survey was used to assess OSCE utility. ResultsThe overall system usability survey score was below our goal of 75. Assessment tool results did not demonstrate high correlation among panelists, although mean scores for each performance showed statistically significant differences in scores (p < .001). ConclusionOur paper showed the process of OSCE design, execution, and utility. Although results did not allow us to strongly recommend use of OSCEs for high-stakes assessment of CRNAs, we are encouraged that our results will provide direction for future papers related to this topic.

‘You are the hero’: teaching great literature in a London Jesuit school

This article argues that Ignatian pedagogy in the Jesuit tradition allows great literature to become transformative. It takes as a case study an extracurricular course – ‘Becoming a Hero: From Homer to Harry Potter’ – taught to a socially diverse group of key stage three ‘gifted and talented’ children in a London Jesuit comprehensive school. While secondary teaching is normally geared towards high-stakes summative assessment, this course aims instead at personal growth. Young people discover themselves as the heroes in a perennial plotline which appears throughout both classic texts and young adult fiction. The teacher (himself a Jesuit) had previously taught the course to undergraduates at Harvard University. Adapting it for younger children, he makes the case for ‘gifted and talented’ provision and narrates his use of assessment for prior learning and creative teaching techniques in order to ensure holistic – spiritual, intellectual, and emotional – learning beyond the curriculum.