Low Stakes Research Articles

Who teaches the teachers? A RCT of peer-to-peer observation and feedback in 181 schools

This paper evaluates a widely used, low stakes, teacher peer-to-peer observation and feedback program under Randomized Control Trial (RCT) conditions. Half of 181 volunteer primary schools in England were randomly selected to participate in a two-year program in which three fourth and fifth grade teachers observed each other. We find that two cohorts of students taught by treated teachers perform no better on externally graded national tests compared to business as usual. However this masks large heterogeneity; in small schools, where there is only one class per grade, we find negative impacts of the training (0.1-0.18SD), whereas we find positive impacts in larger schools (0.06-0.17SD). We outline and explore potential mechanisms for this and conclude that centralised one-size-fits-all teacher training interventions may be harmful.

EEG Correlates of Engagement During Assessment

In educational assessment, low engagement is problematic when tests are low stakes for students but have significant consequences for teachers or schools. In the current study, we sought to establish the electroencephalographic (EEG) correlates of engagement and to distinguish engagement from mental effort. Forty university students participated in a simulated GRE® General Test session while scalp EEG was recorded from 128 channels. Participants completed two verbal and two quantitative GRE test blocks for a total of 40 items each, and after each item, rated either their engagement or mental effort on a scale of 1–6. We computed power for seven frequency bands (delta, theta, alpha, beta, and low, medium, and high gamma) across six regions of interest: left hemisphere (LH) and right hemisphere (RH) frontal, temporal, and parietal. Preliminary results suggested that gamma power (30–150 hertz [Hz]) indexed differences between high‐ and low‐engagement ratings. This pattern was similar but weaker for mental effort. A cumulative logit model with cross‐classified random effects determined that high gamma (90–150 Hz) over the LH temporal cortex predicted engagement ratings, while controlling for reaction time and accuracy. However, for effort ratings, reaction time was the sole significant predictor. These results suggest that high gamma may be a correlate of engagement during complex cognitive tasks, but not a correlate of effort. The findings are a promising step toward the goal of objectively measuring engagement during assessment tasks.

How Chinese higher education students perceive and engage in self-assessment within the Integrated Quality Assessment (IQA) system: Threats to the validity of IQA self-assessment

The Integrated Quality Assessment (IQA) in China is an officially mandated system that uses student self-assessment in addition to information provided by tests and examinations and other sources to make judgements about student achievement across a range of qualities. Officially, this system of assessment serves two purposes: to help students become more aware of their strengths and weaknesses and to provide summative scores and grades that are used to allocate scholarships and other valued outcomes. The small-scale exploratory study reported in this article investigated People’s Republic of China (PRC) students’ experiences and perceptions of IQA and self-assessment within the IQA system. Fifteen Chinese students studying at a New Zealand university who had the experience of IQA self-assessment in their undergraduate degrees participated in individual semistructured interviews. Findings highlighted how, when individuals perceived IQA self-assessment as low stakes, they went through the motions without any deep engagement, while those who perceived the same situation as high stakes did whatever was deemed necessary to achieve their goals. For this latter group of students, IQA self-assessment encouraged, reinforced, and rewarded less-than-honest and inaccurate self-evaluations, calling into question the validity of IQA and self-assessment within this system.

At the Door to the Conference, I Beg the White Guard to Let Me In

At the Door to the Conference, I Beg the White Guard to Let Me In Hazem Fahmy (bio) Badge & ID. I have it, I swear:I paid, I paid. Sorry,it is his job. Nothingpersonal. I knowit is low stakes; an upset pebblerolling into the ocean—not a job,or healthcare. If I don't get in,I will survive. I knowit's just security;protocol—all for mysafety. But, I knowa houndwhen I see one, I countthe steps from hereto a border,to a checkpoint. [End Page 72] I stand at the doorbegging this white guardto believe me when I say I amwho I say I am. Look,it's me in the picture. (I amsmiling). It was a good daywhen I took that picture. [End Page 73] Hazem Fahmy Hazem Fahmy is a Pushcart-nominated poet and critic from Cairo. He is currently pursuing his ma in Middle Eastern Studies and Film Studies from the University of Texas at Austin. His debut chapbook, Red//Jild//Prayer won the 2017 Diode Editions Contest. A Kundiman and Watering Hole Fellow, his poetry has appeared, or is forthcoming in Apogee, AAWW, the Boston Review and The Offing. His performances have been featured on Button Poetry and Write About Now. He is a reader for the Shade Journal, and a contributing writer to Film Inquiry. Copyright © 2021 University of Nebraska Press

Effectiveness, efficiency, engagement: Mapping the impact of pre-lecture quizzes on educational exchange

Our study addresses a systemic issue facing higher education – a lack of rigorous educational research alongside new technology-assisted ways of teaching and learning. The issue highlights the disciplinary disconnect as many academics do not research outside their discipline, yet are tasked with educational modernisation through trying out new educational technology. Addressing this issue, we present our conceptual framework, the course transaction space (CT-space), and use it to analyse the impact of an intervention we designed that involved the use of regular online pre-lecture quizzes in a university mathematics course. The aim of the intervention was to optimise the effect of distributed (spaced) practice on long-term retention. Our findings suggest that a relatively small change in course instruction can improve the efficiency and effectiveness of educational exchange. Our analyses of data from multiple sources provide evidence that our intervention resulted in a sustained increase in the frequency of students’ engagement with the content, increased attendance of lectures, and improved grades. Additionally, we discuss the impact of our intervention on the quality of student engagement with reference to competence related beliefs and self-efficacy. Finally, we discuss how our intervention can be used in other contexts for supporting an evidence-based approach to teaching and learning.
 Implications for practice or policy 
 
 For teachers designing an intervention with the aim of improving students’ learning engagement during a course of tertiary study, we advise incorporating a series of frequent low stakes online quizzes with low level of difficulty.
 For students, these will act as an incentive, enabling improvement in the frequency of their learning engagement and its quality.
 The course transaction space (CT-space) model can be used to explore and analyse the impact of a variety of interventions introduced in tertiary courses through the lens of engagement.

Convention and the Origins of Ownership

We examine contemporary game-theoretic accounts of ownership as a convention. New results from dynamic networks complicate matters, suggesting that if ownership is conventional, it should not be as prevalent as it seems to be. In fact, such models reveal a tendency toward antiownership norms. The value of resources may be crucial: low stakes lead to conventional ownership, but ownership norms rarely evolve; high stakes lead to a predominance of ownership at the cost of its conventionality. We argue that conventional ownership norms can originate in nonconventional ways and discuss some philosophical implications.

Strategic confrontation: Examining the utility of low stakes prodding as a strategy for confronting sexism

ABSTRACT Confronting sexism has been routinely operationalized in experimental research as assertive behavior that expresses disapproval of sexism. The present research examines an indirect confrontation strategy we term low stakes prodding. We first establish the use of this strategy in a real-time sexist situation (Pilot study, N = 59), then test whether this indirect type of confrontation is considered to be an effective strategy for reducing sexism (Study 1, N = 138), and finally investigate whether low stakes prodding is deliberately used to avoid work-related costs (Study 2, N = 129). Study 1 demonstrated that the majority of respondents perceived low stakes prodding as confrontation, but viewed it as less effective when it was delayed. Study 2 showed that low stakes prodding was seen as a somewhat deliberate confrontation strategy that contributes to positive interpersonal interactions. Overall, the current research suggests that definitions of confrontation need to be expanded in order to encompass strategies used in actual situations of sexism.

Simulated Mass Casualty Incident Triage Exercise for Training Medical Personnel.

The target audience is any medical professional who requires training in mass casualty incident (MCI) triage. This could apply to pre-hospital specialists, nurses, medical students, residents, and physicians. Emergency medicine specialists must be able to triage patients quickly, especially in an MCI scenario. The simple triage and rapid treatment (START) system allows providers to categorize patients according to the urgency with which patients must access limited resources. Providers should be comfortable utilizing the START triage system before an MCI or disaster so that they can be prepared to implement it if necessary. This exercise uses simulation and gamification as instructional strategies to encourage knowledge of and comfort with the START triage system for emergency providers. By the end of this exercise, learners should be able to (1) recite the basic START patient categories (2) discuss the physical exam signs associated with each START category, (3) assign roles to medical providers in a mass casualty scenario, (4) accurately categorize patients into triage categories: green, yellow, red, and black, and (5) manage limited resources when demand exceeds availability. Gamification is the use of elements of game design in non-game contexts.1 Gamification was implemented in this scenario by assigning participants to roles and teams, while creating an engaging, fun, and competitive environment. The exercise also uses low fidelity simulation (without simulation equipment) to encourage learners to practice using the START triage system in a low stakes environment.2 It is possible for the learners to be divided into two groups that each have the same patients, resources, and objectives. The team that finishes triaging all patients first would be declared the winner. However, in our implementation, we completed the exercise as a single group of learners and patients. Learners were given a survey at the end of implementation and also given the opportunity to discuss feedback with the instructors in a group discussion after completing the exercise. There was no formal assessment completed after the exercise. Informal feedback was collected at the end of the exercise. Residents and medical students all enjoyed the experience. The feedback was overwhelmingly positive. All participants providing feedback stated they would enjoy participating in the exercise again and suggested that it is implemented annually for review of triage topics. We also received informal feedback for suggested changes which we will discuss in this article. An optional, anonymous survey was given to participants at the end of the exercise. There were six responses. Of those surveyed, 100% of participants stated the effectiveness and value of the exercise was outstanding (a rating of five on a scale of one to five). Regarding the quality of the exercise, and whether the participants felt engaged, 100% of responses gave a rating of five. When asked to consider the relevance of the session, 100% of participants selected a score of five ("I loved this session"). Regarding whether the content was applicable to practice of emergency medicine, 80% of respondents stated the session was highly relevant and 20% of responses selected a score of mostly relevant. One question asked for points of improvement for the session to which there were no responses. Learners were assigned roles in the exercise by the incident commander, fulfilling objective three. The START categories were discussed at the beginning of the exercise by the lead proctor (using PowerPoint) and then utilized throughout the exercise, thus accomplishing objectives one and two. The residents/students filling the triage roles were primarily responsible for fulfilling objective four; however, all participants assisted in categorization of patients throughout the exercise. Finally, objective five was addressed through the various social situations and complications that can be implemented during the exercise. We chose not to implement the additional "radiation contamination" scenario (details available in the article text) due to time constraints; however, this is an additional option to address objective five. The implementation was effective based on informal feedback from participants and proctors as well as evidenced by the responses to the anonymous survey. Learners found the aspects of resource management, review of START triage, repetition of the START triage system, and medical management of various types of trauma informative and meaningful. We received valuable feedback from both learners and proctors, which we will discuss in this article. Mass casualty incident, disaster, START, gamification, simulation, emergency medicine, triage, triage category, contamination, teamwork, trauma, projectile trauma, penetrating injury, blunt trauma, intracranial hemorrhage, fracture, trauma in pregnancy, active shooter, radiation, radio communication.

Independence in the Making: Using Makerspace Experiences to Build Foundational Entrepreneurial Competencies

This paper describes the makerspace challenge, an experiential assignment that builds foundational entrepreneurial competencies through the creation of artifacts within campus makerspaces. This assignment strengthens student ability to navigate ambiguity, learn new technologies, effectively utilize resources, and bring creative ideas to reality in a low stakes supportive environment. Students choose 3 out of 4 pre-identified projects to complete outside of class over a three week period, presenting a tangible artifact in-class each week. Grounded in constructionist pedagogy, this assignment is particularly appropriate for introductory entrepreneurship courses, but can easily be scaffolded to be applicable throughout the entrepreneurship curriculum.

Washback of the performance-based test of reading for EFL learners

The effect of tests on the teaching and learning, generally known as washback, has been recognized by scholars for a long time. However, studies on washback are usually addressed for high-stakes testing. This study investigates the washback effect of low-stakes test namely performance-based test used in measuring reading skills. Such a test is low stakes because it does not bring about serious consequence on the part of the students. The subjects of the research were 10 English teachers and 50 students of junior high schools in Semarang, Central Java, Indonesia. Data were collected through semi-structured interview and a questionnaire for both the teachers and students. The results showed that performance-based test gives positive effect in reading for the EFL learners in the areas of: students� enthusiasm in reading, reducing boredom in reading, students� curiosity on reading text content, and students� improvement on higher-order thinking skills. To the teachers it affects the areas of teaching methods, teaching materials, and time allotment.

Política de responsabilização escolar (accountability school) e formação de professor: cenários e atores na avaliação educacional brasileira

The accountability policy instrument is the large-scale assessments applied by national, state and municipal managers. This study aims to conduct a discussion on academic teacher education and school accountability policy (accountability school) developed in Brazil, observing its applicability in the educational evaluation policy. We carry out a bibliographic and documentary approach, based on the thinking of scholars: Bonamino and Sousa (2012), Brooke (2006), Horta Neto (2010), Hoffmann (2005), Dias Sobrinho (2003), Pontual (2008) among others. The study showed that in Brazil there are three types of educational evaluation policy; a diagnosis that has no implications for the school and the curriculum; another of bland accountability (low stakes) which has symbolic consequences for the school and its actors and the third classified as strong accountability (high stakes) causing inferences with the school and its actors.

Exploring Engagement as a Predictor of Success in the Transition to Online Learning in First Year Chemistry

Rapid campus closure and unexpected transition to emergency online teaching of a large first year chemistry unit provided an opportunity to explore the level of student engagement and factors associated with success in this online transition. Here, we compared the standard offering in 2019 to the online version in 2020 by examining the completion of weekly assessment tasks and the uptake of learning activities. In spite of a preference for face-to-face learning, we found that students were able to quickly adapt to using the technology, with high levels of access to live streamed and recorded lectures, in-class polling, and online tutorials, together with large increases in discussion forum use. The change in level of engagement in low stakes assessment tasks over the course of the semester was similar for both years. Each form of engagement was associated with increased likelihood of success in the unit, whether this was being taught face-to-face or online. Of the multiple forms of engagement measured, only tutorial participation had a significantly lower impact on student success in the online compared to the face-to-face environment.

3D virtual reality simulation in radiography education: The students' experience

IntroductionSimulation forms a key element of undergraduate Radiography education as it enables students to develop their clinical skills in a safe environment. In this study, an immersive three-dimensional (3D) virtual radiography simulation tool was piloted in an undergraduate Radiography curriculum and user feedback retrieved. MethodsThe 3D virtual simulation tool by Virtual Medical Coaching Ltd was introduced to first year radiography students (n = 105). This technology guided students through a comprehensive process of learning anatomy, radiographic positioning and pathology. Students then X-rayed a virtual patient in the VR suite using HTC Vive Pro™ headsets and hand controllers. Instant feedback was provided. An online survey was later disseminated to students to gather user feedback. Thematic and descriptive statistical analyses were applied. ResultsA response rate of 79% (n = 83) was achieved. Most respondents (58%) reported enjoying VR simulation, whilst some felt indifferent towards it (27%). Ninety-four percent would recommend this tool to other students. The mean length of time it took for students to feel comfortable using the technology was 60 min (10–240 min). Most respondents (58%) desired more VR access. Students attributed enhanced confidence in the areas of beam collimation (75%), anatomical marker placement (63%), centring of the X-ray tube (64%) and exposure parameter selection (56%) to their VR practice. Many students (55%) advocated the use of VR in formative or low stakes assessments. Issues flagged included technical glitches, inability to palpate patient and lack of constructive feedback. ConclusionStudent feedback indicates that 3D virtual radiography simulation is a valuable pedagogical tool in radiography education Implications for practice3D immersive VR simulation is perceived by radiography students to be a valuable learning resource. VR needs to be strategically implemented into curricula to maximise its benefits.

Amazon Mechanical Turk workers can provide consistent and economically meaningful data

AbstractAmazon Mechanical Turk (AMT) is an online labor market that is being used increasingly often in the social sciences. This occurs despite significant questions regarding efficacy of the platform. In this article, we attempt to address some of these questions by exploring the consistency of the characteristics of individuals who participate in studies posted on AMT. The primary individuals analyzed in this study are subjects who participated in at least two of eleven experiments that were run on AMT between September of 2012 and January of 2018. We demonstrate subjects consistently report their age, gender, subjective willingness to take risk, and impulsiveness. Further, subjective willingness to take risk is found to be significantly correlated with decisions made in a simple lottery experiment with real stakes—even when the subjective risk measure is reported months, sometimes years, in the past. This suggests the quality of data obtained via AMT is not terribly harmed by the lack of control and low stakes.

Assertion, Stakes and Expected Blameworthiness: An Insensitive Invariantist Solution to the Bank Cases

Contextualists and Subject Sensitive Invariantists often cite the knowledge norm of assertion as part of their argument. They claim that the knowledge norms in conjunction with our intuitions about when a subject is properly asserting in low or high stakes contexts provides strong evidence that what counts as knowledge depends on practical factors. In this paper, I present new data to suggest they are mistaken in the way they think about cases involving high and low stakes and I show how insensitive invariantists can explain the data. I exploit recent work done on the distinction between flouting a norm and being blamed for that violation to formulate a rigorous theory of rational expected blameworthiness that allows insensitive invariantists to explain the data cited.

A Calibrated Item Bank for Computerized Adaptive Testing in Measuring Science TIMSS Performance

The current assessment is demanding for a more personalised and less-time consuming testing environment. Computer Adaptive Testing (CAT) is seemed as a more effective alternative testing method in comparison to conventional test in meeting the current standard of assessment. This research reports on the calibration of the released Grade 8 Science objective items in Trends in International Mathematics and Science Study (TIMSS) 2003-2015 based on Rasch model framework to be used in CAT as an alternative testing tool in a low stakes test. Concurrent common item equating method was used in linking and equating sets of items. Five test sets were produced consisting of 20 unique items and 10 common items in each set. The unique items were available only in a single set while common items were available in the two-consecutive set of items. The sets were administered through Paper and Pencil test to Form 2 (Grade 8) students who had been selected through a purposive sampling method from secondary schools in the northern part of Malaysia. The fit analysis, polarity analysis, unidimensionality analysis, item measure and Person-Map-Item were conducted. The analysis produced 122 calibrated items which meet the Rasch’s requirements and were suitable to be used in CAT.

Caught in the Act: Predicting Cheating in Unproctored Knowledge Assessment.

Cheating is a serious threat in unproctored ability assessment, irrespective of countermeasures taken, anticipated consequences (high vs. low stakes), and test modality (paper-pencil vs. computer-based). In the present study, we examined the power of (a) self-report-based indicators (i.e., Honesty-Humility and Overclaiming scales), (b) test data (i.e., performance with extremely difficult items), and (c) para data (i.e., reaction times, switching between browser tabs) to predict participants' cheating behavior. To this end, 315 participants worked on a knowledge test in an unproctored online assessment and subsequently in a proctored lab assessment. We used multiple regression analysis and an extended latent change score model to assess the potential of the different indicators to predict cheating. In summary, test data and para data performed best, while traditional self-report-based indicators were not predictive. We discuss the findings with respect to unproctored testing in general and provide practical advice on cheating detection in online ability assessments.

The assessment clock: A model to prioritize the principles of the utility of assessment formula in emergency situations, such as the COVID-19 pandemic.

This article was migrated. The article was marked as recommended. Many concerns have been raised regarding the impact of the changes to medical education as a result of the coronavirus disease 2019 (COVID-19) pandemic, particularly the impact of these changes on student assessments. This paper suggests an assessment clock as a conceptual model to enable medical educators to decide which assessment method is suitable under challenging circumstances, such as the COVID-19 pandemic. The assessment clock has five numbers, representing the five principles of the utility of assessment formula, which are arranged from the principle with the lowest weight (cost = 1) to the principle with the highest weight (validity = 5). The numbers are repeated in each half of the clock, and the clock is placed in the middle of two overlapping axes. The vertical axis is related to exam stakes (high or low). The low stakes condition, which represents the normal situation of running assessments at the beginning of each academic year, is placed at the top of the clock. The horizontal axis is related to the type of situation (normal or crisis). The high stakes condition is placed at the bottom of the clock. The right half of the clock represents the normal situation of planning and conducting assessments, while the left half represents an emergency situation, such as the current COVID-19 pandemic. The assessment clock offers assessment planners insights into how to determine the most important assessment principles on which they should focus during a crisis situation. Moreover, it provides practical guidance for educators to help them decide which assessment tool is suitable for use in which situation.

Rethinking Assessment in Medical Education in the time of COVID-19

This article was migrated. The article was not marked as recommended. The COVID-19 pandemic has impacted the design, delivery and processes in medical education. While the crisis continues, it is important to develop creative ways of assessment that continue to maintain the standards of medical education and accommodate the present environmental and social limitations brought on by COVID. This crisis also offers an opportunity to transition programs to a competency based curriculum.The transition should be comprehensive but staged to include current and prospective learners. It is important to first consider the existing framework of assessment and work backwards on design and process. As an immediate and first step would be to add multiple opportunities for formative assessment into the new approaches adopted for teaching/learning (T/L). Standard assessment formats in medical school that include written exams (like MCQs, SAQs and practical skills via OSCE) will need to be re-imagined and re-planned. It may now be time to consider shifting focus of assessment to attainment of competencies. This is a longer term process but can be phased in. As a start using low stakes assessment as part of entrustable professional activities that are task directed and encompass domain of knowledge, skills and attitude.Revisiting older modalities and introducing new ways that accommodate distant engagement may need consideration for e.g. structured viva, use of log books for assessment, virtual patients, e-portfolios.Thus an emphasis in assessment should be on using multimodal tools for formative and summative assessments that focus on mastery in clinical reasoning, problem solving and decision making skills allowing transition to a competency based curriculum.The key task will be the timely creation and monitoring of a balanced face to face and online assessment program keeping in mind resource availability, faculty buy-in and support of regulatory and licensing bodies.

Intricacies of designing and implementing enrichment programs for high-ability students

A common pedagogical approach to foster creativity and develop talents among gifted students is after-school enrichment programs. However, in practice, there are several issues concerning the conceptualization and implementation of such programs. Since these enrichment programs are typically not part of the core curriculum and scheduled outside curriculum time, these programs are relatively low stakes. As an after-school add-on program, such programs tend to be fragmented and ad hoc in nature. Instead of a systematic program designed to stretch gifted and talented students, with the goal of developing their knowledge and competencies in breadth and depth, the programs that are implemented tend to be pitched at exposure level. To effectively meet the needs of the gifted and talented students, it is imperative to examine the intricacies of these programs. The fidelity has significant implications on the quality of the students’ learning experiences. This article shares the issues and challenges they faced.