Low Stakes Research Articles

Stakeholder views on publication bias in health services research.

ObjectivesWhile the presence of publication bias in clinical research is well documented, little is known about its role in the reporting of health services research. This paper explores stakeholder perceptions and experiences with regard to the role of publication and related biases in quantitative research relating to the quality, accessibility and organization of health services.MethodsWe present findings from semi-structured interviews with those responsible for the funding, publishing and/or conduct of quantitative health services research, primarily in the UK. Additional data collection includes interviews with health care decision makers as ‘end users’ of health services research, and a focus group with patient and service user representatives. The final sample comprised 24 interviews and eight focus group participants.ResultsMany study participants felt unable to say with any degree of certainty whether publication bias represents a significant problem in quantitative health services research. Participants drew broad contrasts between externally funded and peer reviewed research on the one hand, and end user funded quality improvement projects on the other, with the latter perceived as more vulnerable to selective publication and author over-claiming. Multiple study objectives, and a general acceptance of ‘mess and noise’ in the data and its interpretation was seen to reduce the importance attached to replicable estimates of effect sizes in health services research. The relative absence of external scrutiny, either from manufacturers of interventions or health system decision makers, added to this general sense of ‘low stakes’ of health services research. As a result, while many participants advocated study pre-registration and using protocols to pre-identify outcomes, others saw this as an unwarranted imposition.ConclusionsThis study finds that incentives towards publication and related bias are likely to be present, but not to the same degree as in clinical research. In health services research, these were seen as being offset by other forms of ‘novelty’ bias in the reporting and publishing of research findings.

Awareness that coal-powered energy is environmentally degrading insignificantly affects its consumption

This paper contributes findings from a social survey conducted to examine individual awareness of coal’s non-renewability, environmental issues, and home energy behaviours. The sample exhibited high (86%) awareness of coal’s non-renewability and 74% self-identified energy issues as key environmental problems affecting their lives. Government presumptions that education campaigns are needed to increase Australian public awareness of carbon emissions and may induce pro-environmental energy behaviours are questioned. Energy issue awareness failed to significantly affect 4 home energy behaviours (choosing A/C settings, type of home temperature control, turning lights off in unoccupied rooms, turning appliances off standby) for ‘environmental sustainability’ reasons. Findings support international research documenting disparity between individual awareness of environmental issues and pro-environmental actions for such issues. Consistent with prior research, age was the most significant factor affecting pro-environmental energy behaviours. Older individuals were most likely to consider environmental sustainability for home temperature and A/C settings. Education, age, and energy issue awareness did not significantly affect ‘low stakes’ consumption behaviours (appliances or lights) for environmental reasons. In locations with non-renewable energy, policy and education initiatives improving knowledge that aggregated carbon consumption from seemingly minor individual behaviours may reduce environmental degradation may make ‘unnecessary’ energy consumption less culturally normative or acceptable.

MAD Agents: Low Stakes, Still Mistakes?

MAD Agents: Low Stakes, Still Mistakes?

The Effects of Probabilistic Forecasts and Stakes on Electoral Participation in an Exterminal Setting

This study looks at the effect of probabilistic forecasts and policy stakes on vote intention in an experimental setting. This study examined stakes and forecasts through exposing subjects to mock newspaper articles in two experiments: a fictitious mayoral election and the 2018 U.S. Senate race in Arizona. The study found that varying the policy stakes of an election had a significant impact on intention to vote, while varying the probabilistic forecasts did not have a significant effect on intention to vote. In the mayoral experiment, when the mock article said that the candidates disagreed on a major policy (high stakes treatment), subjects were significantly more likely to indicate that they would vote in that election in comparison to the article where subjects were told that the two candidates agreed on that same policy issue (low stakes treatment). In the Arizona experiment, I found that low stakes made subjects significantly less likely to say they would vote in comparison to the control treatment. Both stakes and probability had some effects on candidate favorability and candidate preference in my mayoral experiment. My results lead me to conclude that the media has the potential to influence political participation levels through the way they cover elections.

Online Low-Stakes Assignments To Support Scientific Lab Report Writing in Introductory Science Courses.

Using a combination of online tools (discussion board, submission of reports, grading with rubrics) and scaffolded low stakes writing, we designed a comprehensive writing assignment that teaches first and second year undergraduates a basic scientific skill: written communication in the form of a laboratory report. Scaffolding of the assignment into small sections helps students concentrate on topics and writing technique, and ensures the instructor can grade the reports quickly while still providing valuable feedback. Hence, students get a chance to practice lab report writing consistently throughout the semester, simultaneously learning how to do it and improving their results. The assignment prepares students for the lab report in advanced courses or writing scientific reports in their later career. The assignment was used in biology courses, but can be used in other science courses where lab report preparation is required. This system was well received by students.

Parent and student voice in evaluation and planning in schools

Current approaches to the regulation of schools in most jurisdictions tend to combine elements of external inspection with systems of internal self-evaluation. An increasingly important aspect of the theory and practice of both, but particularly the latter, revolves around the role of other actors, primarily parents and students, in the process. Using literature review and documentary analysis as the research method, this article explores the research literature from many countries around the concerns of schools and teachers about giving a more powerful voice to parents and pupils. Then, focusing on Ireland, this article tries to clarify three things, official policy concerning stakeholder voice in school self-evaluation and decision making, the efforts by schools to implement this policy and the response to date of school leaders and teachers to this rather changed environment. Using Hart’s ladder of genuine, as opposed to token, participation, it is argued that policy mandating parental and student involvement has evolved significantly, that schools have responded positively and that there is little evidence, as yet, of teacher concern or resistance. This response is explained by the low stakes and improvement-focused education environment; the controlled, structured and simplified nature of the self-evaluation process; and the limited extent of parental and student participation in decision making.

PSXIV-40 Utilization of a group project in an introductory animal science course to concurrently develop collaborative work skills and increase animal science industry awareness

Abstract It is expected that students entering the workforce should have skills to allow them to effectively perform in the workplace. However, there is a gap between expected development of these “soft skills” and their actual attainment by students. Due to this discrepancy, Texas A&M has developed university mandated learning outcomes that address soft skills within course content. The objective of this paper is to present teaching methodology useful for achieving Texas A&M University level learning outcomes as they relate to the “Professional Conduct” learning outcome, as assessed via performance indicators such as working collaboratively, feedback, and conflict management. Additionally, this project aims to help bridge the gap in understanding between current animal science industry practices and public perception. This project titled “Ag-vocating for Animal Science Industry Practices” utilizes methods appropriate for large classes, conducted face-to-face or online, and targets students at the introductory level. Based on criteria from education literature, the semester-long project is broken down into multiple low stakes assignments comprising half of the project points which allows for multiple feedback opportunities from both the group members and the professor. These assignments include a Catme survey to assign groups based on interest and schedule, as well as submitting topic-related articles, a script, a rough-draft video, final video, and peer assessment survey online through eCampus, the university’s learning management system. This utilization of this collaborative project fulfills university undergraduate learning outcome of demonstrating professional conduct measured by attainment performance indicators such as “identify characteristics of a functional team” and “accept objective and constructive criticism.” This is an innovative approach to assessing soft skill development within the context of the animal science discipline. In conclusion, this project is useful in evaluating groupwork skills in undergraduates, as well as providing students with an opportunity to critically analyze industry practices.

Low stakes, high risks: the problem of intertemporal validity of PISA in Latin America

ABSTRACT Since 2000, the PISA test implemented by OECD has become the prime benchmark for international comparisons in education. The 2015 PISA edition introduced methodological changes that altered the nature of its results. PISA made no longer valid non-reached items of the final part of the test, assuming that those unanswered questions were more a problem of motivation than skills. Latin American countries were the only ones that had a large proportion of non-reached items in the first PISA editions. The methodological change had a large impact in these countries and was miscommunicated by the OECD reports. This makes the first two PISA editions (2000 and 2003) no longer comparable with the latest ones (2006, 2009, 2012 and 2015). We analyze hypotheses for the Latin American exception in PISA, suggesting that the introduction of the new Computer-Based Assessment of PISA in 2015, combined with an increasing culture of evaluation and political influences damaged intertemporal comparisons of PISA. Although PISA is a low-stakes assessment for students, it has become high-stakes for governments. This triggered mechanisms to increase test-taking motivation, and students tended to answer more questions of the test, but at the same time, these changes made intertemporal comparisons flawed.

Institutional strategies related to test-taking behavior in low stakes assessment

Low stakes assessment without grading the performance of students in educational systems has received increasing attention in recent years. It is used in formative assessments to guide the learning process as well as in large-scales assessments to monitor educational programs. Yet, such assessments suffer from high variation in students’ test-taking effort. We aimed to identify institutional strategies related to serious test-taking behavior in low stakes assessment to provide medical schools with practical recommendations on how test-taking effort might be increased. First, we identified strategies that were already used by medical schools to increase the serious test-taking behavior on the low stakes Berlin Progress Test (BPT). Strategies which could be assigned to self-determination theory of Ryan and Deci were chosen for analysis. We conducted the study at nine medical schools in Germany and Austria with a total of 108,140 observations in an established low stakes assessment. A generalized linear-mixed effects model was used to assess the association between institutional strategies and the odds that students will take the BPT seriously. Overall, two institutional strategies were found to be positively related to more serious test-taking behavior: discussing low test performance with the mentor and consequences for not participating. Giving choice was negatively related to more serious test-taking behavior. At medical schools that presented the BPT as evaluation, this effect was larger in comparison to medical schools that presented the BPT as assessment.

What can completion time of quizzes tell us about students’ motivations and learning strategies?

Learning management systems (LMS) offer quiz tools that help students prepare for examinations. The purpose of this study is to investigate quiz tracking variables typically reported by LMS in relation to student achievement, motivation and learning strategies. The data from 143 undergraduate students comprised quiz tracking variables (number of attempts, completion time, and score), exam scores and responses to the Need for Cognition Scale (NfC), and selected components from the Motivated Strategies for Learning Questionnaire and the Achievement Goal Questionnaire. We hypothesized students retrieving information from memory while taking a quiz would complete the quiz in less time than students who searched for answers in the textbook, and consequently, quiz completion time would correlate with exam performance and key motivational and self-regulatory factors. We found quiz completion time correlated positively with performance-avoidance goal orientation. It correlated negatively with exam performance, NfC, self-efficacy, and effort regulation. The results indicated completion time of low stakes quizzes is associated with achievement-related motivations and reliably predicts achievement on summative exams. We attribute these links to the use of retrieval practice by students who successfully regulate their effort and learning strategies.

Patterns of performance in students with frequent low stakes team based learning assessments: Do students change behavior?

Team-based learning offers an active learning strategy that provides a structure for measurement of learning and feedback to the students. Aggregating these results provides a longitudinal pattern of student performance. In this study, we analyzed results from a sequence of assessments related to TBL IRAT and GRAT assessments and traditional quizzes in a second-year musculoskeletal course in an undergraduate medical education program to determine if there are any measurable patterns, or performance trends, that students demonstrate in the course. Analyzing results from four academic years, we found evidence supporting there is predictability in student’s future week’s performance based on past performances across teaching modalities. We hypothesize that students are moderating their own effort regarding weekly low-stake assessments in prioritizing their academic efforts. The results from this study highlight the role of self-efficacy in medical education and suggest a new area of research for assessment of student performance patterns. Future studies could investigate whether these performance patterns are replicated in other assessment modalities and whether the same pattern holds for high-stakes assessments.

Investigating students’ seriousness during selected conceptual inventory surveys

Conceptual inventory surveys are routinely used in education research to identify student learning needs and assess instructional practices. Students might not fully engage with these instruments because of the low stakes attached to them. This paper explores tests that can be used to estimate the percentage of students in a population who might not have taken such surveys seriously. These three seriousness tests are the pattern recognition test, the easy questions test, and the uncommon answers test. These three tests are applied to sets of students who were assessed either by the Force Concept Inventory, the Conceptual Survey of Electricity and Magnetism, or the Brief Electricity and Magnetism Assessment. The results of our investigation are compared to computer simulated populations of random answers.Received 15 May 2019DOI:https://doi.org/10.1103/PhysRevPhysEducRes.15.020118Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.Published by the American Physical SocietyPhysics Subject Headings (PhySH)Research AreasAssessmentResearch methodologyPhysics Education Research

Catch and release? NHL expansion draft endowment effects

Purpose The purpose of this paper is to investigate National Hockey League (NHL) expansion draft decisions to measure divestment aversion and endowment effects, and analyze bias and its affect on presumed rational analytic decision making. Design/methodology/approach A natural experiment with three variables (age, minutes played and presence of a prior relationship with a team’s management), filtered athletes that were exposed or protected to selection. A machine learning algorithm trained on a group of 17 teams was applied to the remaining 13 teams. Findings Athletes with pre-existing management relationships were 1.7 times more likely to be protected. Athletes playing fewer relative position minutes were less likely to be protected, as were older athletes. Athlete selection was predominantly determined by time on ice. Research limitations/implications This represents a single set of independent decisions using publicly available data absent of context. The results may not be generalizable beyond the NHL or sport. Practical implications The research confirms the affect of prior relationships on decision making and provides further evidence of measurable sub-optimal decision making. Social implications Decision making has implications throughout human resources and impacts competitiveness and productivity. This adds to the need for managers to recognize and implement de-biasing in areas such as hiring, performance appraisal and downsizing. Originality/value This natural experiment involving high-stakes decision makers confirms bias in a setting that has been dominated by students, low stakes or artificial settings.

A novel, low fidelity simulator for laryngotracheal reconstruction

IntroductionOpen airway reconstruction is a highly specialized skill. Simulation affords the opportunity to practice surgical skills in a low stakes environment which is particularly important for a high acuity, low frequency operation. Although animal models have been described, these present ethical and financial barriers, and therefore are not ideal to expose residents to airway reconstruction techniques. To our knowledge there is not a commercially available simulator for laryngotracheal reconstruction. ObjectivesThis study describes a novel, low-fidelity simulation technique for laryngotracheal reconstruction using a cartilage graft. MethodsWe designed a low-fidelity simulator to represent the trachea, esophagus, and cartilage graft using tubing from a Luken's trap, vinyl backwash hose, and pig's ears from a non-specialty grocery store. The model was evaluated with a Likert scale (1 = strongly disagree to 5 = strongly agree). ResultsTwelve participants attended simulation sessions. Participants reported a mean score (+/-SD) 4.25 ± 0.75 that the tissue characteristics were adequate and 4.50 ± 0.79 that sutures could be placed. There was universal strong agreement that the tissue could be manipulated appropriately (5 ± 0). The cost per resident was less than 4 dollars. ConclusionWe present a readily available, easy to construct, and low cost simulation model for open airway reconstruction that can be used as a stand-alone simulator or in preparation for an animal dissection course. Our participants reported that the model had acceptable tissue characteristics to practice performing laryngotracheal reconstruction with a cartilage graft.

The Skilled, the Knowledgeable, and the Motivated: Investigating the Strategic Allocation of Time on Task in a Computer-Based Assessment.

In large scale low stakes assessments, students usually choose their own speed at which to work on tasks. At the same time, previous research has shown that in hard tasks, the time students invest is a positive predictor of task performance. From this perspective, a relevant question is whether student dispositions other than the targeted skill might affect students’ time on task behavior, thus potentially affecting their task performance and in turn their estimated skill in the target domain. Using PISA 2009 computer based assessment data, the present research investigated for the domain of reading digital text whether three variables that can be assumed to predict performance in digital reading tasks, comprehension skill, enjoyment of reading, and knowledge of reading strategies would also predict how much time students would devote to digital reading tasks, and in particular, whether they would adapt time on task to task difficulty. To address this question, two linear mixed models were estimated that predicted the time students spent on a task, and the average time students spent on relevant pages within each task, by the interaction of task difficulty with comprehension skill, enjoyment of reading, and knowledge of reading strategies. To account for time on task being nested in students and tasks, random effects for persons and tasks were included. The interaction of task difficulty with gender and Socio-Economic Status (SES) was included for control purposes. Models were estimated individually for 19 countries, and results integrated meta-analytically. In line with predictions, for both time on task indicators, significant positive interactions were found with comprehension skill, enjoyment of reading, and knowledge of reading strategies. These interactions indicated that in students with high comprehension skill, enjoyment of reading, and knowledge of reading strategies there was a stronger association of task difficulty with time on task than in students low in either of these variables. Thus, skilled comprehenders, students enjoying reading, and students in command of reading strategies behaved more adaptively than lower skilled, motivated, or knowledgeable students. Implications of these findings for the validity of self-paced computer-based assessments are discussed.

Use of E-exams in High Stakes Law School Examinations: Student and Staff Reactions

Australian law schools have just begun to move towards e-exams. There are a number of drivers encouraging Australian universities to introduce e-exams, including economic pressures, a desire to reduce administrative burdens, and changes to workplaces. Logistically, e-exams may assist with paper distribution, script legibility and shorter marking times. Pedagogical arguments in favour of e-exams rely on the fact that students already use technology daily and complete the bulk of their university work on computers. In this context, handwriting an exam has been described as an ‘anachronism’. More specifically, traditional handwriting of examinations in contemporary contexts is seen as a ‘misalignment of assessment practice to the learning environment’. Such misalignment can result in diminished learning outcomes. However, there is little literature addressing the adoption of e-exams for formal high stakes summative exams. Much of the research to date has focused on small scale, informal low stakes tests in non-law disciplines. Within Australia, whether e-exams are appropriate or effective has been the subject of a number of reported trials, but none have specifically reported on law students. In 2016 UNSW Law sought to rectify this and undertook such a trial. This article provides empirical insight into the process and student perceptions for a high stakes summative e-exam. It analyses students’ attitudes and concerns prior to the introduction of the e-exam platform, as well as after they had completed the exam. We also provide insights into the important procedural aspects associated with high stakes exams and staff responses to the process, issues not previously reported on.

The conundrum of low achievement and feedback for learning

The literature on improving student engagement with assessment and feedback has a tendency to treat all students as if they are the same. Students with lower levels of attainment are generally under-represented within empirical studies and their feedback behaviours are less well understood. The recent drive to improve student assessment and feedback literacy and the move from ‘feedback’ being information about a task to being a process of understanding and using performance information is a larger conceptual leap for some students than others. In this paper, we consider issues surrounding the transition to new modes of feedback, focussing on what is needed for those who find study difficult and persistently are disappointed by their levels of attainment, to benefit from and take advantage of our feedback pedagogies. We examine literature advocating strategies such as increasing agency, using praise, developing feedback literacy and cultivating a growth mind-set. We argue that students who underachieve may benefit from strong relationships with educators and peers, exposure to feedback rich, low stakes environments, which permit repeated integrations of practice and feedback, and building feedback literacy through peer assessment activities.

Early years teachers’ perspectives on the effects of NAPLAN on stakeholder wellbeing and the impact on early years pedagogy and curriculum

National Assessment Program for Numeracy and Literacy (NAPLAN) is the national assessment programme for literacy and numeracy in Australia administered to children in Years 3, 5, 7 and 9 each year. The testing process was introduced in 2008 and is described by the developers as low stakes, however, research has highlighted that this is not the case. This paper examines the perceptions of teachers in the early years of school on the impact NAPLAN has on wellbeing of stakeholders, and the pedagogy and curriculum in early years teachers’ classrooms. Through focus group interviews, the early childhood teachers in 10 independent Western Australian school sites were asked about their experiences of NAPLAN in terms of their wellbeing and that of the children and families with whom they interact. Findings highlight that early years teachers describe that the impact of NAPLAN is felt in the lower years of school by all stakeholders-parents, children and teachers. This study highlights the need for additional research in this area, particularly in diverse settings, to gain evidence of impact that could inform the practice of the NAPLAN testing programme.

FUTURES OF GAMING: HOW CASINOS AND GAMBLING MIGHT EVOLVE IN THE NEAR FUTURE

FUTURES OF GAMING: HOW CASINOS AND GAMBLING MIGHT EVOLVE IN THE NEAR FUTURE

Differential Performance in High Versus Low Stakes Tests: Evidence from the Gre Test*

Abstract We study how different demographic groups respond to incentives by comparing their performance in ‘high’ and ‘low’ stakes situations. The high stakes situation is the Graduate Record Examination (GRE), and the low stakes situation is a voluntary experimental section of the GRE. We find that males exhibit a larger drop in performance between the high and low stakes examinations than females, and that whites exhibit a larger drop in performance than minorities. Differences between high and low stakes tests are partly explained by the fact that males and whites exert lower effort in low stakes tests compared with females and minorities.