High-stakes Contexts Research Articles

Deception Detection and Truth Detection Are Dependent on Different Cognitive and Emotional Traits: An Investigation of Emotional Intelligence, Theory of Mind, and Attention.

Despite evidence that variation exists between individuals in high-stakes truth and deception detection accuracy rates, little work has investigated what differences in individuals' cognitive and emotional abilities contribute to this variation. Our study addressed this question by examining the role played by cognitive and affective theory of mind (ToM), emotional intelligence (EI), and various aspects of attention (alerting, orienting, executive control) in explaining variation in accuracy rates among 115 individuals (87 women; mean age = 27.04 years [ SD = 11.32]) who responded to video clips of truth-tellers and liars in real-world, high-stakes contexts. Faster attentional alerting supported truth detection, and better cognitive ToM and perception of emotion (an aspect of EI) supported deception detection. This evidence indicates that truth and deception detection are distinct constructs supported by different abilities. Future research may address whether interventions targeting these cognitive and emotional traits can also contribute to improving detection skill.

Predicting self-efficacy in test preparation: Gender, value, anxiety, test performance, and strategies

ABSTRACTConsiderable evidence indicates that self-efficacy, task value, anxiety, and the use of language learning strategies are related. However, there is currently an insufficient understanding about their relations in high-stakes testing contexts. The author aimed to investigate how well social factors, test value, anxiety, test performance, and learning strategies predict high school students' self-efficacy in preparing for the English listening test as part of the University Entrance Examination in Taiwan. This research involved a large-scale questionnaire survey and a collection of students' English test scores. The participants were 636 students in Grade 12 of high school in Taiwan. A quantitative analysis of the questionnaire was conducted via hierarchical regression analysis. The results show that students' recent English test performance and test anxiety were two strong predictors of self-efficacy in preparing for the high-stakes test. In addition, resource management and metacognitive listening strategies had better predictive power over prior English level and test value.

Predicting patterns of change and stability in student performance across a medical degree.

Evidence of predictive validity is essential for making robust selection decisions in high-stakes contexts such as medical student selection. Currently available evidence is limited to the prediction of academic performance at single points in time with little understanding of the factors that might undermine the predictive validity of tests of academic and non-academic qualities considered important for success. This study addressed these issues by predicting students' changing performance across a medical degree and assessing whether factors outside an institution's control (such as the uptake of commercial coaching) impact validity. Three cohorts of students (n = 301) enrolled in an undergraduate medical degree from 2007-2013 were used to identify trajectories of student academic performance using growth mixture modelling. Multinomial logistic regression assessed whether past academic performance, a test of cognitive ability and a multiple mini-interview could predict a student's likely trajectory and whether this predictive validity was different for those who undertook commercial coaching compared with those who didn't. Among the medical students who successfully graduated (n = 268), four unique trajectories of academic performance were identified. In three trajectories, performance changed at the time when learning became more self-directed and focused on clinical specialties. Scores on all selection tests, with the exception of a test of abstract reasoning, significantly affected the odds of following a trajectory that was consistently below average. However, selection tests could not distinguish those whose performance improved across time from those whose performance declined after an average start. Commercial coaching increased the odds of being among the below-average performers, but did not alter the predictive validity of the selection tests. Identifying distinct groups of students has important implications for selection, but also for educating medical students. Commercial coaching may result in selecting students who are less suited for coping with the rigours of medical studies.

Using the early childhood environment rating scale-Revised in high stakes contexts: Does evidence warrant the practice?

Increasingly, states establish different thresholds on the Early Childhood Environment Rating Scale–Revised (ECERS–R), and use these thresholds to inform high-stakes decisions. However, the validity of the ECERS-R for these purposes is not well established. The objective of this study is to identify thresholds on the ECERS-R that are associated with preschool-aged children’s social and cognitive development. Applying non-parametric modeling to the nationally-representative Early Childhood Longitudinal Study Birth Cohort (ECLS-B) dataset, we found that once classrooms achieved a score of 3.4 on the overall ECERS-R composite score, there was a leveling-off effect, such that no additional improvements to children’s social, cognitive, or language outcomes were observed. Additional analyses found that ECERS-R subscales that focused on teaching and caregiving processes, as opposed to the physical environment, did not show leveling-off effects. The findings suggest that the usefulness of the ECERS-R for discerning associations with children’s outcome may be limited to certain score ranges or subscales.

External mentoring for new teachers: mentor learning for a change agenda

PurposeThe purpose of this paper is to report on a qualitative study of the learning and development of 70 external mentors during the first year of their deployment to support early career teachers’ professional learning as part of a national initiative aimed at school improvement in Wales.Design/methodology/approachThe study adopted a narrative methodology that elicited accounts of external mentors’ learning experiences that were captured as textual data and analysed using an inductive approach to identify: first, the manifest themes that appeared at declarative level, and second, the latent (sub-textual) themes of external mentor learning and development.FindingsFour key themes emerged that indicate the complexity of transition to the role of external mentor in high-stakes contexts. From these, eight theoretically-informed principles were derived which support mentors to embrace uncertainty as essential to their learning and development, and to harness the potential they bring as boundary-crossers to support the development of new teachers.Research limitations/implicationsThe study investigated the first year of a three-year programme and worked with one form of qualitative data collection. The research results may lack generalisability and a longitudinal study is necessary to further explore the validity of the findings.Practical implicationsThe eight principles provide a foundation for mentor development programmes that can support ambitious goals for mentoring early career teachers.Originality/valueThe study addresses the under-researched area of the learning and development of external mentors at a national scale.

Rethinking side effects of accountability in education: Insights from a multiple methods study in four German school systems

Based on a research project comprising data from an interview study and a survey with teachers and school principals in four German federal states (Bundesländer), this paper questions the claim that the side effects of accountability in education are bound to high-stakes contexts, and also provides evidence of side effects occurring in no- and low-stakes contexts. The findings suggest that side effects cannot be fully explained by certain implementation features of accountability regimes (e.g. high stakes), but should rather be understood as a result of implementation features as well as systematic effects of accountability in education.

The relationship between error production when performing motor skills in high and low-stakes situations

ABSTRACTConflicting viewpoints regarding the role of errors produced during practice of motor skills raise questions regarding errors in low-stakes simulation of high-stakes performance contexts. Current knowledge regarding the relationship between errors in practice and later sport performance is based on novice performers and relatively simple, drill-like sport skills. The purpose of this study was to describe the relationship between errors produced during volleyball scrimmage in practice (low-stakes simulation) and in gameplay (high-stakes performance). These contexts represent ecologically valid, complex sport situations. Results indicated that players with high error scores during scrimmage were more likely to produce high error scores in gameplay, even when player skill, position and sex were accounted for.

The Empirical Validity of the Common European Framework of Reference Scales. An Exemplary Study for the Vocabulary and Fluency Scales in a Language Testing Context

In spite of the widespread use of Common European Framework of Reference for language learning, teaching, and assessment (CEFR) scales, there is an overwhelming lack of evidence regarding their power to describe empirical learner language (Fulcher 2004; Hulstijn 2007). This article presents results of a study that focused on the empirical robustness (i.e. the power of level descriptions to capture what learners actually do in a language test) of the CEFR vocabulary & fluency scales (A2-B2). Data stem from an Italian & German oral proficiency test (Abel et al. 2012). Results show that the empirical robustness was flawed: Some scale contents were hardly observable or so evenly distributed that they could not distinguish between learners. Contradictory/weak correlations among scale features and heterogeneous cluster solutions suggest that scales did not consistently capture typical learner behaviour. Often, learner language could not be objectively described by any level description. Also, it was only partially possible to link scale contents to research-based measures of fluency and vocabulary. Given the importance of CEFR levels in many high-stakes contexts, the results suggest the need of a large empirical validation project.

Adaptive Appraisals of Anxiety Moderate the Association between Cortisol Reactivity and Performance in Salary Negotiations.

Prior research suggests that stress can be harmful in high-stakes contexts such as negotiations. However, few studies actually measure stress physiologically during negotiations, nor do studies offer interventions to combat the potential negative effects of heightened physiological responses in negotiation contexts. In the current research, we offer evidence that the negative effects of cortisol increases on negotiation performance can be reduced through a reappraisal of anxiety manipulation. We experimentally induced adaptive appraisals by randomly assigning 97 male and female participants to receive either instructions to appraise their anxiety as beneficial to the negotiation or no specific instructions on how to appraise the situation. We also measured participants’ cortisol responses prior to and following the negotiation. Results revealed that cortisol increases were positively related to negotiation performance for participants who were told to view anxiety as beneficial, and not detrimental, for negotiation performance (appraisal condition). In contrast, cortisol increases were negatively related to negotiation performance for participants given no instructions on appraising their anxiety (control condition). These findings offer a means through which to combat the potentially deleterious effects of heightened cortisol reactivity on negotiation outcomes.

Harking, Sharking, and Tharking

In this editorial we discuss the problems associated with HARKing (Hypothesizing After Results Are Known) and draw a distinction between Sharking (Secretly HARKing in the Introduction section) and Tharking (Transparently HARKing in the Discussion section). Although there is never any justification for the process of Sharking, we argue that Tharking can promote the effectiveness and efficiency of both scientific inquiry and cumulative knowledge creation. We argue that the discussion sections of all empirical papers should include a subsection that reports post hoc exploratory data analysis. We explain how authors, reviewers, and editors can best leverage post hoc analyses in the spirit of scientific discovery in a way that does not bias parameter estimates and recognizes the lack of definitiveness associated with any single study or any single replication. We also discuss why the failure to Thark in high-stakes contexts where data is scarce and costly may also be unethical.

Measuring non-cognitive predictors in high-stakes contexts: The effect of self-presentation on self-report instruments used in admission to higher education

Non-cognitive constructs such as personality traits and behavioral tendencies show predictive validity for academic performance and incremental validity over and above cognitive constructs. Therefore, non-cognitive predictors are increasingly used in admission procedures for higher education, typically measured using-self-report instruments. It is well known that self-report instruments are sensitive to self-presentation, especially in high-stakes contexts. However, remarkably few studies investigated the effect of self-presentation on predictive validity. The effect of self-presentation in applicants to an undergraduate psychology program was studied using a repeated measures design. Respondents completed self-report questionnaires measuring non-cognitive predictors of academic performance before admission to the program, and again after admission. Scores were compared between contexts, as well as predictive validity, incremental validity, and potential hiring decisions. Results showed differences in scores between contexts on all scales, attenuated predictive validity for most scales, attenuated incremental validity when scores obtained in the admission context were used, and effects on admission decisions. In conclusion, validity results based on scores measured in low-stakes contexts cannot simply be generalized to high-stakes contexts. Furthermore, results obtained in a high-stakes context may result in self-presentation irrespective of whether participants are informed that their scores are used for selection decisions or not.

We Are What We Repeatedly Do—But in What Context? The Role of Situational Factors in Assessing Personality via Nonverbal Behavior

AbstractWhile the vast majority of assessments designed to measure noncognitive skills rely heavily on self‐report, concerns surrounding faking and response bias have led to an increased demand for the development of new and innovative methods by which to measure such traits, particularly in high stakes contexts. The focus of this report is to address one such method—the inference of noncognitive skills, namely personality, through nonverbal behavior—and the role contextual factors may play in influencing the validity of conclusions drawn from its application. Specifically, this report discusses the various ways in which characteristics of the task (e.g., level of cooperation, physical effort) may facilitate or hinder the assessment of personality when personality is being inferred through observable behavior. This discussion is dovetailed by a review of research from the nonverbal assessment and interpersonal task literatures, as well as a synthesis of these literatures to highlight the degree to which situational factors (as manifested through different tasks) influence the assessment of personality via nonverbal behavior. Additionally, the potential value of using noninvasive tools, such as the sociometer, to collect nonverbal behavioral data during these tasks is discussed. The report concludes with a brief commentary on the feasibility of using nonverbal methods to assess noncognitive skills, with a specific focus on the extent to which structuring the context to elicit certain behaviors may influence the validity and robustness of nonverbal responses as a measurement source.

Study Protocol on Intentional Distortion in Personality Assessment: Relationship with Test Format, Culture, and Cognitive Ability

Self-report personality questionnaires, traditionally offered in a graded-scale format, are widely used in high-stakes contexts such as job selection. However, job applicants may intentionally distort their answers when filling in these questionnaires, undermining the validity of the test results. Forced-choice questionnaires are allegedly more resistant to intentional distortion compared to graded-scale questionnaires, but they generate ipsative data. Ipsativity violates the assumptions of classical test theory, distorting the reliability and construct validity of the scales, and producing interdependencies among the scores. This limitation is overcome in the current study by using the recently developed Thurstonian item response theory model. As online testing in job selection contexts is increasing, the focus will be on the impact of intentional distortion on personality questionnaire data collected online. The present study intends to examine the effect of three different variables on intentional distortion: (a) test format (graded-scale versus forced-choice); (b) culture, as data will be collected in three countries differing in their attitudes toward intentional distortion (the United Kingdom, Serbia, and Turkey); and (c) cognitive ability, as a possible predictor of the ability to choose the more desirable responses. Furthermore, we aim to integrate the findings using a comprehensive model of intentional distortion. In the Anticipated Results section, three main aspects are considered: (a) the limitations of the manipulation, theoretical approach, and analyses employed; (b) practical implications for job selection and for personality assessment in a broader sense; and (c) suggestions for further research.

The Consistencies and Vagaries of the Washington State Inventory of Evidence-Based Practice: The Definition of "Evidence-Based" in a Policy Context.

As states increasingly establish the importance of evidence-based practice through policy and funding mandates, the definition of evidence-based practice can have a significant impact on investment decisions. Not meeting established criteria can mean a loss of funding for established programs and the implementation disruption of programs without a strong research base. Whether the definition of "evidence-based" is influenced by these high stakes contexts is an interesting question that can inform the larger field about the value and utility of evidence-based practice lists/inventories for disseminating knowledge. In this paper we review the development of the Washington State Inventory of Evidence-Based, Research-Based and Promising Practices as a case study for the process of defining evidence-based practice in a policy context. As part of this study we also present a comparison of other well-known evidence-based practice inventories and examine consistencies and differences in the process of identifying and developing program ratings.

Taking and teaching the test are not the same: a case study of first-year teachers’ experiences in high-stakes contexts

Background: Policymakers’ use of high-stakes exams to improve students’ academic achievement affects teachers and their tenure in the field at all levels of schooling. Novice teachers now being inducted into the field have been educated almost exclusively in these high-stakes learning environments. Yet, how their familiarity with these contexts combined with their experiences in their own classrooms affect novices’ induction into the field of teaching has not been fully examined. Aim: This article presents findings from an investigation into the experiences of two first-year teachers who were educated and trained to be teachers in the same high-stakes education system in which they taught. It examines how these first-year teachers viewed policymakers’ reforms affecting their teaching and tenure in the field. Methods: This qualitative case study centers on the experiences of two first-year teachers working in the same high-stakes standards-based accountability teaching context in which they were educated – the case. This study provides insight into the issue of how novices’ familiarity with high-stakes reform combined with their experiences in their own classrooms impacts their conceptions of their teaching and their tenure in the field. Outcomes: The findings from this case study reveal how policymakers’ high-stakes reforms impacted the development of these novice teachers in significant ways. Not only did they have to learn how to teach as they taught, but they also had to ensure they were teaching all of their students to pass the high-stakes exams. Their varied experiences also demonstrate how these high-stakes exams can ‘test’ beginning teachers out of the classroom. Conclusions: These findings demonstrate that first-year teachers’ familiarity with policymakers’ high-stakes reforms is not enough to prepare them for the expectation that they immediately improve students’ academic achievement on high-stakes exams. Such findings not only challenge what it means to be a educator in these contexts, but they also shed light on how larger political and economic forces impact the teaching and tenure of novices. To support new teachers, teacher educators and mentors should rethink the education and induction processes while helping novices understand as well as prepare for the role context plays in their teaching and development as professionals.

Figuring Out How To Be a Teacher in a High-Stakes Context: A Case Study of First-Year Teachers’ Conceptual and Practical Development

High-stakes education reforms across the United States and the globe continue to alter the landscape of teaching and teacher education. One key but understudied aspect of this reform process is the experiences of first-year teachers, particularly those who participated in these high-stakes education systems as students and as a teachers-in-training. This article employs Holland, Lachicotte, Skinner, and Cain’s conception of figured worlds to understand how such reforms affected two first-year teachers’ conceptions of themselves as educators and what it is they were to teach their students. It concludes with suggestions for teacher–educators and mentors to help novice and preservice teachers succeed in these high-stakes contexts so that they can work toward becoming educators who attend to their students’ individual needs and teach them how to be active and reflective citizens.

Selfish play increases during high-stakes NBA games and is rewarded with more lucrative contracts.

High-stakes team competitions can present a social dilemma in which participants must choose between concentrating on their personal performance and assisting teammates as a means of achieving group objectives. We find that despite the seemingly strong group incentive to win the NBA title, cooperative play actually diminishes during playoff games, negatively affecting team performance. Thus team cooperation decreases in the very high stakes contexts in which it is most important to perform well together. Highlighting the mixed incentives that underlie selfish play, personal scoring is rewarded with more lucrative future contracts, whereas assisting teammates to score is associated with reduced pay due to lost opportunities for personal scoring. A combination of misaligned incentives and psychological biases in performance evaluation bring out the “I” in “team” when cooperation is most critical.

Continuous assessment frameworks within university English Pathway Programs: realizing formative assessment within high-stakes contexts

Universities have become increasingly reliant upon English Pathway Programs (EPPs) to expand enrolments of international students who otherwise fail to satisfy standard entry requirements, as determined by standardized tests of language proficiency such as IELTS and TOEFL. EPPs provide foundation programs for mainstream university courses, with particular attention to the language skills required for academic study. This article contributes to the work on alternative assessment within higher education by focusing on the use of classroom-based assessment within such programs, and the formative potential of continuous assessment frameworks in the context of higher education. We report on an Australian case study of teacher practice within one EPP, in which ongoing classroom-based assessment contributed towards 70% of the students' final score. We argue that there is significant value for using continuous, classroom-based assessment, but the high-stakes nature of higher education restricts how its formative potential is realized in practice. We identify implications for promoting more positive learning gains, including the need to re-balance assessment tasks within course structures and the value of greater teacher autonomy.

Examining the factor structure of the Family Child Care Environment Rating Scale—Revised

The purpose of this study was to examine the factor structure of the Family Child Care Environment Rating Scale—Revised (FCCERS-R) in high-stakes contexts. The results of an exploratory factor analysis revealed three dimensions of quality on the FCCERS-R: (1) Activities/Materials, (2) Language/Interaction, and (3) Organization. This study also explored whether abridged versions of the FCCERS-R could serve as a proxy for the full instrument. In addition to subsets of FCCERS-R items created from the factor structure, purposively and randomly chosen item subsets were created. The purposively chosen subsets included 6-, 9-, and 12-item scales comprised of the items with the highest factor loading across the three factors, whereas the randomly chosen subsets consisted of 12 items. Results of a discriminant analysis showed that the factor subsets were poorer proxies for the total FCCERS-R score than were the other subsets, which demonstrated comparable internal consistencies and discriminant power as the full FCCERS-R when classifying homes into general quality categories. Implications for adopting shorter versions of the FCCERS-R are discussed.

On the reliability of high-stakes teacher assessment

For a number of reasons, increasing reliance is being placed on teacher assessment in high-stakes contexts in many countries around the world. Simultaneously, countries that have for some time relied to greater or lesser degrees on teacher assessment for high-stakes purposes are in the process of questioning the validity of that reliance. In principle, teacher assessment has an important role to play in increasing assessment validity by complementing testing to cover subject domains more comprehensively than otherwise would be possible. But what is the evidence regarding the reliability of teacher assessment in high-stakes contexts? The answer is that the evidence is limited and often ambiguous. Research has revealed that teachers can be influenced by a number of construct-irrelevant factors as they work towards their judgements, factors such as gender, socio-economic background, effort and behaviour, that risk biasing their assessments. And when considering construct-relevant achievement evidence teachers are often expected to use verbal or semi-verbal sets of criteria, such as level descriptions, which typically require a degree of subjective interpretation in application and so are themselves a source of unwanted variation in judging standards. Arguably the most effective strategy for addressing these issues is participation in consensus moderation. Yet there have been few attempts to provide evidence of the effectiveness of moderation in practice. The potential value of, and the growing reliance upon, teacher assessment in high-stakes applications demand that evaluation of consensus moderation become a built-in part of the process.