Evaluative Judgments Research Articles

Shifting power in evaluation: Lessons from child-led evaluations

Background: In 2015, Plan International UK undertook a bold experiment: enabling children in participating in the multi-sectoral programme Building Skills for Life to evaluate the programme.Objectives: The primary objective of this experiment was to assess if a child-led evaluation is feasible, valuable and desirable. Feasible, in consideration of children’s abilities and the intricacies of a multisectoral evaluation; valuable in comparison with expert-led evaluations and desirable in relation to the evidence already available.Method: These experiments used a range of methodologies to facilitate children’s collecting and analysing data to return full evaluative judgements. While these experiments were deemed successful and credible on account of the reviews and support received by the monitoring and evaluation (ME) community and the donor, the years that followed did not see child-led monitoring and evaluation flourish across the international development sector, despite renewed interest and international commitments, such as the Convention on the Rights of the Child.Results: This article explores the contribution these experiences can bring to today’s evaluation practice and argues that child participation in monitoring and evaluation is not simply desirable; it is a right and an opportunity to sharpen the objectives of programmes addressed to children.Conclusion: This article concludes that it is time to abandon ME practices carried out on children particularly in child-focused programmes and insist on ME to be, at the very least, carried out with children, if not, as is preferable, by children.Contribution: This article highlights that involving children in social development aimed at changing the societies in which they will grow up and live, is not a matter of good practice or inclusion, but a matter of justice.

SYMPOSIUM REPORT MALAYSIAN ASSOCIATION OF APPLIED LINGUISTICS (MAAL) SYMPOSIUM ON “PROMOTING INNOVATIVE LEARNING-ORIENTED FEEDBACK PRACTICE”

The authors had the privilege of co-chairing the symposium titled “Promoting Innovative Learning-Oriented Feedback Practice”, which took place online, hosted by the Malaysian Association of Applied Linguistics (MAAL) on 19th August 2023. The event featured distinguished speakers including Lawrence Jun Zhang, a Professor from the University of Auckland; Junjie Gavin Wu, a Lecturer from Macao Polytechnic University; Kayatri Vasu, an Assistant Professor from Universiti Tunku Abdul Rahman; Orlando Xiao Xie, a PhD Candidate from Universiti Putra Malaysia; Wei Wei, an Associate Professor from Macao Polytechnic University; and Greg Parry, the Co-Founder and CEO of Global Services in Education. With a virtual audience of over 100 participants, the symposium delved into various captivating topics. Lawrence Jun Zhang shared insights on written corrective feedback (WCF) and metacognition, while Junjie Gavin Wu explored the intersection of feedback and technology. Kayatri Vasu’s presentation revolved around self-regulated learning (SRL), whereas Orlando Xiao Xie shed light on evaluative judgement. Wei Wei delved into the realm of mobile applications in vocabulary learning, and Greg Parry discussed authentic quality standards. The symposium provided a platform for dynamic discussions and valuable knowledge exchange, advancing the discourse on effective feedback practices and their impact on learning.

Causal mapping for evaluators

Evaluators are interested in capturing how things causally influence one another. They are also interested in capturing how stakeholders think things causally influence one another. Causal mapping – the collection, coding and visualisation of interconnected causal claims – has been used widely for several decades across many disciplines for this purpose. It makes the provenance or source of such claims explicit and provides tools for gathering and dealing with this kind of data and for managing its Janus-like double-life: on the one hand, providing information about what people believe causes what, and on the other hand, preparing this information for possible evaluative judgements about what causes what. Specific reference to causal mapping in the evaluation literature is sparse, which we aim to redress here. In particular, the authors address the Janus dilemma by suggesting that causal maps can be understood neither as models of beliefs about causal pathways nor as models of causal pathways per se but as repositories of evidence for those pathways.

An Exploratory Study of EFL Learners’ Use of ChatGPT for Language Learning Tasks: Experience and Perceptions

ChatGPT, a general-purpose intelligent chatbot developed by OpenAI, has introduced numerous opportunities and challenges in the field of language education. With its remarkable ability to generate diverse forms of text, answer questions, and provide translations within minutes, ChatGPT has become an influential tool in the era of advanced AI technology. However, to what extent ChatGPT can be used to assist students in completing language learning tasks remains largely unexplored. Against this background, this study aimed to investigate students’ experiences with ChatGPT and their perceptions of its role in language learning through a small-scale qualitative study. The data were collected through semi-structured interviews with five students at a top-tier international university in China. Students’ responses revealed that ChatGPT has the potential to serve as a valuable learning partner and aid students in completing language-related tasks. Furthermore, participants exhibited critical judgment in evaluating the quality of ideas and outputs generated by ChatGPT, as well as the ability to modify prompts to maximize learning benefits. Such critical judgment offsets the potential threats to academic integrity posed by ChatGPT. Our findings contribute to the understanding of the potential of ChatGPT in language education by adding empirical evidence from students’ perspectives. This study supports the idea that ChatGPT can work as an effective tool for providing students with immediate feedback and personalized learning experiences. Such findings generate implications for future pedagogical practices in the new era by providing students with personalized guidance, designing technology-embedded language support, and developing students’ lifelong learning skills (e.g., autonomy and evaluative judgment) with the support of ChatGPT.

Fluent Processing Amplifies Affective Judgments

Abstract: The importance of processing fluency in evaluative judgments has been repeatedly demonstrated across many domains such as liking, beauty, and truth. However, a clear picture of the nature of processing fluency has yet to emerge. Fluent processing has been suggested to form evaluative judgments in a hedonic nature, in which existing judgmental tendencies always shift in a positive direction. Alternatively, fluency has been proposed to amplify evaluative judgments bidirectionally. However, uncertainty remains regarding the influence of processing fluency on pre-existing judgmental tendencies. Specifically, the extent to which the effect of stimuli belonging to specific categories varies within an individual remains unclear. This study assessed the influence of fluent processing on two specific categories (cats/spiders) using a visual search task. Fluency was manipulated by the set size of the stimuli and presentation duration. Fluency intensified pre-existing judgmental tendencies in two divergent directions: The initially favored stimuli were liked more, while the initially unfavored ones were liked less when the processing of stimuli was fluent. There was a significant correlation between favored and unfavored stimuli in terms of the magnitude of the effect, and such effect was influenced by visual attention, suggesting that processing fluency goes beyond a hedonic and unidimensional nature.

“Touching the Author's Mind”: Judgment and Intention in Jasper Heywood's Translations of Seneca

This essay draws on the prefatory material in Jasper Heywood's translations of three Senecan tragedies—Troas (1559), Thyestes (1560), and Hercules Furens (1561)—to show that a focus on judgment helps to expound Heywood's theory of translation. The judgments envisioned in these prefatory texts, namely, the evaluative judgments of others and the philological judgments required of the translator, highlight Heywood's understanding of Seneca's original intention as the only true measure of the English translations. For Heywood, there can be no “intentional fallacy” because intention is the only remedy against the difficulties of translation (complexity of the Latin original, deficient sources, unreliability of the printed text). Ultimately Heywood's efforts to approximate Seneca's original “sense,” culminating in the creation of a fictional Seneca, endow the translator with an authorial intention.

Semantic Differential of the Concept of Beauty

This study focuses on an exploration of the concept of beauty using the semantic differential method. 2,080 participants (53.46% were female) aged 18 to 89 (M = 41.36 years; SD = 16.48) were asked to evaluate the concept of beauty using twenty-one bipolar adjectives across three semantic differential dimensions – activity, evaluation, and potency. The results indicate that the most descriptive adjectives used to describe beauty are: pleasant, inviting, good, inspiring, pure, and kind, indicating that the concept of beauty is mostly associated with positive connotations. Furthermore, the majority of the characteristic adjectives are related to the evaluation dimension, in which statistically significant differences were identified in the evaluations made by men and women. Women perceive beauty as a concept that is significantly more strongly associated with evaluative judgments than men (p <.01). The potential of the semantic differential method can be used to compare the meaning of the concept of beauty with synonymous and oppositional concepts.

Why do evaluative judgments affect emotion attributions? The roles of judgments about fittingness and the true self

Past research has found that the value of a person's activities can affect observers' judgments about whether that person is experiencing certain emotions (e.g., people consider morally good agents happier than morally bad agents). One proposed explanation for this effect is that emotion attributions are influenced by judgments about fittingness (whether the emotion is merited). Another hypothesis is that emotion attributions are influenced by judgments about the agent's true self (whether the emotion reflects how the agent feels “deep down”). We tested these hypotheses in six studies. After finding that people think a wide range of emotions can be fitting and reflect a person's true self (Study 1), we tested the predictions of these two hypotheses for attributions of happiness, love, sadness, and hatred. We manipulated the emotions' fittingness (Studies 2a-b and 4) and whether the emotions reflected an agent's true self (Studies 3 and 5), measuring emotion attributions as well as fittingness judgments and true self judgments. The fittingness manipulation only impacted emotion attributions in the cases where it also impacted true self judgments, whereas the true self manipulation impacted emotion attribution in all cases, including those where it did not impact fittingness judgments. These results cast serious doubt on the fittingness hypothesis and offer some support for the true self hypothesis, which could be developed further in future work.

Middleness: Provincial Fiction and the Aesthetics of Dull Life

Abstract This article argues that the rise of the provincial novel in Britain during the 1860s set loose a radically democratizing aesthetic of everyday discriminations. The article suggests – through examples from the works of Elizabeth Gaskell and George Eliot – that provincial fiction embodies an aesthetics of what is termed ‘middleness’: a means of reading in which the social, cultural and geographical middle became an object of careful evaluative judgment. The article examines the aesthetic affordances of this form in relation to Sianne Ngai’s exploration of the category of the ‘interesting’. It places this practice of attention to the common and the dull in contrast to Matthew Arnold’s contemporaneous critique of provincialism as the chief weakness at the heart of English culture as it moved towards democracy. If Arnold’s mission in his critical essays was to establish a global standard of taste – collective agreement grounded in the judgement of what was ‘the best that has been thought and known in the world’ – then provincial fiction effected another sort of aesthetic community: one that valued the common and the dull.

A noise audit of the peer review of a scientific article: a WPOM journal case study

This study aims to be one of the first to analyse the noise level in the peer review process of scientific articles. Noise is defined as the undesired variability in the judgements made by professionals on the same topic or subject. We refer to evaluative judgements in which experts are expected to agree. This is what happens when we try to judge the quality of a scientific work. To measure noise, the only information needed is to have several judgements made by different people on the same case to analyse their dispersion (what Kahneman et al. call a noise audit). This was the procedure followed in this research. We asked a set of reviewers from the journal WPOM (Working Papers on Operations Management) to review the same manuscript which had been previously accepted for publication in this journal, although the reviewers were unaware of that fact. The results indicated that if two reviewers were used, the probability of this manuscript not being published would be close to 8%, while the probability of it having an uncertain future would be 40% (one favorable opinion and one unfavorable opinion or both suggesting substantial changes). In the case of employing only one reviewer, in 25% of the cases, the audited work would have encountered significant challenges for publication. The great advantage of measuring noise is, once measured, it is usually possible to reduce it. This article concludes by outlining some of the measures which can be put in place by scientific journals to improve their peer review processes.

Neuroleadership: Affective experiences in the workplace and their influence on employees’ evaluative judgements

Neuroscience-based approaches have had little presence in studies in the field of hotel firms. The literature indicates that neuroleadership behaviors can change the employees’ perceptions of their work environment. The purpose of this study is to analyze how affective events in the workplace influence employees' evaluative judgements of certain organizational factors. The paper presents the results of a six-month study involving 150 lower-level employees and 25 middle managers of a hotel tourism chain. The study was carried out by a staff member of the company in question, a characteristic that is beyond the scope of most research. The results show that the affective events significantly improve employees' evaluative judgements of different organizational variables, highlighting the importance of neuroleadership skills in managing emotions and improving employee well-being.

How Do Social Media Machines Affect Self-Concept Research?

Advanced digital technologies broadly penetrate self-activities, such as algorithms, machine learning, or artificial intelligence. This trend is most evident on social media, where contents, attitudes and evaluative judgments meet on technology-driven platforms. Moreover, human networks also started communicating with social bots or conversational interfaces. All these challenges can trigger a redesign of self-concept via technology. Therefore, the paper investigates how social media machines affect self-concept-related academic research. First, pioneers of the field are presented. Second, the self-concept research in digital technology and social media is summarised. Topic networks illustrate critical research fields with the latest trends and future implications. Last but not least, we also investigate how emerging media phenomena affect academic trends in the case of social bots or fake news. The study aims to support the connected research in psychology, business, management, education, political science, medicine and media studies with an understanding of the latest trends. The additional goal is to highlight the potential of market-based research cooperation with academia supporting significant developments and funding.

FUNCTIONAL AND PRAGMATIC FEATURES OF NICKNAMES (BASED ON THE MATERIAL OF THE KOLO WEBSITE)

Nickname is a tool of self-presentation of the user in Internet communication. A new virtual personality is created with the help of a network name and by the person who endows it with fictional, partially or fully strands, characteristics and features. The functional and pragmatic features of network names are considered in the article on the material of nicknames from the website of the KOLO operational assistance fund of the special service "Holiday KOLO", which are determined by the pragmatic needs of the author’s personality.
 The functional-pragmatic features of the website’s nicknames of aid funds and volunteer organizations are a special object of analysis, as they make it possible to trace the process of functioning and development of the language at the current stage of its existence. While creating network names, communicators prefer the communicatively significant meaning of "own" over "other". It was found that nicknames appear as markers of trust, confidence, enthusiasm, and are indicators of evaluative and emotional judgments.
 In the process of combining two components into one, the semantic and pragmatic potential of a nickname is formed. Two-component network names are singled out among the studied material, the main word of which is a noun, and the extender (micro-context) is an adjective. Falling into different micro-contexts, the same constituent has different pragmatic meanings. The lexical conjugation of the components of a nickname produces the appearance of functional-pragmatic nuances in the meaning. In the process of researching network names, it was found that the types of displacement of presuppositions in two-component nicknames are regular and related to thematic groups of supporting elements.
 It is substantiated that for the successful interpretation of network names, communicators must have a common base of non-linguistic knowledge. On the one hand, communication participants must understand the lexical meaning of individual components, on the other hand, they should interpret their common functional and pragmatic potential.

Toward the Deimplementation of Computed Tomography Urogram for Patients With Low- to Intermediate-risk Microscopic Hematuria: A Mixed-method Study of Factors Influencing Continued Use.

Citing high costs, limited diagnostic benefit, and ionizing radiation-associated risk from CT urogram, in 2020 the AUA revised its guidelines from recommending CT urogram for all patients with microscopic hematuria to a deintensified risk-stratified approach, including the deimplementation of low-value CT urogram (ie, not recommending CT urogram for patients with low- to intermediate-risk microscopic hematuria). Adherence to revised guidelines and reasons for continued low-value CT urogram are unknown. With the overarching objective of improving guideline implementation, we used a mixed-method convergent explanatory design with electronic health record data for a retrospective cohort at a single academic tertiary medical center in the southeastern United States and semistructured interviews with urology and nonurology providers to describe determinants of low-value CT urogram following guideline revision. Of 391 patients with microscopic hematuria, 198 (51%) had a low-value CT urogram (136 [69%] pre-guideline revision, 62 [31%] postrevision). The odds of ordering a low-value CT urogram were lower after guideline revisions, but the change was not statistically significant (OR: 0.44, P = .08); odds were 1.89 higher (P = .06) among nonurology providers than urology providers, but the difference was not statistically significant. Provider interviews suggested low-value CT urogram related to nonurology providers' limited awareness of revised guidelines, the role of clinical judgment in microscopic hematuria evaluation, and professional and patient influences. Our findings suggest low-value CT urogram deimplementation may be improved with guidelines and implementation support directed at both urology and nonurology providers and algorithms to support guideline-concordant microscopic hematuria evaluation approaches. Future studies should test these strategies.

청각장애 중〮고등학생의 미디어 이용과 교육 현황

Purpose: In this study, media use and media education status of Deaf and Hard of Hearing(DHH) adolescents were analyzed, and the results of the study are as follows. method: To achieve the purpose of the study, questionnaire items were developed and a questionnaire survey was conducted through literature analysis and expert consultation. Results: First, the most used social media by DHH adolescents was SNS, followed by video platform, messenger service, internet portal, online cafe, online blog, and metaverse platform. The reason for using social media was not only for conversation/communication, but also for the purpose of obtaining various contents or information/news, so the reasons for using social media were more diverse than that of hearing teenagers. As a result of examining applications for DHH adolescents, it was found that there is a lack of useful applications for DHH students as most of them have no experience in using them. Second, the contents of media education received at school accounted for a high proportion of online etiquette(netiquette) education, online personal information protection methods, information search, video filming and editing, followed by fact-checking and critical criticism. Evaluation and fairness judgment of news were relatively low. The path of learning to use media was in the order of self-practice, school teacher, friend or senior, parents, and out-of-school teachers such as media education instructors. This suggests that media education is not being carried out systematically and efficiently for students with hearing impairments in schools.

Young adults’ job satisfaction in Poland and the Czech Republic: a comparative analysis

Motivation: Job satisfaction as an attitude that reflects an evaluation judgment of the various aspects of a job is an essential factor influencing employees as well as employers. The well-being of individuals, activity in the labor market, productivity, and turnover are examples of effects of job satisfaction. The inclusion of the circumstances of Poland and the Czech Republic countries and additionally young generations (GenY and GenZ) context enables the formulation of practical implications for increasing job satisfaction for both employers and employees in CEE countries.Aim: The research aims to identify factors that shape job satisfaction patterns among young adults, including age as an indicator variable in Poland and the Czech Republic. The identification is based on the two separate ordered logit models estimated in the group of young adults ages 18–29. The models were built upon the data collected in 2021 in a survey using the CAWI technique on a sample of 304 respondents in Poland and 259 in the Czech Republic.Result: The results of the logit models estimation indicate that job satisfaction in Poland and the Czech Republic is determined by various sets of factors considered in four dimensions, i.e., behavioral, economic, educational, and socio-demographic. Substantial differences between young adults in both countries were recognized. The greatest impact on job satisfaction in Poland has the willingness to control expenses and wage satisfaction. In the Czech Republic, job satisfaction is most strongly influenced by the imposition of rules that discipline expenditure and wage satisfaction.

Children's evaluative judgements of conformers and nonconformers

AbstractThe present research examined children's generalisations about people based on their decisions to conform to descriptive norms. This issue was examined in South Korea, where conformity tends to be highly valued. Younger (4–6 years) and older (9–11 years) children (total N = 197) were asked to evaluate the personal qualities of conformers and nonconformers. Results showed that children in both age groups rated nonconformers more negatively than conformers in terms of moral character, intelligence, popularity, and likeability. Ratings of conformers showed minimal age‐related change. In contrast, ratings of nonconformers were more positive with age, except for popularity ratings. This work sheds light on how children use information about conformity to make specific inferences about what individuals are generally like.

The development and validation of an instrument to measure evaluative judgement: a focus on engineering students’ judgement of intercultural competence

Intensive research attention has focused on developing students’ evaluative judgement within the higher education curriculum, but little has addressed how it can be measured. The contextual nature of evaluative judgement makes it difficult to generate an encompassing instrument, highlighting the need for situated measurement tools. Against this background, this study develops and validates a self-report instrument to measure engineering students’ evaluative judgement of intercultural competence. A total of 815 Chinese engineering students participated in the pilot and formal phases of validation. Following sequential exploratory-confirmatory factor analyses, the final instrument contains 27 items covering six dimensions. Validity and reliability evidence supports the correlated six-factor model and a higher-order factor model. The study provides educators and policymakers with a useful tool to understand students’ development in their evaluative judgement of intercultural competence. More importantly, for students themselves, the instrument facilitates self-reflection and helps them achieve greater autonomy in directing their development and growth. Although the study adopts a situated approach to measure evaluative judgement, we also discuss the possibility of adapting this instrument for other judgement contexts.

The Problem with the Anthropocene: Kainos, Not Anthropos

The Problem with the Anthropocene: <i>Kainos</i>, Not <i>Anthropos</i>

Teaching Presence in Asynchronous Online Classes: It’s Not Just a Façade

The expanding scale and scope of online education options, both in terms of design and delivery, creates significant questions that increasingly warrant research attention. This empirical study investigates students’ perceptions and evaluative judgments of a number of methods of setting and sustaining Instructor Presence in an online asynchronous course. Based on factor analysis of our data, we propose refining the idea of Instructor Presence in terms of stylistic versus substantive methods. Tests of student survey data indicate that, while students see value in both types of Instructor Presence, they perceive significantly greater benefit from substantive relative to stylistic methods.