Evaluative Judgments Research Articles

Learning to work with the black box: Pedagogy for a world with artificial intelligence

AbstractArtificial intelligence (AI) is increasingly integrating into our society. University education needs to maintain its relevance in an AI‐mediated world, but the higher education sector is only beginning to engage deeply with the implications of AI within society. We define AI according to a relational epistemology, where, in the context of a particular interaction, a computational artefact provides a judgement about an optimal course of action and that this judgement cannot be traced. Therefore, by definition, AI must always act as a ‘black box’. Rather than seeking to explain ‘black boxes’, we argue that a pedagogy for an AI‐mediated world involves learning to work with opaque, partial and ambiguous situations, which reflect the entangled relationships between people and technologies. Such a pedagogy asks learners locate AI as socially bounded, where AI is always understood within the contexts of its use. We outline two particular approaches to achieve this: (a) orienting students to quality standards that surround AIs, what might be called the tacit and explicit ‘rules of the game’; and (b) providing meaningful interactions with AI systems. Practitioner notesWhat is already known about this topic Artificial intelligence (AI) is conceptualised in many different ways but is rarely defined in the higher education literature. Experts have outlined a range of graduate capabilities for working in a world of AI such as teamwork or ethical thinking. The higher education literature outlines an imperative need to respond to AI, as underlined by recent commentary on ChatGPT. What this paper adds A definition of an AI that is relational: A particular interaction where a computational artefact provides a judgement about an optimal course of action, which cannot be easily traced. Focusing on working with AI black boxes rather than trying to see inside the technology. Describing a pedagogy for an AI‐mediated world that promotes working in complex situations with partial and indeterminate information. Implications for practice and/or policy Focusing on quality standards helps learners understand the social regulating boundaries around AI. Promoting learner interactions with AI as part of a sociotechnical ensemble helps build evaluative judgement in weighting AI's contribution to work. Asking learners to work with AI systems prompts understanding of the evaluative, ethical and practical necessities of working with a black box.

Key capabilities for promoting student feedback literacy

AbstractFeedback literacy in higher education has come as a reaction to the ineffective focus on teacher's delivery of feedback and its passive reception by students rather than on students' active participation in receiving feedback and constructing it (Nicol & Macfarlane-dick, 2006; Nicol, 2019). Therefore, the aim of this theoretical paper is to identify students' capabilities of feedback literacy. To meet this objective, three questions are targeted: 1) How is feedback literacy defined? 2) How is it characterized? 3) What are its practical dimensions from student's perspective? Based on the analysis of theoretical and empirical studies retrieved via reliable databases, a thorough description of the existing definitions of feedback literacy and its frameworks (Carless & Boud, 2018; Carless & Winstone, 2020; Molloy, Boud, & Henderson, 2020; Sutton, 2012) can be highlighted. Four dimensions of student feedback literacy and their adaptation to the writing context are also targeted: appreciating feedback, making evaluative judgments, managing affect, and acting upon feedback (Carless & Boud, 2018). This paper is relevant because it addresses the shift from feedback as a one-way process of transmitting information from instructors to feedback as a process of using diverse communication channels, through which teachers and students collaborate to improve learning outcomes.

Factor analysis of causes of corruption risks in the educational and scientific sphere of Ukraine

In recent years, the attention of the world scientific community has been drawn to the search and solution of problems related to overcoming corruption phenomena in all spheres of public life. Corruption as a socio-economic, ethical, moral phenomenon and philosophical-legal construct is studied at the interdisciplinary level, because its origins are primarily in the motivated behavior of an active and integral participant and founder of public and state institutions – a person. Our research deals with the study of the sources of corruption risks in the educational-scientific process at the institutional level of the education and science system of Ukraine: school, institute, university. We conducted a social survey to establish personal attitudes to corruption actions by officials and participants of the educational process, awareness of key concepts and principles of academic integrity, providing a subjective assessment on the effectiveness of functioning at the level of the respondent’s institution of the internal anti-corruption system. Particular attention was paid to establishing the content of the subjective-evaluative opinion of the respondents on the causes and existence of corruption in educational institutions of Ukraine. The aggregation of individual factors, which are segmented by certain areas of professional and life activity of the respondents: financial-economic, professional-career, socio-political, educational-preventive was carried out through the use of factor analysis. This allowed to reduce the dimension of the initial space of the studied causes and symptoms and combine them according to the evaluative judgments of the respondents into certain leading factors that characterize the common source of arising corruption risks in the educational-scientific process at the institutional level of the system of education and science of Ukraine.

Towards Informative and Diverse Dialogue Systems Over Hierarchical Crowd Intelligence Knowledge Graph

Knowledge-enhanced dialogue systems aim at generating factually correct and coherent responses by reasoning over knowledge sources, which is a promising research trend. The truly harmonious human-agent dialogue systems need to conduct engaging conversations from three aspects as humans, namely (1) stating factual contents (e.g., records in Wikipedia), (2) conveying subjective and informative opinions about objects (e.g., user discussions on Twitter), and (3) impressing interlocutors with diverse expression styles (e.g., personalized expression habits). The existing knowledge base is a standardized and unified coding for factual knowledge, which could not portray the other two kinds of knowledge to make responses more informative and expressive diverse. To address this, we present CrowdDialog , a crowd intelligence knowledge-enhanced dialogue system, which takes advantage of “crowd intelligence knowledge” extracted from social media (with rich subjective descriptions and diversified expression styles) to promote the performance of dialogue systems. Firstly, to thoroughly mine and organize the crowd intelligence knowledge underlying large-scale and unstructured online contents, we elaborately design the C rowd I ntelligence K nowledge G raph ( CIKG ) structure, including the domain commonsense subgraph, descriptive subgraph, and expressive subgraph. Secondly, to reasonably integrate heterogeneous crowd intelligence knowledge into responses while ensuring logicality and fluency, we propose the G ated F usion with D ynamic Knowledge- D ependent ( GFDD ) model, which generates responses from the semantic and syntactic perspective with the context-aware knowledge gate and dynamic knowledge decoding. Finally, extensive experiments over both Chinese and English dialogue datasets demonstrate that our approach GFDD outperforms competitive baselines in terms of both automatic evaluation and human judgments. Besides, ablation studies indicate that the proposed CIKG has the potential to promote dialogue systems to generate fluent, informative, and diverse dialogue responses.

Ценностно-мировоззренческие основы восприятия молодежью современного российского государства в условиях глобальной нестабильности

The article is devoted to the study of the worldview of modern youth as a key element of perception, setting the framework for perceiving reality, through the prism of which there is a perception and understanding of what is happening and acting as the ideological and semantic core of the image of the state. The authors consider the worldview in the system (triad) of the concepts of «worldview - image – perception», which make up the conceptual scheme of the article and are combined by a single perceptual mechanism. The authors focus on the dynamic nature of the phenomenon under study, the factors of its formation and modification in conditions of global instability, proxy wars, modern challenges and threats. The purpose of this article was to study the system of ideas, meanings and values, evaluative judgments of youth regarding the modern Russian state, the mechanism and factors of their formation in a spatial-temporal dimension. The conclusions of the article are based on the materials of three research sections (2019, 2021 and 2022), revealing the structural and meaningful features of the worldview of youth, the dynamics and trends of its development at the current stage. Through a cross-sectional analysis of survey data and an associative test, a generalized image of the modern Russian state, its value-worldview core, characteristics of social representations, as well as the general estimated background of perception were revealed. The political worldview of modern youth is characterized by a general positive orientation, pragmatism, sobriety and adequacy of assessments of the situation, risks and challenges, a vision of urgent problems, a rational and critical perception of the modern state and at the same time a mismatch of its elements, leading to a violation of the integrity of the worldview.

The Value of Unity

The Value of Unity

Incipient Fault Detection and Reconstruction Using an Adaptive Sliding-Mode Observer for the Actuators of Fixed-Wing Aircraft

This paper proposes a new method of detection and reconstruction of the incipient fault of fixed-wing actuators based on an adaptive sliding-mode observer. First, a mathematical model of a fixed-wing aircraft is derived under certain assumptions, considering nonlinear terms and system disturbances. Second, by introducing a nonsingular coordinate transformation, the incipient faults are separated from the disturbances. For a subsystem with no disturbances, the Luenberger observer can estimate the incipient fault. For a subsystem with disturbances, the sliding-mode observer is robust against these disturbances. The Lyapunov stability theory guarantees dynamic error convergence and system stability. The evaluation function was designed to realize residual evaluation and threshold judgment. Third, based on the concept of equivalent output injection, an adaptive sliding-mode observer method is proposed to reconstruct actuator faults precisely under the condition of the uncertain system structure. The design steps of the proposed reconstruction method are introduced in the form of a linear matrix inequality problem, which provides an effective method for calculating the design parameters. Finally, the simulation results of the De Havilland DHC-2 “Beaver” aircraft demonstrate the correctness and effectiveness of the proposed method.

The role of feedback in supporting trainees who underperform in clinical environments.

Underperformance in clinical environments can be costly and emotional for all stakeholders. Feedback is an important pedagogical strategy for working with underperformance - both formal and informal strategies can make a difference. Feedback is a typical feature of remediation programs, and yet there is little consensus on how feedback should unfold in the context of underperformance. This narrative review synthesises literature at the intersections of feedback and underperformance in clinical environments where service, learning and safety need to be considered. We do so with a critical eye towards generating insights for working with underperformance in the clinical environment. There are compounding and multi-level factors that contribute to underperformance and subsequent failure. This complexity overwrites simplistic notions of 'earned' failure through individual traits and deficit. Working with such complexity requires feedback that goes beyond educator input or 'telling'. When we shift beyond feedback as input to process, we recognise that these processes are fundamentally relational, where trust and safety are necessary for trainees to share their weaknesses and doubts. Emotions are always present and they signal action. Feedback literacy might help us consider how to engage trainees with feedback so that they take an active (autonomous) role in developing their evaluative judgements. Finally, feedback cultures can be influential and take effort to shift if at all. A key mechanism running through all these considerations of feedback is enabling internal motivation, and creating conditions for trainees to feel relatedness, competence and autonomy. Broadening our perceptions of feedback, beyond telling, might help create environments for learning to flourish.

How does assessment drive learning? A focus on students’ development of evaluative judgement

Summative assessment is often considered a motivator that drives students’ learning. Higher education has a responsibility in promoting lifelong learning and assessment plays an important role in supporting students’ capability to make evaluative judgements about their work and that of others. However, as research often focuses on formal pedagogical design, it is unclear what behaviours summative assessment prompts, thus the relationship between summative assessment, learning and evaluative judgement requires further investigation. Drawing on a small-scale ethnography-informed study, this paper adopts a practice theory approach to explore how undergraduate physics students from three year levels make evaluative judgements in the context of summative assessment tasks. The contexts explored through observations and interviews include a graded in-class tutorial, an out-of-class study group for an in-semester assignment, and individual preparation for examinations. The findings suggest that while summative assessment is a crucial aspect of students’ learning context, it does not fully shape students’ practices. Instead, students engage in incidental conversations about the quality of their work and how to do things in their studies. By focusing on what students actually do, this study integrates formal and informal aspects of students’ learning, highlighting the tensions between undergraduate practices and intended learning outcomes.

Variability and individual differences in L2 sociolinguistic evaluations: The GROUP, the INDIVIDUAL and the HOMOGENEOUS ENSEMBLE

AbstractThis study is the first to investigate subject-level variability in sociolinguistic evaluative judgements by 30 adult L2 German learners and explore whether the observed variability is characterizable as a function of individual differences in proficiency, exposure, and motivation. Because group-level estimates did not paint an accurate picture of the individual, we propose methods capable of integrating population-level estimates with person- and ensemble-centered approaches so as to reconcile generalizability and individuality. Using random effects from Bayesian mixed-effects models, we found that global subject-level variability in evaluative judgements was not predicted by individual differences. By building homogeneous ensembles (i.e., subgroups of individuals with similar evaluative judgements), however, it was possible to assess whether ensembles were characteristic of certain levels of individual differences. This ensemble-centered approach presents an innovative way to address the group-to-individual generalizability issue in cross-sectional data and transcend individual variability in order to make tentative generalizations of individual cases to wider populations.

PUBLIC LEGAL AND PRIVATE LEGAL ASPECTS OF DEFINING THE DEFINITION OF «ECOCIDE» DURING MILITARY AGGRESSION

The article examines issues that became relevant during the military aggression of Russia and Ukraine, namely ecocide. A comparative analysis of the composition of the crime «ecocide» and «ecocide during war» is provided, the need to distinguish between these concepts is argued, since the emphasis in these cases is on the nuances of defining the object of the crime, its objective side, subjective side and subjective object It is indicated that the blurring of the concept of «ecocide» due to evaluative judgments does not allow to fully bring the guilty parties to justice. A more extensive definition of ecocide is given in view of the military actions that are taking place today in Ukraine due to russian aggression, emphasizing the current position regarding value judgments, which have no place in criminal law and can enable war criminals to deny almost every case of crime. The position regarding the expediency of enshrining at the legislative level a separate article «ecocide during war» is substantiated. Also, in addition to the public-legal characteristics of ecocide, attention is drawn to the need to compensate for damage caused through instruments of a private legal nature. Issues of collective responsibility are highlighted which is usually not inherent in criminal law, but under certain conditions is more effective than individual responsibility, for example in the case of ecocide during war. It is proposed that in the case of recognition of a state recognized as an aggressor, a terrorist according to the signs, then in a forced manner without the ratification of any conventions on its part, a peacekeeping contingent of those countries that have decided on its illegal status should be introduced into it. Research is being conducted on the issue of ecocide from the standpoint of international law, in particular the Rome Statute, and positions on the prospects of Ukraine’s ratification of this international regulatory act are being discussed.

THE IMAGOLOGICAL SINUSOID OF "PILIGRIMAGE TO THE EAST FROM 1724 TO 1747" BY V. HRYHOROVYCH-BARSKYI

The paper deals with the peculiarities of the literary narration in the pilgrimage work by V. Hryhorovych-Barskyi "Pilgrimage...", represented through the prism of binary oppositions, in particular the antithesis "native / foreign". This study shows the attempt to investigate the process of perception and comprehension of cultural facts connected with different nations of the author-narrator and already formed stereotypes in the Ukrainian environment of the Baroque period. The article emphasizes on the ways how to determine the textual tools of embodying the "foreign" in the text of the literary work and interpret them in the context of national and cultural identity.
 The article notes that the main principle of reflecting everything that V. Hryhorovych-Barskyi saw should be characterized as the "true objectivism". The writer compares typical Ukrainian traditions related to making bread with Italian traditions, focusing on the fact that Italian bread differs significantly from the Ukrainian one in taste. He also draws attention to the differences in the communication of residents belonging to the same region, as well as the benefits of residents settling in hotels, etc.
 Watchful eyes of the author-narrator are concentrated on routine practices of many European cities in the 18th century. This paper shows that "Pilgrimage..." by V. Hryhorovych-Barskyi belongs to the pilgrimage literature or travel literature, which has specific features, such as: freedom of narrative strategies, the presence of a foreign locus, stylistic freedom. The narrator in this literary work acts as an observer. Consequently the thematic field of the text is not built exclusively around the meeting of native and foreign cultures. The author of "Pilgrimage..." only records the differences between Native and the Strange without making any evaluative judgments.
 The article highlights that the names of some subsections of "Pilgrimage..." have an imagological aspect, for example, "About the glorious city of Rome, about local customs", where the lexeme "local" attracts the recipient’s attention to content that will describe an exact and specific area.
 Sometimes the narrator still resorts to subjective evaluations in the formation of the heteroimage of the Strange, which considerably highlights the author’s imagological position.

The role of emotional awareness in evaluative judgment: evidence from alexithymia

Evaluative judgments imply positive or negative regard. But there are different ways in which something can be positive or negative. How do we tell them apart? According to Evaluative Sentimentalism, different evaluations (e.g., dangerousness vs. offensiveness) are grounded on different emotions (e.g., fear vs. anger). If this is the case, evaluation differentiation requires emotional awareness. Here, we test this hypothesis by looking at alexithymia, a deficit in emotional awareness consisting of problems identifying, describing, and thinking about emotions. The results of Study 1 suggest that high alexithymia is not only related to problems distinguishing emotions, but also to problems distinguishing evaluations. Study 2 replicated this latter effect after controlling for individual differences in attentional impulsiveness and reflective reasoning, and found that reasoning makes an independent contribution to evaluation differentiation. These results suggest that emotional sensibilities play an irreducible role in evaluative judgment while affording a role for reasoning.

The importance of situation evaluation and the ability to identify criteria in a construct-driven situational judgment test

Situational judgment tests (SJTs) typically focus on candidates' judgment of various response options and ignore the process of arriving at that judgment. In three studies, we developed two situational construal measures (i.e., situation evaluation and the ability to identify criteria [ATIC]) based on a construct-driven SJT measuring cooperativeness. We tested their role in the response judgment process and examined their incremental validity in predicting work performance. Study 1 (N = 30) provided qualitative evidence that situation evaluation and ATIC shape response judgment. Study 2 (Nstudent = 136, Nemployee = 154) showed that situation evaluation predicts response judgment. Moreover, situation evaluation had incremental validity in predicting supervisor-rated task performance and peer-rated interpersonal performance over and above response judgment and ATIC. Furthermore, ATIC predicted response judgment and had incremental validity in predicting interpersonal performance but not task performance over and above response judgment. Finally, Study 3 (N = 266) replicated the findings of Study 2 using a new sample and yielded similar results regardless of the order in which situation evaluation and response judgment were measured.

Knowledge graph assisted end-to-end medical dialog generation

Medical dialog systems have the potential to assist e-medicine in improving access to healthcare services, improving patient treatment quality, and lowering medical expenses. In this research, we describe a knowledge-grounded conversation generation model that demonstrates how large-scale medical information in the form of knowledge graphs can aid in language comprehension and generation in medical dialog systems. Generic responses are often produced by existing generative dialog systems, resulting in monotonous and uninteresting conversations. To solve this problem, we combine various pre-trained language models with a medical knowledge base (UMLS) to generate clinically correct and human-like medical conversations using the recently released MedDialog-EN dataset. The medical-specific knowledge graph contains broadly 3 types of medical-related information, including disease, symptom and laboratory test. We perform reasoning over the retrieved knowledge graph by reading the triples in each graph using MedFact attention, which allows us to use semantic information from the graphs for better response generation. In order to preserve medical information, we employ a policy network, which effectively injects relevant entities associated with each dialog into the response. We also study how transfer learning can significantly improve the performance by utilizing a relatively small corpus, created by extending the recently released CovidDialog dataset, containing the dialogs for diseases that are symptoms of Covid-19. Empirical results on the MedDialog corpus and the extended CovidDialog dataset demonstrate that our proposed model significantly outperforms the state-of-the-art methods in terms of both automatic evaluation and human judgment.

Neither Progress nor Regress: The Theological Substructure of T. F. Torrance's Genealogy of Modern Theology

AbstractT. F. Torrance's corpus of historical and theological writings contains genealogical reflections on the field of Christian doctrine. The basic shape of the genealogy is determined by what Torrance calls certain “ultimates,” theological commitments that derive their justification not from other beliefs that possess more authority than they themselves do, but from the way in which they seek to depict God in himself and in his economic activity. Core ultimate beliefs include the doctrine of the Trinity, the incarnation, and the ascension and the resurrection of Jesus Christ. These beliefs, more than the idealization of any particular stretch of past history, underlie all of Torrance's genealogical work, for instance, the subjects against which he develops sustained polemic; the overall structure of his genealogical account, which bears only a superficial resemblance to a decline narrative; and, most importantly, the point around which the genealogy revolves, namely, the epistemic reconciliation between human beings and God. This essay illustrates the nature of the genealogical narrative by outlining Torrance's treatment of Scripture and its interpretation and closes by assessing his effort as a whole. While the genealogy contains drawbacks that are worth registering, Torrance's narrative rightly avoids the sort of sweeping evaluative judgments associated with some often‐discussed genealogies, and it properly places its focus on the perennial issue of divine‐human reconciliation, which manifests itself differently in a variety of historical circumstances, even as it is not ultimately contingent upon them.

Examining the process of developing evaluative judgement in Japanese elementary schools—utilising the co-regulation and evaluative judgement model

ABSTRACT To enhance the effectiveness of formative assessment and self-regulated learning, this study focused on evaluative judgement. A process for developing evaluative judgement and co-regulation had proposed. However, this co-regulation and evaluative judgement model lacks validation for use in classroom settings; the process of developing evaluative judgement remains unclear. Thus, this study aims to examine the processes of co-regulation and development of evaluative judgement by applying the co-regulation and evaluative judgement model in a Japanese elementary school. We confirmed that evaluative judgements are shared with students through co-regulation and that both evaluative judgements and learning outcomes are enhanced. The results support the co-regulation and evaluative judgement model and partially reveal the process of the development of evaluative judgement. Evaluative judgements are expected to expand the effectiveness of formative assessment and self-regulation.

Threats to social security of the population in the South of Ukraine: new challenges and ways of minimization

The article attempts to analyze the level of social security in the Southern region of the country before and after the start of the full-scale invasion of Russia into the territory of Ukraine. It is defined that social security is determined by the nature of interaction between social groups on the basis of partnership, it is manifested at different levels (personal, group, regional, societal) in the capacity of state authorities to reproduce a stable, safe life. It has a subjective-objective nature and is determined both by the level of security of society and the region, as well as by individual or group well-being and awareness of one's security. The problems of assessing the level of social security have been identified, which are associated with the lack of a complete system of criteria assessment indicators and the inconsistency of the existing methodology and methods of data collection. It is proposed to monitor changes in the level of social security of the region under the influence of certain types of threats using an indicative sociological analysis based on the generalization of the values of objective sociogenic criteria and subjective evaluation judgments of the residents of the Southern region as objects of protection. It is proven that the unsatisfactory level of social security of the regional society is determined by a high level of insecurity, pessimistic perspectives regarding various types of social activity, and the impossibility of a quick return to normal life. There are outlined new threats to individual components of the social security of the population of Southern Ukraine associated with a full-scale war, in particular, demographic, economic, housing, educational, criminogenic, environmental security, etc. An attempt was made to identify possible ways of minimizing new threats to the social security of the South through the introduction of a number of economic, social and sociological mechanisms to minimize/overcome the consequences of these threats.

Willing and not being able: Nietzsche on akratic action

ABSTRACT Nietzsche claims that weakness of will is a pervasive feature of modernity: ‘Nothing is as timely [zeitgemass] as weakness of will’ (BGE 212). In this paper I explore a textual puzzle regarding the phenomenon traditionally identified with weakness of will, akrasia. Specifically, I draw attention to an apparent inconsistency between Nietzsche’s views regarding the origins of action and evaluative judgment, on the one hand, and his commitment to the possibility of akratic action, on the other. Nietzsche appears to account for both action and ‘conscience’ – which functions like what philosophers today would call ‘better judgment’—by appealing to the strength of the drives. Thus, it is not immediately clear how action could ever diverge from conscience in the manner required for akratic action. As I argue, however, Nietzsche’s theoretical commitments regarding action and conscience are ultimately consistent with the possibility of akratic action. After motivating the interpretive tension, I offer a closer examination of Nietzsche’s remarks regarding the sources of action and conscience, respectively. I then demonstrate, drawing on his description of the relations between drives and his distinction between conscious and unconscious thought, that Nietzsche conceives of akratic action as a kind of ‘willing and not being able.’

Disentangling Social Class–based Inequality: How Social Position Affects Evaluations of Economic and Cultural Markers of Social Class

How do the economic and cultural components of social class separately contribute to social class–based inequality? I argue that one approach to disentangle the effects of economic and cultural markers is to consider how decision-makers’ own social positions influence their evaluations of others in micro-level processes. I posit that decision-makers’ social positions influence their understandings and evaluations of the economic and cultural components of social class, giving rise to bias and inequality. In a series of original survey experiments, I manipulate the economic and cultural markers of a fictitious college applicant on subjects with elite and nonelite university degrees. The results show that the markers of social class affect individuals with elite degrees and individuals without elite degrees differently. I find that it is the cultural markers of social class, not the economic markers, that affect the judgments of evaluators with elite degrees. Applicants’ perceived competence and status help to explain the positive effect of cultural markers on evaluators with elite degrees. These results show the importance of social position and micro-level evaluation processes to help explain social class–based inequality.