Relative Judgments Research Articles

Judgments of learning (JOLs) impact item memory but not source memory: Insights into JOL reactivity using a multinomial model.

Past research has evaluated participants' understanding of their memory by soliciting judgments of learning (JOLs). Importantly, JOLs sometimes change memory for the judged material, leading to JOL reactivity. The cue-strengthening account (Soderstrom et al., 2015) and changed-goal account (Mitchum et al., 2016) propose different mechanisms that lead to JOL reactivity. In the present study, we collected measures that can provide further insight into these mechanisms. Specifically, participants studied related and unrelated word pairs in different colored fonts for a source recognition test. Across three experiments, data were analyzed using a hierarchical Bayesian model of multidimensional source memory to determine how JOLs impact item memory as well as relatedness and color source memory. In Experiment 2, we also compared the effects of making JOLs to making judgments of relatedness (JORs), and Experiment 3 examined how JOLs impact study time allocation. The results of our experiments failed to fully follow predictions of either account. Making JOLs (Experiments 1-3) and JORs (Experiment 2) strengthened item memory for related as well as unrelated pairs, the latter finding not predicted by either account. In addition, JOLs and JORs did not specifically strengthen source memory for relatedness, as the cue-strengthening account predicts, nor did JOLs change study time (Experiment 3), as suggested by the changed-goal account. In all, our results provide novel insight that enhanced item memory may be largely responsible for JOL reactivity, thus adjudicating between candidate explanations. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Nothing Comes Without Its World – Practical Challenges of Aligning LLMs to Situated Human Values through RLHF

Work on value alignment aims to ensure that human values are respected by AI systems. However, existing approaches tend to rely on universal framings of human values that obscure the question of which values the systems should capture and align with, given the variety of operational situations. This often results in AI systems that privilege only a selected few while perpetuating problematic norms grounded on biases, ultimately causing equity and justice issues. In this perspective paper, we unpack the limitations of predominant alignment practices of reinforcement learning from human feedback (RLHF) for LLMs through the lens of situated values. We build on feminist epistemology to argue that at the design-time, RLHF has problems with representation in the subjects providing feedback and implicitness in the conceptualization of values and situations of real-world users while lacking system adaptation to real user situations at the use time. To address these shortcomings, we propose three research directions: 1) situated annotation to capture information about the crowdworker’s and user’s values and judgments in relation to specific situations at both the design and use-time, 2) expressive instruction to encode plural values for instructing LLMs systems at design-time, and 3) reflexive adaptation to leverage situational knowledge for system adaption at use-time. We conclude by reflecting on the practical challenges of pursuing these research directions and situated value alignment of AI more broadly.

The principal components of meaning, revisited.

Osgood, Suci, and Tannebaum were the first to attempt to identify the principal components of semantics using dimensional reduction of a high-dimensional model of semantics constructed from human judgments of word relatedness. Modern word-embedding models analyze patterns of words to construct higher dimensional models of semantics that can be similarly subjected to dimensional reduction. Hollis and Westbury characterized the first eight principal components (PCs) of a word-embedding model by correlating them with several well-known lexical measures, such as logged word frequency, age of acquisition, valence, arousal, dominance, and concreteness. The results show some clear differentiation of interpretation between the PCs. Here, we extend this work by analyzing a larger word-embedding matrix using semantic measures initially derived from subjective inspection of the PCs. We then use quantitative analysis to confirm the utility of these subjective measures for predicting PC values and cross-validate them on two word-embedding matrices developed on distinct corpora. Several semantic and word class measures are strongly predictive of early PC values, including first-person and second-person verbs, personal relevance of abstract and concrete words, affect terms, and names of places and people. The predictors of the lowest magnitude PCs generalized well to word-embedding matrices constructed from separate corpora, including matrices constructed using different word-embedding methods. The predictive categories we describe are consistent with Wittgenstein's argument that an autonomous level of social interaction grounds linguistic meaning.

Envisioning scenarios in designs for Networked Learning

This article presents an application of a design methodology to envision implicit value hierarchies in the design process of a digital learning platform meant to encompass learning processes and activities conducive to experiential collaborative learning (ECL). The authors argue that many technologies for the field of education fall short of their purposes and neglect intended underpinning pedagogy and didactics. Previous research efforts in networked learning have primarily focused on conceptual critiques of the implementation of EdTech and warned of technological euphoria undermining relevant caution. This means, that when a design team tries to conceptualize technological artefacts into a script the more ethical and value-oriented parts of the learning process tend to be ignored. While we agree with the conceptual critiques, our approach has instead been to engage with the design process and implement appropriate methodologies in an attempt to highlight implicit value hierarchies in the underlying learning theory. When using technologies in Networked learning we thus emphasize that both designers and stakeholders should engage in a systematic discussion and reflection of values and related judgements while constructing a value hierarchy. Through a Value-based design methodology based on semantic zooming we thus present 7 interconnected envisioning scenarios developed in the UnFoLD project to demonstrate how it is possible to operationalize values into detailed design briefs or technological scripts. This article will through presented experiences from a design process, show how the methodology of envisioning scenarios can be applied to mitigate the risks of implementation technology in a learning situation. We argue that an awareness and mapping of values as a part of the design process is essential and that an increased focus on the ethical and moral responsibilities of designers and involved researchers are important as technologies should not be seen as isolated, value-neutral, or uncomplicated translations of analogue teaching activities. The purpose of the article is to inspire other researchers and designers to implement value hierarchies, envisioning scenarios, or other similar methods to ensure that pedagogical and didactic priorities are not lost in accommodating marketability, practicalities, or technological constraints.

Misconducts in research and methods to uphold research integrity.

Research misconduct refers to deliberate or accidental manipulation or misrepresentation of research data, findings, or processes. It can take many forms, such as fabricating data, plagiarism, or failing to disclose conflicts of interest. Data falsification is a serious problem in the field of medical research, as it can lead to the promotion of false or misleading information. Researchers might engage in p-hacking - the practice of using someone else's research results or ideas without giving them proper attribution. Conflict of interest (COI) occurs when an individual's personal, financial, or professional interests could potentially influence their judgment or actions in relation to their research. Nondisclosure of COI can be considered research misconduct and can damage the reputation of the authors and institutions. Hypothesis after results are known can lead to the promotion of false or misleading information. Cherry-picking data is the practice of focusing attention on certain data points or results that support a particular hypothesis, while ignoring or downplaying results that do not. Researchers should be transparent about their methods and report their findings honestly and accurately. Research institutions should have clear and stringent policies in place to address scientific misconduct. This knowledge must become widespread, so that researchers and readers understand what approaches to statistical analysis and reporting amount to scientific misconduct. It is imperative that readers and researchers alike are aware of the methods of statistical analysis and reporting that constitute scientific misconduct.

Creative connections: the neural correlates of semantic relatedness are associated with creativity

The associative theory of creativity proposes that creative ideas result from connecting remotely related concepts in memory. Previous research found that higher creative individuals exhibit a more flexible organization of semantic memory, generate more uncommon word associations, and judge remote concepts as more related. In this study (N = 93), we used fMRI to investigate brain regions involved in judging the relatedness of concepts that vary in their semantic distance, and how such neural involvement relates to individual differences in creativity. Brain regions where activity increased with semantic relatedness mainly overlapped with default, control, salience, semantic control, and multiple demand networks. The default and semantic control networks exhibited increased involvement when evaluating more remote associations. Finally, higher creative people, who provided higher relatedness judgements on average, exhibited lower activity in those regions, possibly reflecting higher neural efficiency. We discuss these findings in the context of the neurocognitive processing underlying creativity. Overall, our findings indicate that judging remote concepts as related reflects a cognitive mechanism underlying creativity and shed light on the neural correlates of this mechanism.

Can an isolated middle-series item make a "Dent" in the bow-shaped serial-position curve of comparative judgments?

Studies have shown that local (e.g., midseries) items' specific properties, including being isolated from rest of the items, can generate a local distinctiveness effect, enhancing the memory performance for the local items in serial recall or absolute judgments. However, this has not been the case in relative (comparative) judgments. For the first time, the present study found a local distinctiveness effect in comparative judgments by using an opposite-gender name for the midseries item in an otherwise uniformly one-gender name serial list. The reasons for the previous studies' failure to produce this effect in comparative judgments and the present study's success in obtaining it were discussed. The implication of the finding for the item/order information opponent-process theories was also suggested.

HAGCN: A relation extraction model based on heterogeneous graph convolutional neural network and graph attention

Relation extraction is one of the core tasks of natural language processing, which aims to identify entities in unstructured text and judge the semantic relationships between them. In the traditional methods, the extraction of rich features and the judgment of complex semantic relations are inadequate. Therefore, in this paper, we propose a relation extraction model, HAGCN, based on heterogeneous graph convolutional neural network and graph attention mechanism. We have constructed two different types of nodes, words and relations, in a heterogeneous graph convolutional neural network, which are used to extract different semantic types and attributes and further extract contextual semantic representations. By incorporating the graph attention mechanism to distinguish the importance of different information, and the model has stronger representation ability. In addition, an information update mechanism is designed in the model. Relation extraction is performed after iteratively fusing the node semantic information to obtain a more comprehensive node representation. The experimental results show that the HAGCN model achieves good relation extraction performance, and its F1 value reaches 91.51% in the SemEval-2010 Task 8 dataset. In addition, the HAGCN model also has good results in the WebNLG dataset, verifying the generalization ability of the model.

Neural Correlates of Analogical Reasoning on Syntactic Patterns.

Analogical reasoning is central to thought and learning. However, previous neuroscience studies have focused mainly on neural substrates for visuospatial and semantic analogies. There has not yet been research on the neural correlates of analogical reasoning on syntactic patterns generated by the syntactic rules, a key feature of human language faculty. The present investigation took an initial step to address this paucity. Twenty-four participants, whose brain activity was monitored by fMRI, engaged in first-order and second-order relational judgments of syntactic patterns as well as simple and complex working memory tasks. After scanning, participants rated the difficulty of each step during analogical reasoning; these ratings were related to signal intensities in activated regions of interest using Spearman correlation analyses. After prior research, differences in activation levels during second-order and first-order relational judgments were taken as evidence of analogical reasoning. These analyses showed that analogical reasoning on syntactic patterns recruited brain regions consistent with those supporting visuospatial and semantic analogies, including the anterior and posterior parts of the left middle frontal gyrus, anatomically corresponding to the left rostrolateral pFC and the left dorsolateral pFC. The correlation results further revealed that the posterior middle frontal gyrus might be involved in analogical access and mapping with syntactic patterns. Our study is the first to investigate the process of analogical reasoning on syntactic patterns at the neurobiological level and provide evidence of the specific functional roles of related regions during subprocesses of analogical reasoning.

Doxing, political affiliation, and type of information: Effects on suspicion, perceived similarity, and hiring-related judgments.

Researchers have begun to focus on the influence of political affiliation in organizations. In this context, we investigated how doxing (i.e., using social media to post information online with malintent) influences hiring-related decisions. Based on the integration of a political affiliation and state suspicion model, we investigated how a dox containing different types of information (affirming a political party affiliation vs. providing derogatory/negative information about an opposing party) and political party affiliation similarity influenced hiring-related perceptions of job applicants. Given doxing's characteristics, we expanded the "decision space" to include effects about expected organizational image and expected retaliation. In Study 1, we found that the type of information and party similarity influenced suspicion of the applicant and perceived similarity with the applicant, whereas doxing only influenced suspicion. In turn, suspicion and perceived similarity predicted expected task performance and organizational image, and exploratory analyses suggested an interactive effect of these variables. Suspicion also predicted expected retaliation from individuals outside the organization. In Study 2, we confirmed that doxing was related to suspicion as well as the interactive effect of information type and party similarity. We explain that interaction using the notion of symbolic threat. In both studies, the effects of type of information and party similarity were pervasive. Our results support the similarity-attraction paradigm and a model of political affiliation. Expanding relevant theories to include suspicion helps better understand politically related judgments and the additional outcomes of expected organizational image and retaliation. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The impact of spatial and verbal working memory load on semantic relatedness judgements.

Studies using a relatedness judgement task have found differences between prime-target word pairs that vary in the degree of semantic relatedness. However, the influence of working memory load on semantic processing in this task and the role of the type of working memory task have not yet been investigated. The present study therefore investigated for the first time the effect of working memory load (low vs. high) and working memory type (verbal vs. spatial) on semantic relatedness judgements. Semantically strongly related (e.g., hip - KNEE), weakly related (e.g., muscle - KNEE) and unrelated (e.g., office - KNEE) Polish word pairs were presented in an experiment involving a dual working memory and semantic relatedness task. The data revealed that, relative to semantically unrelated word pairs, responses were faster for strongly related pairs but slower for weakly related pairs. Importantly, the verbal working memory task decreased facilitation for strongly related pairs and increased inhibition for weakly related pairs relative to the spatial working memory task. Furthermore, working memory load impacted only weakly related pairs in the verbal but not in the spatial working memory task. These results show that working memory type and load influence semantic relatedness judgements, but the direction and size of the impact depend on the strength of semantic relations.

The Ordinal Consistency of a Hesitant Fuzzy Preference Relation

An efficient solution to the misleading solutions of decision-making problems is the study of consistency when the decision makers express their opinions by means of fuzzy preference relations. To elucidate the consistency of HFPRs, the S-ordinal consistency of HFPRs was proposed, and S-OCI was also proposed to evaluate the degree of consistency of a hesitant fuzzy preference relation by calculating the unreasonable 3-cycles in the directed graph. Two novel methods were also proposed for calculation of the S-OCI. Moreover, the inconsistent judgment in hesitation fuzzy preference relation was developed. In order to repair the inconsistency of a hesitant fuzzy preference relation, an algorithm for finding and removing 3-cycles in digraph was developed. Finally, some illustrative examples were given to prove the effectiveness of the method.

Improving the accuracy of social work judgements: A proof‐of‐concept study of a training programme

AbstractChild and family social workers routinely make professional judgements involving significant legal and moral questions (e.g. whether a child has been abused) and more ‘everyday’ issues (e.g. will the child be re‐referred again if we close the case now?) Yet the world is capricious, and we rarely know with certainty what is going to happen in future or the likely impact of our different choices. Given the consequences of their judgements and decisions, it is imperative that social workers are provided with the best possible support. This paper reports a proof‐of‐concept study of a set of interventions to improve the judgemental accuracy of social workers: (i) a survey to identify respondents with above‐average existing abilities, (ii) training sessions on cognitive debiasing and (iii) structured group working and (iv) three methods for aggregating individual judgements. Findings indicate that it is possible to measure the accuracy of social work judgements in relation to case‐study materials and retrospective questions, while the feedback about the training was largely positive. Any future studies should aim to recruit a more diverse set of respondents, test judgemental accuracy in relation to prospective judgements and explore what types of questions would be most helpful for real‐world decision‐making.

Considerations of communication challenges facing sonographers in Australia beyond 2024: An ageing population, ageism and maintaining the focus on patient‐centred care

AbstractIn the coming decades, the number of people in the Australian population over the age of 65 is predicted to increase to 23% by the year 2066. The process of physiological ageing is associated with an increase in the presentation of disease pathologies, an increased potential for physical and cognitive deficits and a decrease in health literacy. The ageing of the Australian population presents challenges for the sonography profession including an increasing age gap between the sonographer and their patient as well as increasing complexities in performing ultrasounds on older patients. The natural changes that accompany ageing can create barriers to effective communication between sonographers and their patients. Ageist beliefs can lead to conscious and unconscious bias and unfounded judgements by sonographers decreasing the quality of patient care. To combat the decline in the quality of healthcare that is reported for older patients, sonographers can endeavour to put into place deliberate practical tools and strategies to avoid ageism and improve communication with older patients. Conversely, institutional awareness, education and policies that address age related judgements and bias from older patients towards younger sonographers can foster respectful dialogue, promote understanding and stimulate positive communication across different age groups.

AGREE: a new benchmark for the evaluation of distributional semantic models of ancient Greek

Abstract The last years have seen the application of Natural Language Processing, in particular, language models, to the study of the Semantics of ancient Greek, but only a little work has been done to create gold data for the evaluation of such models. In this contribution we introduce AGREE, the first benchmark for intrinsic evaluation of semantic models of ancient Greek created from expert judgements. In the absence of native speakers, eliciting expert judgements to create a gold standard is a way to leverage a competence that is the closest to that of natives. Moreover, this method allows for collecting data in a uniform way and giving precise instructions to participants. Human judgements about word relatedness were collected via two questionnaires: in the first, experts provided related lemmas to some proposed seeds, while in the second, they assigned relatedness judgements to pairs of lemmas. AGREE was built from a selection of the collected data.

Forming pre-service English language teachers’ competence in critical reading through authentic short stories

The article is devoted to the problem of developing critical reading competence and critical thinking skills of pre-service English language teachers by means of interactive teaching on the material of authentic short stories. The phenomenon of critical thinking is considered thoroughly, its definition is given, and its features are described. Critical thinking is defined as the process of using cognitive techniques and strategies with the purpose of skilful and active understanding and evaluation of observations, communication, information, and argumentation that increase the probability of obtaining the desired final result. The critical thinking skills are outlined, as well as the stages of their formation in correlation with the stages of interactive foreign language teaching. The definitions of critical reading and critical reading competence are worked out. It is established that critical reading is a process of reading which presupposes the use of reflection with the purpose of content evaluation and reasonable judgments in relation to it. Critical reading competence is defined as a person’s ability to read authentic texts of various genres with full understanding, analysis, and evaluation of their content; which is based on relevant abilities, skills, knowledge, and communication skills, in order to respond intelligently and adequately to what is read. The potential of using authentic short stories to develop critical reading competence of pre-service English language teachers has been explored. The difficulties of forming critical reading competence on the material of authentic short stories are considered. The methodology of teaching critical reading on the material of authentic short stories to pre-service English language teachers is suggested: the stages of developing critical reading skills are distinguished and described; the strategies and techniques of teaching critical reading on the material of authentic short stories with interactive method are presented.

Age differences in priming as a function of processing at encoding

It is unclear whether implicit memory (priming) is affected by aging. Some studies have reported no difference between young and older adults, while others have uncovered reliable reductions. An important factor that may explain these discrepancies is the manner of encoding. Processing requirements (perceptual/conceptual) have varied considerably between studies, yet processing abilities are not equally affected by aging. This study examined whether processing during encoding moderates age effects on priming. Young and older participants studied object-word pairs and made natural/manufactured (conceptual) and left/right rotation (perceptual) judgements in relation to the word or object. Objects served as targets on a subsequent continuous identification with recognition task to assess priming and recognition. Priming and recognition were greater in young than older adults for attended items, with a larger effect size in the conceptual than the perceptual condition. Findings suggest that age differences in priming may be a function of processing at encoding.

Inter- and Intra- Hemispheric Interactions in Reading Ambiguous Words

The present study investigated how the brain processes words with multiple meanings. Specifically, we examined the inter- and intra-hemispheric connectivity of unambiguous words compared to two types of ambiguous words: homophonic homographs, which have multiple meanings mapped to a single phonological representation and orthography, and heterophonic homographs, which have multiple meanings mapped to different phonological representations but the same orthography. Using a semantic relatedness judgment task and effective connectivity analysis via Dynamic Causal Modeling (DCM) on previously published fMRI data (Bitan et al., 2017), we found that the two hemispheres compete in orthographic processing during the reading of unambiguous words. For heterophonic homographs, we observed increased connectivity within the left hemisphere, highlighting the importance of top-down re-activation of orthographic representations by phonological ones for considering alternative meanings. For homophonic homographs, we found a flow of information from the left to the right hemisphere and from the right to the left, indicating that the brain retrieves different meanings using different pathways. These findings provide novel insights into the complex mechanisms involved in language processing and shed light on the different communication patterns within and between hemispheres during the processing of ambiguous and unambiguous words.

Perceptual anchoring effects: Evidence of response bias and a change in estimates sensitivity.

People's estimates of perceptual quantities are commonly biased by the contextual presence of other quantities (like numbers). In this study, we address assimilation anchoring effects (approximation of real quantities to contextual quantities) that occur for visually displayed proportions, defining a new methodological setting for the effect. Similar to classic approaches, we asked participants across several trials whether the display contained a feature in a proportion higher or lower than "a randomly selected value" (relative judgments), and then estimated the feature proportions (absolute judgments). Across all trials, we presented seven anchors ranging from .20 to .80, each with a visually displayed representation of the same seven proportions (49 judgments in total). This allowed for a psychophysical approach to individual estimates and signal detection indexes, providing new insights into how the anchoring effect is generated in this setting. Our findings suggest that anchoring effects occur both as a bias (changes in response criteria) and as a change in the ability to discriminate stimuli (affecting sensitivity indexes). Moreover, anchors modulate the level of stimuli features for which estimates were more uncertain. Finally, our results indicate that anchor effects occur immediately in the first phase of the two-phase paradigm, leading to the availability of values for supporting absolute estimates. By using a psychophysical approach to the anchoring effects, for the first time, we could clarify that this effect is the result of both bias and changes in the ability to discriminate quantity.

Evaluating the effects of uncertainty in interval pairwise comparison matrices

In the context of multicriteria decision analysis, Pairwise Comparison Matrices (PCMs) represent a widely-used mathematical structure to capture comparisons about criteria or alternatives. When a confidence (or uncertainty) level is associated to the provided comparison measures, PCMs are naturally extended to the so-called Interval Pairwise Comparison Matrices (IPCM). Classical approaches, based on the eigenvector associated to the largest eigenvalue of the matrix (Analytic Hierarchy Process) or optimization problems (e.g. the Logarithmic Least Squares approach), are frequently adopted in order to estimate absolute utilities from relative judgements in absence of expert's uncertainty. Conversely, in the presence of uncertain measures, computationally intensive approaches based on Monte Carlo analysis have been adopted. In this paper, starting from a given PCM, we provide an approach able to find the smallest multiplicative perturbations of its elements able to perturb the obtained ranking from the ordinal perspective. On the basis of such approach, we provide theoretical results which can be adopted also in the case of uncertain measures (e.g., IPCMs) with the aim to characterize the stability of the obtained ranking on the basis of the given uncertainty. As testified by numerical examples that conclude the paper, the proposed framework can greatly simplify the classical approaches by providing the possibility to make direct assumptions on the final outcome of Monte Carlo analysis and reducing or avoiding its required computational effort.