Comparative Judgments Research Articles

Finite love

ABSTRACT It seems like a problem to say that love can be merited – its value is located in its transcendence of comparative judgments. However, we commonly make judgments about who is and is not worth loving. We deem certain romantic partners, friends, and family members as worth or not worth our loving time, attention, and effort. In this paper, I argue that love can be merited, and it is merited because of a person’s intrinsic valuable qualities. However, it is not the fact that such qualities can be compared that is relevant. What is relevant is the fact that such qualities are gradable, and what it means for someone to merit love is that their qualities meet a threshold or standard of value for love. Moreover, love requires some amount of practical wisdom in discerning these thresholds and when someone meets or fails to meet them. This is important to recognize since we are creatures of finite love. We do not have the capacity to love everyone who is both worthy and unworthy of love. Given our finite capacities, I argue that making wise and discerning judgments about whom to love should be kept in mind when philosophizing about love.

An Open-Source Adaptive Comparative Judgement App for Technology Education Research and Practice: Alpha Version

An Open-Source Adaptive Comparative Judgement App for Technology Education Research and Practice: Alpha Version

Identifying and Prioritizing the Educational Needs Professors of Medical Sciences in the Field of E-Learning with the Analytic Hierarchy Process Approach

Background: Professors in universities and educational institutions hold a crucial role in crafting educational approaches utilizing cutting-edge technologies. They need to possess significant expertise in e-learning and tackle the obstacles associated with delivering education and utilizing Information and communication technologies within higher education. To effectively navigate these challenges and adaptations, professors need to equip themselves with new educational techniques through training and readiness to adeptly incorporate emerging technologies into their teaching practices, thus enhancing the educational journey for students. By identifying, understanding, and ranking professors' requirements and preferences, effective and diverse educational initiatives can be tailored to meet their needs. Objectives: This research aimed to recognize the educational needs of medical science professors in the field of e-learning from the experts' point of view and prioritize these needs utilizing the analytic hierarchy process (AHP). Methods: This practical study was conducted in 2022 using a process-oriented approach and AHP. The main stages included the creation of a hierarchical structure, comparative judgment, and determining priorities. In the first step, 13 criteria were identified, which were derived from scientific articles and previous research. In the second step, the pairwise comparison matrix was used for comparative judgment, and the weight of each criterion was calculated. The third step involved determining priorities based on the weight of each criterion and the inconsistency ratio. The participants of this study included professors of basic and clinical sciences from different faculties. Results: The results of this study demonstrated that from the point of view of experts and participants, among the 13 criteria, the skills of virtual teaching (with weights of 0.146 and 0.148), electronic content production (with weights of 0.093 and 0.092), and holding online training courses (with the weights of 0.092 and 0.100) formed the three main needs of medical science professors in the field of e-learning and had the highest degree of importance and necessity compared to other criteria. Conclusions: Identifying the educational needs of medical science professors for organizing and designing necessary programs to enhance knowledge, skills, and empowerment will lead to improving the quality of e-learning, and eventually, reducing professors’ concerns in this area.

Does counting change what counts? Quantification fixation biases decision-making

People often rely on numeric metrics to make decisions and form judgments. Numbers can be difficult to process, leading to their underutilization, but they are also uniquely suited to making comparisons. Do people decide differently when some dimensions of a choice are quantified and others are not? We explore this question across 21 preregistered experiments (8 in the main text, N = 9,303; 13 in supplement, N = 13,936) involving managerial, policy, and consumer decisions. Participants face choices that involve tradeoffs (e.g., choosing between employees, one of whom has a higher likelihood of advancement but lower likelihood of retention), and we randomize which dimension of each tradeoff is presented numerically and which is presented qualitatively (using verbal estimates, discrete visualizations, or continuous visualizations). We show that people systematically shift their preferences toward options that dominate on tradeoff dimensions conveyed numerically-a pattern we dub "quantification fixation." Further, we show that quantification fixation has financial consequences-it emerges in incentive-compatible hiring tasks and in charitable donation decisions. We identify one key mechanism that underlies quantification fixation and moderates its strength: When making comparative judgments, which are essential to tradeoff decisions, numeric information is more fluent than non-numeric information. Our findings suggest that when we count, we change what counts.

Non-Monotonic Theories of Aesthetic Value

ABSTRACT Theorists of aesthetic value since Hume have traditionally aimed to justify at least some comparative judgments of aesthetic value and to explain why we thereby have more reason to appreciate some aesthetic objects than others. I argue that three recent theories of aesthetic value—Thi Nguyen’s and Matthew Strohl’s engagement theories, Nick Riggle’s communitarian theory, and Dominic McIver Lopes’ network theory—face a challenge to carry out this explanatory task in a satisfactory way. I defend a monotonicity principle according to which the strength of our aesthetic reasons to appreciate varies monotonically with aesthetic value and claim that these theories, because they do not respect the principle, are non-monotonic. If they cannot find a plausible way to preserve the link between the aesthetic goodness of an object and our aesthetic reasons to appreciate it, these non-monotonic theories should be rejected.

Cheating the evidence to get to best interest and the presumption of unfitness

AbstractThis article explores our national reactive child protection model which has focused on removing poor people from poor families and has resulted in conflating poverty with neglect. Significant issues of class and race are implicated given dramatic systemic racial and ethnic disparities and questions must be asked whether current practices have done more harm than good. In reviewing this landscape this article is descriptive and prescriptive and calls for adopting proactive public health‐oriented policies which support children and families in their homes and a culture of due process and fairness in juvenile and family court hearings involving children and their families. Dependency or abuse and neglect cases involve state intervention. These are not disputes between private parties where the sole focus is the best interest of the child regarding which guardian to be placed with. Cheating the Evidence warns against diluting the constitutional requirement of proving the lack of parental unfitness by clear and convincing evidence by making comparative judgments about placement which are based on often subjective assessments on best interests of the child which are resource based. Given that high percentage of cases involving allegations of neglect versus inflicted abuse these concerns are of paramount importance. There is a need for more rigor in fact‐finding and adherence to constitutionally required burdens of proof. Due process and fairness are best‐served by supporting client‐directed advocacy and greater transparency, including presumptively opening up the doors to closed juvenile and family court hearings as recommended by the National Council of Juvenile and Family Court Judges in 2005. Opening the doors is not a binary or either or proposition‐ we can preclude dissemination of records of youth, and protect confidentiality as a growing number of states have done. Transparency includes more than raising the level of practice of lawyers and judges‐ it includes encouraging systemic accountability from child serving agencies and educating the public about the complexities which are related to the birth or cradle to prison pipeline.

How does the Dunning-Kruger effect happen in creativity? The creative self-concept matters

The Dunning-Kruger effect refers to the tendency of unskilled individuals to overestimate their abilities while skilled individuals tend to underestimate themselves. The purpose of this study is to investigate the existence of the Dunning-Kruger effect in creativity and identify its key determinants. Using the cue-utilization framework, we explored how inaccurate metacognitive judgments result from creative self-concept, compared to those who can accurately evaluate themselves. Two hundred and ten university students completed an Alternative Uses Task (AUT) and evaluated their performance with self- and comparative judgments, then they reported the creative self-concept. The study conducted a two-step clustering analysis based on individuals' creative performance, judgments, and Bias Index of metacognitive judgments, and identified four distinct groups: 'skilled but underestimate' (SU group), 'unskilled but aware' (UA group), 'skilled and aware' (SA group), and 'unskilled and overestimate' (UO group). The SU and UO groups were typical examples of the Dunning-Kruger effect. Using binary logistic regression analysis, we found that high creative self-concept enabled skilled individuals to assess their creativity accurately, while it caused unskilled individuals to overestimate their creativity. This study identified the Dunning-Kruger effect in creativity while also offering new insights into the underlying mechanism of the Dunning-Kruger effect.

The Effects of Speaker Head Posture on Auditory Perception of Vocal Masculinity

ObjectiveThe purpose of this study was to investigate the impact of head position on listeners’ perception of vocal masculinity. MethodsTwelve cisgender women were recorded reciting two voiced sentences with varying head positions: baseline, flexed, and extended. Voice samples were cropped and fundamental frequency (fo) was resynthesized to control for any changes in fo across conditions. Twelve cisgender adults were recruited as listeners. Listeners were presented with 144 paired comparisons of speaker samples and were prompted to select the sample that sounded more masculine in each presented pairing. Ratings of masculinity were analyzed using Thurstone’s law of comparative judgment. A repeated measures analysis of variance (ANOVA) assessed the effects of head positioning, followed by Dunnett’s post hoc tests. ResultsThe ANOVA showed a statistically significant effect of head position on listener perceptions of masculinity: speech in the flexed position was perceived as statistically more masculine than that in the baseline condition. ConclusionsThe results of this study support the use of head posture manipulation to achieve increased vocal masculinity, which adds to the limited research related to voice masculinization strategies for those seeking gender-affirming voice care.

How consumers process online hotel ratings

Online travel agencies commonly utilize 5-point and 10-point scales to display hotel ratings. When processing and comparing hotel ratings on different numerical scales (e.g., 4.1/5 vs. 8.4/10), what method(s) do consumers use? Through nine studies, this research shows that consumers tend to employ absolute differences (e.g., 5–4.1 vs. 10–8.4) or relative differences (e.g., 4.1 ÷ 5 vs. 8.4 ÷ 10) when making comparative judgments. Notably, the choice of method can lead to preference reversals. We further reveal that higher numeracy leads to greater reliance on relative differences. However, such an impact is attenuated under a utilitarian motive. Additionally, a greater promotion focus leads to greater reliance on absolute differences. The findings contribute to a more nuanced understanding of how consumers process hotel ratings.

Investigating insight and rigour as separate constructs in mathematical proof

ABSTRACT We investigate undergraduate mathematics students' conceptions of rigour and insight. We conducted comparative judgement experiments in which students were asked to judge different proofs of the same theorem with five separate criteria: rigour, insight, understanding, simplicity and assessment marks. We predicted, and our experiment found, that rigour is a reliable construct. We predicted that insight is also a reliable construct but asking students to judge on the basis of “which proof gives you more insight into why a theorem is true” did not result in a reliable judging scale. Our analysis suggests two distinct dimensions: rigour and marks contribute to one factor whereas simplicity and personal understanding relate to a second factor. We suggest reasons why insight was related almost equally to both factors. First, students may not have developed an aesthetic sense in which they appreciate insight in ways that are regularly discussed by mathematics educators. Second, insight may not be a coherent and well-defined construct after all.

Algebraic analysis of pairwise comparisons with non-reciprocal property

The law of comparative judgement usually implies the reciprocal property of pairwise comparisons. But time error and position preference could yield some judgements that contradict this identity assumption. This paper proposes an algebraic approach to dealing with the challenge of non-reciprocal property in pairwise comparisons. The concept of non-reciprocal pairwise comparison matrix (NrPCM) is used to characterize the situation of non-reciprocal property. Algebraic analysis of NrPCMs is carried out, where a linear space is constructed by proposing a set of basis. Then the linear space is decomposed into the direct sum of some subspaces. The properties of the space are analyzed such as inner product, decomposition and coordinate. Finally, we apply the algebraic method to investigate the Condorcet paradox under the impact of indifferent voters. A novel transformation formula of non-reciprocal pairwise comparisons is proposed, where the non-reciprocal property holds. It is revealed that the occurrence of voting paradox can be understood clearly according to the transitivity property of NrPCMs.

The Effects of Psychological Distance on the Diagnosticity of Digits to the Left Versus Right of a Separator

Many numerical expressions contain a separator such as a comma (e.g., 1,234) or a decimal point (e.g., 12.34) that divides the number into left and right parts. This research examines how individuals compare such numbers as a function of psychological distance. Integrating insights from research on diagnosticity, numerical cognition, and mental construal, the authors hypothesize that as psychological distance increases, the perceived diagnosticity of digits to the right of a separator (hereinafter “right digits”) decreases faster than that of the digits to the left of a separator (hereinafter “left digits”), which leads individuals to assign less weight to the right digits in comparative judgment. Four studies offer triangulating support for this theorizing and the underlying mechanism. In addition, the authors show that this effect is attenuated or suppressed when individuals perceive that numerical ratings are more stable over time (thereby increasing the perceived diagnosticity of right digits; Study 3) and when the distinction between left and right digits is made less obvious (e.g., when removing the digit separator; Study 4). This research offers additional nuance to our understanding of numerical cognition and how psychological distance influences the processing of different types of information.

Crowdsourced Comparative Judgement for Evaluating Learner Texts: How Reliable are Judges Recruited from an Online Crowdsourcing Platform?

Abstract Recent studies of proficiency measurement and reporting practices in applied linguists have revealed widespread use of unsatisfactory practices such as the use of proxy measures of proficiency in place of explicit tests. Learner corpus research is one specific area affected by this problem: few learner corpora contain reliable, valid evaluations of text proficiency. This has led to calls for the development of new L2 writing proficiency measures for use in research contexts. Answering this call, a recent study by Paquot et al. (2022) generated assessments of learner corpus texts using a community-driven approach in which judges, recruited from the linguistic community, conducted assessments using comparative judgement. Although the approach generated reliable assessments, its practical use is limited because linguists are not always available to contribute to data collections. This paper, therefore, explores an alternative approach, in which judges are recruited through a crowdsourcing platform. We find that assessments generated in this way can reach near identical levels of reliability and concurrent validity to those produced by members of the linguistic community.

Learning analytics dashboard design: Workplace learner preferences for reference frames in immersive training in practice

AbstractBackgroundLearning analytics dashboards are increasingly being used to communicate feedback to learners. However, little is known about learner preferences for dashboard designs and how they differ depending on the self‐regulated learning (SRL) phases the dashboards are presented (i.e., forethought, performance, and self‐reflection phases) and SRL skills. Insight into design preferences for dashboards with different reference frames (i.e., progress, social, internal achievement and external achievement) is important because the effectiveness of feedback can depend upon how a learner perceives it.ObjectiveThis study examines workplace learner preferences for four dashboard designs for each SRL phase and how SRL skills relate to these preferences.MethodsSeventy participants enrolled in a chemical process apprenticeship program took part in the study. Preferences were determined using a method of adaptive comparative judgement and SRL skills were measured using a questionnaire. Preferences were tested on four dashboard designs informed by social and temporal comparison theory and goal setting theory. Multinomial logistic regressions were used to examine the relationship between dashboard preferences and SRL.Results and ConclusionsResults show that the progress reference frame is more preferred before and after task performance, and the social reference frame is less preferred before and after task performance. It was found that the higher the SRL skill score the higher the probability a learner preferred the progress reference frame compared to having no preference before task performance. The results are consistent with other findings, which suggest caution when using social comparison in designing dashboards which provide feedback.

Do credences model guesses?

AbstractWhat are credences? Where do the numbers come from? Some have argued that they are brute and primitive; others, that they model our dispositions to bet, our comparative confidence judgments, or our all‐out beliefs. This paper explores a new answer to this question: credences model our dispositions to guess. I argue that we can think of credences this way, and then consider: should we?

Revitalizing university performance evaluation: the case of SETARA model in Malaysia

PurposeSETARA is a well-known university rating tool in Malaysia. The study aims to enhance the transparency, accuracy, and reliability of SETARA assessment instrument by improving its weighting scheme for the domains, sub-domains, criteria, and indicators.Design/methodology/approachThe study utilized a quantitative research design and collected responses from 29 academic administrators in both public and private higher education institutions in Malaysia. The four steps of the Analytic Hierarchy Process (AHP) were employed. An AHP questionnaire was developed, adopting the domains, sub-domains, criteria, and indicators of the SETARA assessment instrument. The “Superdecision” software was used to synthesize the pairwise comparison judgments and compute the consistency ratio for all group pairwise comparison matrices.FindingsThe AHP-assigned weights differed significantly from those assigned by the SETARA rating system. For instance, the weight for “Input” increased from 20 to 41.05 out of 100, and the weight for “Output” decreased from 40 to 25.52. The new SETARA rating instrument is expected to be used by the Ministry of Higher Education (MOHE) due to its scientifically justified foundation to ensure transparency, accuracy, and reliability.Research limitations/implicationsThe study is limited by small sample size for data collection. Future research could benefit from a larger and more diverse sample, including participants from a number of national-level universities and relevant agencies like Malaysia Qualification Agency (MQA) and MOHE. Additionally, the research focused solely on recalculating weights and did not consider modifying the criteria set. Further investigation is needed to determine if some criteria can be added or removed.Practical implicationsThis research offers practical implications for refining the SETARA assessment mechanism and improving the quality of higher education in Malaysia. The modified weights provide guidance to Higher Education Institutions (HEIs) on where to focus their efforts to ensure quality. The transparent methodology introduced by this study can serve as a model for enhancing educational quality evaluations in Malaysia and potentially in other higher education systems worldwide.Originality/valueThe originality of this study lies in its innovative application of the AHP to the SETARA assessment tool, addressing a crucial gap in the assessment process. By providing a comprehensive and transparent methodology for assigning percentages and weights, this research offers a practical blueprint for enhancing the precision and impact of educational quality evaluations in Malaysia and beyond.

Can an isolated middle-series item make a "Dent" in the bow-shaped serial-position curve of comparative judgments?

Studies have shown that local (e.g., midseries) items' specific properties, including being isolated from rest of the items, can generate a local distinctiveness effect, enhancing the memory performance for the local items in serial recall or absolute judgments. However, this has not been the case in relative (comparative) judgments. For the first time, the present study found a local distinctiveness effect in comparative judgments by using an opposite-gender name for the midseries item in an otherwise uniformly one-gender name serial list. The reasons for the previous studies' failure to produce this effect in comparative judgments and the present study's success in obtaining it were discussed. The implication of the finding for the item/order information opponent-process theories was also suggested.

Familiar Sequences Are Processed Faster Than Unfamiliar Sequences, Even When They Do Not Match the Count-List.

In order processing, consecutive sequences (e.g., 1-2-3) are generally processed faster than nonconsecutive sequences (e.g., 1-3-5) (also referred to as the reverse distance effect). A common explanation for this effect is that order processing operates via a memory-based associative mechanism whereby consecutive sequences are processed faster because they are more familiar and thus more easily retrieved from memory. Conflicting with this proposal, however, is the finding that this effect is often absent. A possible explanation for these absences is that familiarity may vary both within and across sequence types; therefore, not all consecutive sequences are necessarily more familiar than all nonconsecutive sequences. Accordingly, under this familiarity perspective, familiar sequences should always be processed faster than unfamiliar sequences, but consecutive sequences may not always be processed faster than nonconsecutive sequences. To test this hypothesis in an adult population, we used a comparative judgment approach to measure familiarity at the individual sequence level. Using this measure, we found that although not all participants showed a reverse distance effect, all participants displayed a familiarity effect. Notably, this familiarity effect appeared stronger than the reverse distance effect at both the group and individual level; thus, suggesting the reverse distance effect may be better conceptualized as a specific instance of a more general familiarity effect.

Appraising the Conceptual and Lexical Retrieval in Persons with Fluent and Nonfluent Aphasia

Abstract Objective Conceptual knowledge refers to understanding and interpreting concepts and their relationships. Lexical knowledge is the knowledge that can be expressed through words. It entails meaning of words and their relationships. Word retrieval impairments are the most radical deficits that restrict communicative functions for individuals with aphasia. The current study aimed to compare the conceptual and linguistic impairments associated with fluent and nonfluent persons with aphasia (PWAs). Materials and Methods A total of 15 PWAs were recruited for the study, out of which, 8 were fluent aphasia and 7 were nonfluent aphasia. One picture-based conceptual task (picture comparison judgments) and one verb-based linguistic task (word comparison judgments) were used as the experimental paradigm. Results and Conclusion The findings of the study showed no significant difference in the performance of conceptual and lexical retrieval abilities in both fluent and nonfluent aphasia. The current study highlights the fact that both conceptual and lexical retrieval impairments prevail in both fluent and nonfluent aphasia.

A Bayesian active learning approach to comparative judgement within education assessment

Assessment is a crucial part of education. Traditional marking is a source of inconsistencies and unconscious bias, placing a high cognitive load on the assessors. One approach to address these issues is comparative judgement (CJ). In CJ, the assessor is presented with a pair of items of work, and asked to select the better one. Following a series of comparisons, a rank for any item may be derived using a ranking model, for example, the Bradley-Terry model, based on the pairwise comparisons. While CJ is considered to be a reliable method for conducting marking, there are concerns surrounding its transparency, and the ideal number of pairwise comparisons to generate a reliable estimation of the rank order is not known. Additionally, there have been attempts to generate a method of selecting pairs that should be compared next in an informative manner, but some existing methods are known to have created their own bias within results inflating the reliability metric used within the process. As a consequence, a random selection approach is usually deployed.In this paper, we propose a novel Bayesian approach to CJ (which we call BCJ) for determining the ranks of a range of items under scrutiny alongside a new way to select the pairs to present to the marker(s) using active learning, addressing the key shortcomings of traditional CJ. Furthermore, we demonstrate how the entire approach may provide transparency by providing the user insights into how it is making its decisions and, at the same time, being more efficient. Results from our synthetic experiments confirm that the proposed BCJ combined with entropy-driven active learning pair-selection method is superior (i.e. always equal to or significantly better) than other alternatives, for example, the traditional CJ method with differing selection methods such as uniformly random, or the popular no repeating pairs where pairs are selected in a round-robin fashion. We also find that the more comparisons that are conducted, the more accurate BCJ becomes, which solves the issue the current method has of the model deteriorating if too many comparisons are performed. As our approach can generate the complete predicted rank distribution for an item, we also show how this can be utilised in probabilistically devising a predicted grade, guided by the choice of the assessor. Finally, we demonstrate our approach on a real dataset on assessing GCSE (UK school-level) essays, highlighting the advantages of BCJ over CJ.