Comparative Judgments Research Articles

Comparative Learning

AbstractThis article concerns the diachronic rationality norms for comparative confidence judgments, that is, judgments of the form “I am at least as confident in $p$ as I am in $q$.” Specifically, it identifies, characterizes, and evaluates an intuitively compelling learning rule called comparative conditionalization that specifies how agents should revise their comparative confidence judgments in the face of novel evidence.

Validating a forced-choice method for eliciting quality-of-reasoning judgments

In this paper we investigate the criterion validity of forced-choice comparisons of the quality of written arguments with normative solutions. Across two studies, novices and experts assessing quality of reasoning through a forced-choice design were both able to choose arguments supporting more accurate solutions—62.2% (SE = 1%) of the time for novices and 74.4% (SE = 1%) for experts—and arguments produced by larger teams—up to 82% of the time for novices and 85% for experts—with high inter-rater reliability, namely 70.58% (95% CI = 1.18) agreement for novices and 80.98% (95% CI = 2.26) for experts. We also explored two methods for increasing efficiency. We found that the number of comparative judgments needed could be substantially reduced with little accuracy loss by leveraging transitivity and producing quality-of-reasoning assessments using an AVL tree method. Moreover, a regression model trained to predict scores based on automatically derived linguistic features of participants’ judgments achieved a high correlation with the objective accuracy scores of the arguments in our dataset. Despite the inherent subjectivity involved in evaluating differing quality of reasoning, the forced-choice paradigm allows even novice raters to perform beyond chance and can provide a valid, reliable, and efficient method for producing quality-of-reasoning assessments at scale.

Learning by Evaluating (LbE): promoting meaningful reasoning in the context of engineering design thinking using Adaptive Comparative Judgment (ACJ)

Learning by Evaluating (LbE): promoting meaningful reasoning in the context of engineering design thinking using Adaptive Comparative Judgment (ACJ)

The influence of natural image statistics on upright orientation judgements

Humans have well-documented priors for many features present in nature that guide visual perception. Despite being putatively grounded in the statistical regularities of the environment, scene priors are frequently violated due to the inherent variability of visual features from one scene to the next. However, these repeated violations do not appreciably challenge visuo-cognitive function, necessitating the broad use of priors in conjunction with context-specific information. We investigated the trade-off between participants' internal expectations formed from both longer-term priors and those formed from immediate contextual information using a perceptual inference task and naturalistic stimuli. Notably, our task required participants to make perceptual inferences about naturalistic images using their own internal criteria, rather than making comparative judgements. Nonetheless, we show that observers' performance is well approximated by a model that makes inferences using a prior for low-level image statistics, aggregated over many images. We further show that the dependence on this prior is rapidly re-weighted against contextual information, even when misleading. Our results therefore provide insight into how apparent high-level interpretations of scene appearances follow from the most basic of perceptual processes, which are grounded in the statistics of natural images.

Comparative judgement for experimental philosophy: A method for assessing ordinary meaning in vehicles in the park cases

ABSTRACT This paper demonstrates the value to experimental philosophy of an empirical method from the social sciences – that of comparative judgment. Comparative judgment is a method of assigning scores to (perceptions of) objects using paired comparisons. We use this method to explore the “ordinary meaning” of words, and the classic case of vehicles in the park in particular. We present an empirical study comprising three conditions. Given a pair of potential vehicles, participants were asked to judge either 1) the better example of a vehicle, 2) the worse violation of a sign that reads “no vehicles in the park”, or 3) the bigger nuisance in a park. We find that both the meaning of the wording of the rule and the intention behind it influence participants judgments of rule-violations, consistent with previous studies. More importantly, comparative judgment provides more fine-grained information about agreement and the weighted rankings of the potential vehicles than other methods, with widespread potential applications in experimental philosophy.

Mentor teachers’ professional vision: A study of the differences with classroom teachers and student teachers

ABSTRACT A crucial competence for mentor teachers is the ability to analyse classroom practices as they are expected to model effective teaching practices and to provide feedback to student teachers. This ability is referred to in the literature as professional vision. The present study assesses mentor teachers’ (n = 137) professional vision regarding teacher-student interactions and differentiated instruction, using a validated video-based comparative judgement measurement instrument. The results indicate that mentor teachers have a high professional vision. It can thus be assumed that mentor teachers can support student teachers. Additionally, their professional vision is compared with that of classroom teachers (n = 996) and student teachers (n = 2168), expecting it to be significantly higher than that of classroom teachers and student teachers. The results show no significant difference between mentor teachers and classroom teachers but a significant difference between mentor teachers and student teachers. Hence, mentor teachers and classroom teachers are equally able to identify and interpret crucial aspects of effective teaching behaviour and both groups are better able than student teachers in this regard. This study contributes to the current state of the art on mentor teachers from a theoretical, empirical and methodological point of view.

The model selection of propeller turbine construction using Analytical Hierarchy Process (AHP)

This study aimed to develop an innovative propeller turbine design to facilitate easy manufacturing and maintenance processes, leading to a reduction in costs. Furthermore, the Analytical Hierarchy Process (AHP) method was employed to identify the most optimal model and design for the propeller turbine. Problem-solving within the AHP framework was guided by three fundamental principles, namely decomposition, Comparative Judgment, and Logical Consistency. The procedure included problem decomposition, assessment/ weighting to compare elements, matrix preparation and consistency testing, setting priorities for each hierarchy, priority synthesis, and decision-making. To establish a benchmark, three types of propeller turbines currently available in the market served as references. Meanwhile, the selection criteria for the model were based on several factors, including power factor, time efficiency, ease of manufacture, as well as production and maintenance costs. Considering the criteria, modifications were made to these reference models, resulting in the development of alternatives, denoted as A, B, and C. The results showed that alternative type A as the most suitable choice for further development. Therefore, this particular design was granted foremost priority to develop a low-head generator that possessed ease of manufacturing and surpassed alternative models in terms of feasibility.

Comparative approaches to the assessment of writing: Reliability and validity of benchmark rating and comparative judgement

In the past years, comparative assessment approaches have gained ground as a viable method to assess text quality. Instead of providing absolute scores to a text as in holistic or analytic scoring methods, raters in comparative assessments rate text quality by comparing texts either to pre-selected benchmarks representing different levels of writing quality (i.e., benchmark rating method) or by a series of pairwise comparisons to other texts in the sample (i.e., comparative judgement; CJ). In the present study, text quality scores from the benchmarking method and CJ are compared in terms of their reliability, convergent validity and scoring distribution. Results show that benchmark ratings and CJ-ratings were highly consistent and converged to the same construct of text quality. However, the distribution of benchmark ratings showed a central tendency. It is discussed how both methods can be integrated and used such that writing can be assessed reliably, validly, but also efficiently in both writing research and practice.

Would an unapportioned US federal wealth tax be constitutional, and what does that mean?

AbstractTo assess the constitutionality of an unapportioned federal wealth tax, and/or of a federal income tax provision reaching wealthy taxpayers’ unrealized gains, one needs an underlying framework for making judgements about legal claims. While no such framework can be entirely specified, at least to general agreement, this does not support nihilistically rejecting all comparative judgements about better versus worse, or more versus less convincing, instances of legal analysis.While it appears nearly certain that the Supreme Court’s current right-wing majority would strike down an unapportioned federal wealth tax, the ‘correct’ answer to this constitutional question cannot be proven to general agreement. I myself would disagree with such a holding by the Court. An important variable in resolving the issue for oneself is how much continuing precedential weight one should give to Pollock v. Farmers’ Loan & Trust Co.—a case that itself blatantly disrespected precedent, but that did so more than a century ago, and that (on the other hand) some argue has already ceased to be good law. The principle of respecting precedent both has instrumental value and has been long (if fitfully) honoured within the American legal system, but all agree that its weight is not absolute.An unapportioned minimum income tax on the wealthiest Americans that included in income their unrealized capital gains would likely be constitutional under currently prevalent legal doctrine. It is true, however, that Eisner v. Macomber (1920), if it were held to remain good law beyond its immediate facts, would support holding it unconstitutional. The prospect that the current Supreme Court’s six right-wingers will decide to revive Macomber provides only one reason for suspecting that they might strike down such a provision, perhaps while also launching a broader constitutional war against central tenets of the current regime for taxing wealthy individuals and capital income. However, it lies beyond the power of conventional legal analysis to predict either what form such a war would take, or on what terms it might end up being resolved.

Comparative judgement in education research

ABSTRACT Educational researchers often need to construct precise and reliable measurement scales of complex and varied representations such as participants’ written work, videoed lesson segments and policy documents. Developing such scales using can be resource-intensive and time-consuming, and the outcomes are not always reliable. Here we present alternative methods based on comparative judgement (CJ) that have been growing in popularity over recent years. We consider the contexts in which CJ-based methods are appropriate before describing in detail what they are and how they can be applied to construct measurement scales in a range of educational research contexts. We also provide an overview to evaluating the reliability and validity of the resultant measurement scales.

How we compare areas: The underlying mechanism of the elongation bias.

Across four experiments, we investigate the mechanism that underlies the elongation bias. We find individuals tasked with assessing the area of two objects do so by comparing the objects' dimensions, and thus subtle changes in the objects' dimensions can impact area assessments. Because a typical elongation bias experiment places two objects side-by-side horizontally and varies the elongation ratio while maintaining the same area, height is generally easier to compare than width. Thus, there will exist a region where the change in height noticeably crosses a perceptual just noticeable difference boundary, but the corresponding change in width does not, and individuals will tend to perceive that the taller object has a greater area or volume. Consistent with this proposed process, we suggest that, although the elongation bias occurs under a comparative judgment, it does not do so under a single judgment situation. This research contributes to our wider understanding of the visual processes underlying area comparisons.

Developmental Trajectory of Anticipation: Insights from Sequential Comparative Judgments.

Reaction time (RT) is a critical measure of performance, and studying its distribution at the group or individual level provides useful information on the cognitive processes or strategies used to perform a task. In a previous study measuring RT in children and adults asked to compare two successive stimuli (quantities or words), we discovered that the group RT distribution was bimodal, with some subjects responding with a mean RT of around 1100 ms and others with a mean RT of around 500 ms. This bimodal distribution suggested two distinct response strategies, one reactive, the other anticipatory. In the present study, we tested whether subjects' segregation into fast and slow responders (1) extended to other sequential comparative judgments (2) evolved from age 8 to adulthood, (3) could be linked to anticipation as assessed using computer modeling (4) stemmed from individual-specific strategies amenable to instruction. To test the first three predictions, we conducted a distributional and theoretical analysis of the RT of 158 subjects tested earlier using four different sequential comparative judgment tasks (numerosity, phonological, multiplication, subtraction). Group RT distributions were bimodal in all tasks, with the two strategies differing in speed and sometimes accuracy too. The fast strategy, which was rare or absent in 8- to 9-year-olds, steadily increased through childhood. Its frequency in adolescence remained, however, lower than in adulthood. A mixture model confirmed this developmental evolution, while a diffusion model corroborated the idea that the difference between the two strategies concerns anticipatory processes preceding decision processes. To test the fourth prediction, we conducted an online experiment where 236 participants made numerosity comparisons before and after an instruction favoring either reactive or anticipatory responses. The results provide out-of-the-lab evidence of the bimodal RT distribution associated with sequential comparisons and demonstrated that the proportions of fast vs. slow responders can be modulated simply by asking subjects to anticipate or not the future result of the comparison. Although anticipation of the future is as important for cognition as memory of the past, its evolution after the first year of life is much more poorly known. The present study is a step toward meeting this challenge. It also illustrates how analyzing individual RT distributions in addition to group RT distributions and using computational models can improve the assessment of decision making cognitive processes.

Leveraging supplementary modalities in automated real estate valuation using comparative judgments and deep learning

PurposeIn this study the authors aim to outline new ways of information extraction for automated valuation models, which in turn would help to increase transparency in valuation procedures and thus contribute to more reliable statements about the value of real estate.Design/methodology/approachThe authors hypothesize that empirical error in the interpretation and qualitative assessment of visual content can be minimized by collating the assessments of multiple individuals and through use of repeated trials. Motivated by this problem, the authors developed an experimental approach for semi-automatic extraction of qualitative real estate metadata based on Comparative Judgments and Deep Learning. The authors evaluate the feasibility of our approach with the help of Hedonic Models.FindingsThe results show that the collated assessments of qualitative features of interior images show a notable effect on the price models and thus over potential for further research within this paradigm.Originality/valueTo the best of the authors’ knowledge, this is the first approach that combines and collates the subjective ratings of visual features and deep learning for real estate use cases.

The Effects of Concreteness on Mathematical Manipulative Choice

AbstractThere is mixed evidence as to whether concrete manipulatives (e.g., toy animals) are better than abstract manipulatives (e.g., counters) for teaching mathematical concepts to children. Concreteness is defined as the amount of extraneous information a manipulative provides, and in this study we aimed to unpick which dimensions of concreteness influence manipulative choice. Researchers, teachers, and parents completed a comparative judgment task comparing images of manipulatives varying in different dimensions of concreteness, selecting which they would choose to teach arithmetic to children. The findings indicated homogeneous, 3‐dimensional manipulatives were the most preferred across all groups to teach arithmetic to children, regardless of more extraneous features. This contradicts research recommendations to minimize the use of concrete manipulatives due to their distractive qualities. Instead, it suggests that some concrete features may be preferred in more naturalistic contexts. More research is required to investigate how different dimensions of concreteness influence learning outcomes for children both in artificial research contexts and in practice.

An extended ISM and MICMAC method under neutrosophic environment

PurposeISM technique is one of the tools of interest in soft operations research. The soft nature of this technique has made inevitable use of indeterminacy theories. The present research attempts to develop ISM technique and MICMAC analysis in a neutrosophic space due to the complexity and uncertainty of the decision-making environment.Design/methodology/approachIn this study, single-valued triangular neutrosophic numbers is used to develop Neutrosophic ISM (NISM) and Neutrosophic MICMAC (NMICMAC). First, the general algorithm of NISM and NMICMAC is provided. In the following, the complete description of NISM steps including level value determination, Factor Leveling Algorithm and NISM digraph algorithm are presented. Finally, a description of the NMICMAC steps is described.FindingsAn illustrative example – supplier selection problem – is given to verify the effectiveness of the proposed method and in the discussion section; the comparison and analysis of different aspects of the NISM with the previous methods are discussed.Originality/valueIn this study, NISM and NMICMAC are presented for the first time, so that each pairwise comparison judgment is provided as single valued triangular neutrosophic numbers. The development of the model is continued until the final stages of calculations with neutrosophic numbers, and only in the final stage the results are crispy presented. In addition, not only the factors of process are leveled, but at each level the factors are lined up and their importance is determined.

Determinants of First College Major Choice among Yarmouk University Students and Estimating their Scaling Values Using Three Scaling Methods

The study aimed at determining the determinants of first college major choice for Yarmouk University students, and estimating their scaling values using three different scaling methods, and determining the extent of agreement among these methods. To achieve the purpose of the study, (165) male and female students from the university colleges volunteered to determine the determinants primarily, and two equivalent samples with regard to gender and college type were used to collect the scaling data. The first consisted of (120) male and female students studying a major which is their first choice, the second consisted of (120) male and female students studying a major differs from their first choice. The scaling values on importance scale were estimated using the Law of Comparative Judgments, Direct Estimation and the Law of Catogerical Judgments. The results showed that the most important major choice determinants are: opportunities to obtain work, secondary school exam scores mean, personal desire, and admission opportunities. Also, it was found that there was a consistency among scaling methods in ranking the determinants on the importance scale.

Comparative Optimism and Perceptions of Specific Online Risks

Adults hold optimistic beliefs for online behaviours, displaying a general tendency to believe that they are less likely to experience general risks than others. Study 1 explored whether young adults displayed comparative optimism judgements for four types of specific risks that were relevant to a UK sample. Study 2 addressed methodological weaknesses associated with assessing comparative optimism. Study 1 explored 227 (194 female, 31 male, 1 non-binary, and 1 gender not disclosed) young adults’ (MAge = 20.75, SDAge = 4.22) comparative optimistic beliefs for unwanted contact, hoaxes, behavioural, and sexual online risks that were contextualised to the UK. In study 2, young adults (134 female, 30 male, 1 non-binary, and 1 gender fluid, MAge = 20.72, SDAge = 3.39) provided judgements for four online risks for the general public and the comparator groups used in study 1. Participants were then provided with the prevalence rate for each risk according to data for the UK and then asked to repeat the judgments for each comparator group. Study 1 identified optimistic beliefs across the four risks, with family and the self judged to be less at risk. Study 2 revealed that again family was perceived to be at the lowest risk followed by the self. Knowledge of the actual risk led to a reduction in perceived risk for unwanted contact and hoaxes but little change for behavioural or sexual risks. Together, the findings have implications for understanding perceptions of online risk and how campaigns to promote digital safety are designed.

An Item Response Theory Model for Incorporating Response Times in Forced-Choice Measures

Forced-choice (FC) measures have been widely used in many personality or attitude tests as an alternative to rating scales, which employ comparative rather than absolute judgments. Several response biases, such as social desirability, response styles, and acquiescence bias, can be reduced effectively. Another type of data linked with comparative judgments is response time (RT), which contains potential information concerning respondents’ decision-making process. It would be challenging but exciting to combine RT into FC measures better to reveal respondents’ behaviors or preferences in personality measurement. Given this situation, this study aims to propose a new item response theory (IRT) model that incorporates RT into FC measures to improve personality assessment. Simulation studies show that the proposed model can effectively improve the estimation accuracy of personality traits with the ancillary information contained in RT. Also, an application on a real data set reveals that the proposed model estimates similar but different parameter values compared with the conventional Thurstonian IRT model. The RT information can explain these differences.

Research on the construction of a collaborative ability evaluation system for the joint graduation design of new engineering specialty groups based on digital technology

Conducting research on the construction of a collaborative ability evaluation system for the joint graduation design of new engineering specialty groups based on digital technology holds great practical relevance. In this paper, which is based on a comprehensive analysis and research of the current situation pertaining to the joint graduation design of college graduates in China and elsewhere and on the construction of a collaborative ability evaluation system, combined with the talent training program of the joint graduation design, the Delphi method and the analytic hierarchy process (AHP) are adopted to establish a hierarchical structure model of the collaborative ability evaluation system for joint graduation design. In this system, collaborative abilities in the areas of cognition, behavior and emergency management are used as the criteria level evaluation indices. Additionally, collaborative ability in regard to targets, to knowledge, to relationships, to software, to the workflow, to organization, to culture, to learning and to conflict are used as evaluation indices. The comparison judgment matrix of the evaluation indices is constructed at the collaborative ability criterion level and at the index level. By calculating the maximum eigenvalue and corresponding eigenvector of the judgment matrix, the weight assignment of the evaluation indices is obtained, and the evaluation indices are sorted. Finally, the related research content is evaluated. The research results show that the key evaluation indicators for the collaborative ability evaluation system of joint graduation design that need to be considered are easy to determine, and these indicators provide a theoretical reference for the reform of graduation design teaching of new engineering specialty groups.

Pedagogical features of interactive apps for effective learning of foundational skills

Interactive apps are commonly used to support the acquisition of foundational skills. Yet little is known about how pedagogical features of such apps affect learning outcomes, attainment and motivation—particularly when deployed in lower‐income contexts, where educational gains are most needed. In this study, we analyse which app features are most effective in supporting the acquisition of foundational literacy and numeracy skills. We compare five apps developed for the Global Learning XPRIZE and deployed to 2041 out‐of‐school children in 172 remote Tanzanian villages. A total of 41 non‐expert participants each provided 165 comparative judgements of the five apps from the competition, across 15 pedagogical features. Analysis and modelling of these 6765 comparisons indicate that the apps created by the joint winners of the XPRIZE, who produced the greatest learning outcomes over the 15‐month field trial, shared six pedagogical features—autonomous learning, motor skills, task structure, engagement, language demand and personalisation. Results demonstrate that this combination of features is effective at supporting learning of foundational skills and has a positive impact on educational outcomes. To maximise learning potential in environments with both limited resources and deployment opportunities, developers should focus attention on this combination of features, especially for out‐of‐school children in low‐ and middle‐income countries. Practitioner notesWhat is already known about this topic Interactive apps are becoming common to support foundational learning for children both in and out of school settings. The Global Learning XPRIZE competition demonstrates that learning apps can facilitate learning improvements in out‐of‐school children living in sub‐Saharan Africa. To understand which app features are most important in supporting learning in these contexts, we need to establish which pedagogical features were shared by the winning apps. What this paper adds Effective learning of foundational skills can be achieved with a range of pedagogical features. To maximise learning, apps should focus on combining elements of autonomous learning, motor skills, task structure, engagement, language demand and personalisation. Free Play is not a key pedagogical feature to facilitate learning within this context. Implications for practice and/or policy When developing learning apps with primary‐aged, out‐of‐school children in low‐income contexts, app developers should try to incorporate the six key features associated with improving learning outcomes. Governments, school leaders and parents should use these findings to inform their decisions when choosing an appropriate learning app for children.