Comparative Judgments Research Articles

Do mathematicians and undergraduates agree about explanation quality?

Offering explanations is a central part of teaching mathematics, and understanding those explanations is a vital activity for learners. Given this, it is natural to ask what makes a good mathematical explanation. This question has received surprisingly little attention in the mathematics education literature, perhaps because the field has no agreed method by which explanation quality can be reliably assessed. In this paper, we explore this issue by asking whether mathematicians and undergraduates agree with each other about explanation quality. A corpus of 10 explanations produced by 10 mathematicians was used. Using a comparative judgement method, we analysed 320 paired comparisons from 16 mathematicians and 320 from 32 undergraduate students. We found that both mathematicians and undergraduates were able to reliably assess the quality of a set of mathematical explanations. Furthermore, the assessments were largely consistent across the two groups. Implications for theories of mathematical explanation are discussed. We conclude by arguing that comparative judgement is a promising technique for exploring explanation quality.

Peer overmarking and insufficient diagnosticity: the impact of the rating method for peer assessment.

The present study explores two rating methods for peer assessment (analytical rating using criteria and comparative judgement) in light of concurrent validity, reliability and insufficient diagnosticity (i.e. the degree to which substandard work is recognised by the peer raters). During a second-year undergraduate course, students wrote a one-page essay on an air pollutant. A first cohort (N = 260) relied on analytical rating using criteria to assess their peers' essays. A total of 1297 evaluations were made, and each essay received at least four peer ratings. Results indicate a small correlation between peer and teacher marks, and three essays of substandard quality were not recognised by the group of peer raters. A second cohort (N = 230) used comparative judgement. They completed 1289 comparisons, from which a rank order was calculated. Results suggest a large correlation between the university teacher marks and the peer scores and acceptable reliability of the rank order. In addition, the three essays of substandard quality were discerned as such by the group of peer raters. Although replication research is warranted, the results provide the first evidence that, when peer raters overmark and fail to identify substandard work using analytical rating with criteria, university teachers may consider changing the rating method of the peer assessment to comparative judgement.

Untypical Contrast Normalization Explains the "Weak Outnumber Strong" Numerosity Illusion.

Less salient, lower contrast disks appear to be more numerous than more salient, higher contrast disks when intermingled in equal numbers into the same display (Lei and Reeves, 2018), but they are equal in perceived numerosity when segregated into different displays. Comparative judgements indicate that the apparent numerosity of the lower contrast disks is unaffected by being intermingled with high contrast disks, whereas the high contrast disks are reduced in numerosity by being intermingled with the low contrast ones (Lei and Reeves, 2018). Here, we report that this illusion also occurs for absolute judgements of the numerosities of displays of from 20 to 80 disks. A model based on luminance-difference contrast normalization (LDCN) explains the illusory loss of high-contrast (salient) items along with veridical perception of the low-contrast ones. The model correctly predicts that perceived numerosity is linearly related to the square-root of the number of disks, with the extent of the illusion depending on an attentionally-weighted function of contrast and assimilation.

Text Mining to Alleviate the Cold-Start Problem of Adaptive Comparative Judgments

Comparative judgments permit the assessment of open-ended student works by constructing a latent quality scale through repeated pairwise comparisons (i.e., which works “win” or “lose”). Adaptive comparative judgments speed up the judgment process by maximizing the Fisher information of the next comparison. However, at the start of a judgment process, such an adaptive algorithm will not perform well. In order to reliably approximate the Fisher Information of possible pairs well, multiple comparisons are needed. In addition, adaptive comparative judgments have been shown to inflate the scale separation coefficient, which is a reliability estimator for the quality estimates. Current methods to solve the inflation issue increase the number of required comparisons. The goal of this study is to alleviate the cold-start problem of adaptive comparative judgments for essays or other textual assignments, but also to minimize the bias of the scale separation coefficient. By using text-mining techniques, which can be performed before the first judgment, essays can be adaptively compared from the start. More specifically, we propose a selection rule that is based both on a high (1) cosine similarity of the vector representations and (2) Fisher Information of essay pairs. At the start of the judgment process, the cosine similarity has the highest weight in the selection rule. With more judgments, this weight decreases progressively, whereas the weight of the Fisher Information increases. Using simulated data, the proposed strategy is compared with existing approaches. The results indicate that the proposed selection rule can mitigate both the cold-start. That is, fewer judgments are needed to obtain accurate and reliable quality estimates. In addition, the selection rule was found to reduce the inflation of the scale separation reliability.

Using cognitive discourse functions and comparative judgement to build teachers’ knowledge of content and language integration for assessment in a bilingual education program

Abstract This paper reports a study carried out in the context of a seminar designed to build teachers’ knowledge of content and language integration for assessment in a bilingual education program in Madrid, Spain. The participants were seven teachers (two primary, five secondary) of English and science (primary), and art, history, Spanish language arts and English (secondary). The teachers were introduced to the concept of cognitive discourse function (CDF) and assessed samples of students’ work expressing the functions define, evaluate, and explore, using comparative judgement. In sharing and justifying their assessments, they articulated the criteria they used to take their decisions. The findings show that participants emphasized content quality over quantity and language form over function. There was also evidence that the concept of CDFs enabled them to express new understandings of the content-language relationship in assessment. The study has implications for building the knowledge base for content and language integrated assessment across CBLE programs.

Linguistically based scales for assessment of young students’ writing

This study addresses the question of how different aspects of students’ writing achievement can be recognised and evaluated. We developed a linguistically based framework for criteria-based assessment, anchored in a functional view of language and language learning. The framework was used to determine what traits characterise texts at different Proficiency Groups based on comparative judgement and what traits characterise texts assessed differently. Altogether, 100 texts (written by students ages 6–9) representing four text genres were assessed and ranked using both comparative judgement (holistic assessment) and criteria-based analysis. The results indicate that texts generally are assessed as stronger (i.e., placed in a higher Proficiency Group) when comparative judgement is used than what the assessment of a specific language resource indicates. The results also indicate that assessment differences might be a result of different quality expectations for different genres. This points towards the need for genre- and subject-specific assessment criteria to scaffold students in their emergent disciplinary writing development.

The lasting effects of sandwiched events

ABSTRACT It’s been reported that the presentation of auditory flickering stimuli (click trains) induces time dilation. However, this effect could be associated with the presence rather than the discontinuity of the auditory signal. We hypothesized that based on the internal clock models, flickering would further manipulate the speed of the internal clock. To this end, we examined the impact of sandwiched events (click train, auditory continuous stimulus, or no sound) and a very short gap (150 ms) between target and comparison interval) on duration perception. Our results showed that regardless of the type of the sound, an addition of an auditory stimulus improves the accuracy and response time of comparison judgment, which can be interpreted as a result of increasing the speed of the pacemaker. Besides, click train has a stronger footprint on time dilation than a continuous sound. We conclusively ascertain that the effect of dilation does not wear off within the applied gap.

Counterfactual Reasoning in Art Criticism

Abstract When we evaluate artworks, we often point to what an artist could have done or what a work could have been in order to say something about the work as it actually is. Call this counterfactual reasoning in art criticism. On my account, counterfactual claims about artworks involve comparative aesthetic judgments between actual artworks and hypothetical variations of those works. The practice of imagining what an artwork could have been is critically useful because it can help us understand how artworks achieve specific aesthetic effects. I conclude by responding to an objection to my account on the basis that it violates the widely accepted acquaintance principle in aesthetics, on which aesthetic judgments must be based on firsthand perceptual encounters with their objects.

Validity of Comparative Judgment Scores: How Assessors Evaluate Aspects of Text Quality When Comparing Argumentative Texts

The advantage of comparative judgment is that it is particularly suited to assess multidimensional and complex constructs as text quality. This is because assessors are asked to compare texts holistically and to make a quality judgment for each text in a pairwise comparison based upon on the most salient and critical differences. Also, the resulted rank order is based on the judgment of all assessors, representing the shared consensus. In order to be able to select the right number of assessors, the question is to what extent the conceptualization of assessors prevails in the aspects they base their judgment on, or whether comparative judgment minimizes the differences between assessors. In other words, can we detect types of assessors who tend to consider certain aspects of text quality more often than others? A total of 64 assessors compared argumentative texts, after which they provided decision statements on what aspects of text quality had informed their judgment. These decision statements were coded on six overarching themes of text quality: argumentation, organization, language use, language conventions, source use, references, and layout. Using a multilevel-latent class analysis, four different types of assessors could be distinguished: narrowly focused, broadly focused, source-focused, and language-focused. However, the analysis also showed that all assessor types mainly focused on argumentation and organization, and that assessor types only partly explained whether the aspect of text quality was mentioned in a decision statement. We conclude that comparative judgment is a strong method for comparing complex constructs like text quality. First, because the rank order combines different views on text quality, but foremost because the method of comparative judgment minimizes differences between assessors.

Using computerised comparative judgement to assess translation

Abstract Translation assessment represents a productive line of research in Translation Studies. An array of methods has been trialled to assess translation quality, ranging from intuitive assessment to error analysis and from rubric scoring to item-based assessment. In this article, we introduce a lesser-known approach to translation assessment called comparative judgement. Rooted in psychophysical analysis, comparative judgement grounds itself on the assumption that humans tend to be more accurate in making relative judgements than in making absolute judgements. We conducted an experiment, as both a methodological exploration and a feasibility investigation, in which novice and experienced judges were recruited to assess English-Chinese translation, using a computerised comparative judgement platform. The collected data were analysed to shed light on the validity and reliability of assessment results and the judges’ perceptions. Our analysis shows that (1) overall, comparative judgement produced valid measures and facilitated judgement reliability, although such results seemed to be affected by translation directionality and judges’ experience, and (2) the judges were generally confident about their decisions, despite some emergent factors undermining the validity of their decision making. Finally, we discuss the use of comparative judgement as a possible method in translation assessment and its implications for future practice and research.

Crowdsourced Adaptive Comparative Judgment: A Community‐Based Solution for Proficiency Rating

AbstractThe main objective of this Methods Showcase Article is to show how the technique of adaptive comparative judgment, coupled with a crowdsourcing approach, can offer practical solutions to reliability issues as well as to address the time and cost difficulties associated with a text‐based approach to proficiency assessment in L2 research. We showcased this method by reporting on the methodological framework implemented in the Crowdsourcing Language Assessment Project and by presenting the results of a first study that demonstrated that a crowd is able to assess learner texts with high reliability. We found no effect of language skills or language assessment experience on the assessment task, but judges who had received formal language assessment training seemed to differ in their decisions from judges who had not received such training. However, the scores generated by the crowdsourced task exhibited a strong positive correlation with the rubric‐based scores provided with the learner corpus used.

The Accuracy and Validity of the Simplified Pairs Method of Comparative Judgement in Highly Structured Papers

Comparative judgement (CJ) is often said to be more suitable for judging exam questions inviting extended responses, as it is easier for judges to make holistic judgements on a small number of large, extended tasks than a large number of smaller tasks. On the other hand, there is evidence it may also be appropriate for judging responses to papers made up of many smaller structured tasks. We report on two CJ exercises on mathematics and science exam papers, which are constructed mainly of highly structured items. This is to explore whether judgements processed by the simplified pairs version of CJ can approximate the empirical difference in difficulty of pairs of papers. This can then be used to maintain standards between exam papers. This use of CJ, not its other use as an alternative to marking, is the focus of this paper. Within the exercises discussed, panels of experienced judges looked at pairs of scripts, from different sessions of the same test, and their judgements were processed via the simplified pairs CJ method. This produces a single figure for the estimated difference in difficulty between versions. We compared this figure to the difference obtained from traditional equating, used as a benchmark. In the mathematics study the difference derived from judgement via simplified pairs closely approximated the empirical equating difference. However, in science, the CJ outcome did not closely align with the empirical difference in difficulty. Reasons for the discrepancy may include the differences in the content of the exams or the specific judges. However, clearly, comparative judgement need not lead to an accurate impression of the relative difficulty of different exams. We discuss self-reported judge views on how they judged, including what questions they focused on, and the implications of these for the validity of CJ. Processes used when judging papers made up of highly structured tasks were varied, but judges were generally consistent enough. Some potential challenges to the validity of comparative judgement are present with judges sometimes using re-marking strategies, and sometimes focusing attention on subsets of the paper, and we explore these. A greater understanding of what judges are doing when they judge comparatively brings to the fore questions of judgement validity that remain implicit in marking and non-comparative judgement contexts.

Exploring the Validity of Comparative Judgement: Do Judges Attend to Construct-Irrelevant Features?

When completing a comparative judgment (CJ) exercise, judges are asked to make holistic decisions about the quality of the work they are comparing. A key consideration is the validity of expert judgements. This article details a study where an aspect of validity, whether or not judges are attending to construct-irrelevant features, was investigated. There are a number of potentially construct-irrelevant features indicated in the assessment literature, and we focused on four features: appearance; handwriting; spelling, punctuation, and grammar (SPaG); and missing response vs. incorrect answer. This study explored this through an empirical experiment supplemented by judge observation and survey. The study was conducted within an awarding organisation. The particular context was within a programme of work trialling, a new method of maintaining examination standards involving the comparative judgement of candidates’ examination responses from the same subject from two different years. Judgements in this context are cognitively demanding, and there is a possibility that judges may attend to superficial features of the responses they are comparing. It is, therefore, important to understand how CJ decisions are made and what they are or are not based on so that we can have confidence in judgements and know that any use of them is valid.

Effective uncertainty visualization for aftershock forecast maps

Abstract. Earthquake models can produce aftershock forecasts, which have recently been released to lay audiences. While visualization literature suggests that displaying forecast uncertainty can improve how forecast maps are used, research on uncertainty visualization is missing from earthquake science. We designed a pre-registered online experiment to test the effectiveness of three visualization techniques for displaying aftershock forecast maps and their uncertainty. These maps showed the forecasted number of aftershocks at each location for a week following a hypothetical mainshock, along with the uncertainty around each location's forecast. Three different uncertainty visualizations were produced: (1) forecast and uncertainty maps adjacent to one another; (2) the forecast map depicted in a color scheme, with the uncertainty shown by the transparency of the color; and (3) two maps that showed the lower and upper bounds of the forecast distribution at each location. We compared the three uncertainty visualizations using tasks that were specifically designed to address broadly applicable and user-generated communication goals. We compared task responses between participants using uncertainty visualizations and using the forecast map shown without its uncertainty (the current practice). Participants completed two map-reading tasks that targeted several dimensions of the readability of uncertainty visualizations. Participants then performed a Comparative Judgment task, which demonstrated whether a visualization was successful in reaching two key communication goals: indicating where many aftershocks and no aftershocks are likely (sure bets) and where the forecast is low but the uncertainty is high enough to imply potential risk (surprises). All visualizations performed equally well in the goal of communicating sure bet situations. But the visualization with lower and upper bounds was substantially better than the other designs at communicating surprises. These results have implications for the visual communication of forecast uncertainty both within and beyond earthquake science.

An Analytic Hierarchy Process (AHP) Framework for Selecting Flexible Manufacturing System

The main objective of this paper is to propose a structured model for evaluating different Flexible Manufacturing System (FMS) using the Analytic Hierarchy Process (AHP). The paper aims to demonstrate how the model can help in taking correct decisions regarding the type of manufacturing flexibility needed for best FMS. This is because huge amount of money is invested to implement FMS in the industry. Hence, managers need to take judicious decision regarding the type and level of manufacturing flexibility. There are a number of manufacturing flexibility being reported in the literature. In this paper the most fundamental types i.e., machine, routing and product flexibility are examined. The AHP is aimed at integrating different measures into a single overall score for ranking decision alternatives. Its main characteristic is that it is base on pairwise comparison judgment. A usability evaluation of the AHP based model of FMS along with structure of the hierarchy is developed. The framework that is used in this example could serve as one of the tools for making a strategic decision. The effectiveness of our model is demonstrated through numerical examples.

Security Assessment of Coal Mine Power Grid Voltage Based on an Improved AHP-FCE

The voltage level of a coal mine power grid directly affects the starting and normal operation of the electric motor, but the current grid monitoring system cannot use real-time grid structure parameters and power flow distribution data to conduct early warning research on the running status and voltage level of the power grid, so it cannot eliminate hidden danger in the operation of the power grid. Therefore, it is necessary to accurately and effectively evaluate the voltage safety level of the coal mine power grid. In this paper, an improved AHP-FCE method is proposed to evaluate the voltage safety level of a coal mine power grid. By replacing the comparison judgment matrix with the interval judgment matrix, the objectivity of the weight vector is improved while retaining expert subjectivity. According to the real-time power flow distribution data of the power grid monitoring system, the current voltage safety level of the power grid is evaluated. The results show that the improved AHP-FCE method can accurately evaluate the current grid voltage level, analyze the voltage development trend, give early warnings regarding coal mine grid voltage operation, and improve the monitoring function of the power monitoring system.

Professional vision as a mediator for inclusive education? Unravelling the interplay between teachers’ beliefs, professional vision and reported practice of differentiated instruction

ABSTRACT To ensure inclusion and equity in education, both teacher beliefs and practice are cited as influential. However, the connection between beliefs on student diversity and inclusive practice is often more complex than straightforward. Professional vision is theorised to be important in aligning beliefs with practice. Hence, this study investigates whether teachers’ professional vision of differentiated instruction (DI) mediates between teachers’ beliefs about teaching diverse learners and teachers’ practice of DI, and is unique in investigating these constructs in concert with each other. Data are part of the Potential-project and were collected in a sample of secondary education teachers (N = 461) in Flemish schools (N = 23). Survey data measuring teachers’ beliefs (i.e. growth mindset, professional beliefs about diversity and about differentiating the curriculum) and teachers’ self-reported DI practice were combined with video-based comparative judgement data measuring teachers’ professional vision of DI. Multilevel models show that both teachers with a more expert professional vision and less expert professional vision implement DI. For a more expert professional vision, beliefs about teaching diverse learners explain the association. For a less expert professional vision, teachers’ reflective and experimental practice explains the relationship. Results suggest that aspects of teachers’ competence are related in more complex ways than linear theoretical frameworks propose. Implications for teacher training and professionalisation are discussed.

Task and information conflicts in the numerical Stroop task

Studies of the Stroop color‐word task have provided evidence for the existence of two conflicts: (1) an early task conflict between noting the relevant color and reading afforded by the irrelevant word (or word‐like stimuli), and (2) a late information conflict between the information provided by the word and the information provided by the color. Measurements of pupil changes, in addition to reaction time (RT), have extended understanding regarding these two conflicts. The current work examines the generalizability of such understanding. We ask whether similar processes work in the comparative judgment of numbers (e.g., in the numerical Stroop task). We present two experiments that support and extend the knowledge gained in the word‐color context to numerical processing. Similar to results with the Stroop color‐word task, we found a dissociation between RT and pupillometry and an early task conflict followed by an information conflict.

Using Teachers’ Judgments of Quality to Establish Performance Standards in Technology Education Across Schools, Communities, and Nations

The establishment and maintenance of national examination standards remains a serious issue for teachers and learners, whilst the levers of control remain firmly in the hands of Awarding Bodies and supervising politicians. Significantly, holistic assessment presents an agility and collective approach to establishing in the minds of teachers “what is of value” when determining the comparative evidence of pupil performance. It is argued in this paper that the collation of the comparative judgment process can initially identify and subsequently maintain standards of performance that can be defined on a cluster, regional or even national level. Much comparative judgment research centers on the formative benefits for learners, but here we place the focus on teachers operating in collaborative groups to establish standards within and beyond their own schools, and ultimately across the nation. We model a proof-of-concept research project. A rank is produced by the collective consensus of the participating teachers and used to simulate a definition of standard. Extrapolations are statistically modeled to demonstrate the potential for this approach to establishing a robust definition of national standards. But central to the process is what is going on in the minds of teachers as they make their judgements of quality. The research aims to draw out teachers’ constructs of quality; to make them explicit; to share them across classrooms and schools; and to empower teachers to debate and agree their standards across schools. This research brings to the fore the symbiotic relationship between teaching, learning and assessment.

Travelers' Tales: Their Values and Virtues

Travelers' Tales: Their Values and Virtues