Law Of Comparative Judgment Research Articles

Assessing Faculty Performance for Merit: An Academic Accomplishment Index

This paper outlines a quantitative procedure developed for assessing the scholarly, teaching, and service accomplishments of faculty members being reviewed for promotion and tenure. This Academic Assessment Index (AAI) is based on the responses of 109 university and college administrators (chairpersons, department heads, and deans) in a national survey. Based on the Law of Comparative Judgment, the AAI was developed from a statistical analysis of these responses. The results may provide an important perspective on the difficult issue of how faculty career advancement portfolios including teaching, scholarly works, and service activities are fudged by senior colleagues and administrators.

PIN23: THE FIRST STEP TO GENERATING UTILITY FROM QUALITY OF LIFE: A COMPARISON OF ITEM PREFERENCES AND ITEM SEVERITY

OBJECTIVE: The Recurrent Genital Herpes Quality of Life questionnaire (RGHQoL) is a needs-based quality of life (QoL) instrument specifically for use with recurrent genital herpes (RGH). It has been shown to have excellent psychometric properties. However, it cannot currently be used in the calculation of utility for economic analyses. Existing utility instruments cover symptoms and functioning, rather than QoL. Concern has been expressed over their relevance for RGH and, consequently, their ability to show differences between interventions. The aim of this project is to explore the potential for using the RGHQoL to produce meaningful utility scores. The first stage was to investigate whether a preference elicitation exercise was required or whether RGHQoL responses could be incorporated directly into utility analyses. This paper presents findings from the first stage. METHODS: Structured interviews were conducted with 100 RGH patients. Patients undertook preference exercises (paired comparison and ranking) with six RGHQoL items and completed the RGHQoL. Thurstone’s law of comparative judgement was used to calculate item preference weights. These were compared with item severity weights derived via application of the Rasch model to RGHQoL responses. RESULTS: Item ordering of severity was: B-E-D-C-A-F (paired comparisons), B-C-D-E-A-F (ranking) and F-A-DC-E-B (Rasch severity). The paired comparisons and ranking exercises resulted in different ordering of some items, even though the methods are considered direct alternatives for eliciting preference information. The Rasch severity and preference weights provided very different orderings. For example, the most severe item according to patients’ responses to the RGHQoL was judged the least severe in the paired comparison and ranking situations. CONCLUSIONS: The results suggest that individuals’ ratings of what would have the most impact on them are very different from what actually has an impact. RGHQoL responses cannot be used directly in place of preference elicitation.

Information needs, uncertainty, and anxiety in women who had a breast biopsy with benign outcome.

A retrospective, descriptive study to determine the information needs of women who underwent a breast biopsy with a benign outcome and to ascertain the levels of uncertainty and anxiety they experienced was conducted in two community health care sites in Winnipeg, Manitoba. A sample of 70 women completed a four-part survey after learning the benign breast biopsy diagnosis. The survey consisted of an Information Needs Questionnaire, Mishel Uncertainty in Illness Scale--Community Form, State-Trait Anxiety Inventory, and a Demographic Questionnaire. Before the study, nine information needs were identified in a focus group composed of women (n = 9) who had a benign breast biopsy. The nine information needs were arranged in 36 pairs in the Information Needs Questionnaire. Profiles of information needs were developed through Statistical Analysis Systems analysis using Thurstone's Law of Comparative Judgement--Case V. The most important information need of women who underwent a benign breast biopsy was knowing when they would learn the diagnosis. The next four information needs were categorized as information about the risks of developing breast cancer. Information about follow-up and diagnostic tests were less important than information about the threat of breast cancer. Women experienced heightened uncertainty and anxiety levels related to the benign breast biopsy experience. Profiles of information needs were developed for women experiencing various levels of uncertainty and anxiety, women of different age groups, and women who experienced with a loved one with breast cancer.

Evaluation of image metrics for target discrimination using psychophysical experiments

Image clutter affects the perceptual ability of any system for object detection. A procedure for conducting psychophysical experiments has been developed to test computational models for the perceptual similarity or difference of texture patterns, which contributes to image clutter. This experimental procedure is based on Thurstone’s law of comparative judgment, which is used along with the method of paired comparisons to assign relative psychological scale values to image stimuli. To facilitate consistency in the presentation of stimuli and collection of data, an X-windows testing environment has been developed called the X-based perceptual experiment testbed. Using this experimental procedure, a pilot study was conducted in which the image stimuli consisted of targets and backgrounds with texture patterns of uncorrelated Gaussian noise. With such patterns, only first-order image statistics are of significance. The psychological scale values relating the level of ‘‘target distinctness’’ in each of the image stimuli were compared to several first-order image metrics. Correlation coefficients as high as 0.9881 were found between the scale values and the image metrics.

Cathode-ray-tube to reflection-print matching under mixed chromatic adaptation using RLAB

Color-appearance models are used to relate chromatic stimuli viewed under one set of viewing and illuminating conditions to a differing set such that when each stimulus is viewed in its respective conditions, the stimuli match in color appearance. These models assume the observer has a steady-state adaptation to each condition. In practice, observers often view stimuli under mixed adaptation; this could occur when viewing CRT and reflection-print stimuli simultaneously. A visual experiment was performed to determine whether the RLAB color-appearance model could be used successfully to generate reflection prints that match the appearance of the CRT when viewed under mixed states of adaptation and in turn as stand-alone images viewed under a single state of adaptation. Sixteen observers viewed four pictorial images displayed on a D 65 balanced CRT display in a room lit with cool-white fluorescent luminaries. The RLAB color-appearance model was used to calculate corresponding images where the observer's state of chromatic adaptation was assumed to be one of the following: adaptation to each device condition, a single adaptation at the midpoint of the two device conditions, adaptation to the CRT condition and a print adaptation shifted 25% toward the CRT condition, adaptation to the print condition and a CRT adaptation shifted 25% toward the print condition, and a CRT condition shifted 25% toward the print condition and a print condition shifted 25% toward the CRT condition. Each condition was compared pairwise and Thurstone's law of comparative judgments was used to calculate interval scales of quality. Observers first judged the reflection prints adjacent to the CRT display selecting the image closest in color appearance to the CRT image; they also categorized the closest image as marginally acceptable, or not acceptable. The images were again scaled except the display was turned off; this determined the best standalone color reproduction.

A law of comparative judgment.

A law of comparative judgment.

A law of comparative judgment.

A law of comparative judgment.

Classical Psychophysics and the Assessment of Agreement and Accuracy In Judgments of Personality

ABSTRACT The just noticeable difference (jnd) unit of classical psycho‐physics is introduced as a new way to describe accuracy and agreement in observer evaluations of personality. A formula for estimating jnd's from typically available summary statistics is derived from Thurstone's law of comparative judgment. A study examining four traits judged in 10 samples of subjects where the design permitted the calculation of jnd's by the method of paired comparisons indicated that the formula predicted the empirically derived jnd's associated with the mean judge ratings with considerable precision. Jnd's of criterion measures were also predicted, and while fit was somewhat less impressive in this case, there was still appreciable convergence between predicted and empirical values. The implications of jnd measures of agreement and accuracy are discussed. These implications include (a) possibilities for increased understanding of bias in observer judgments, (b) a new recognition that equal correlations to external criteria do not necessarily imply equal accuracy, and (c) alternative ways of describing the magnitude of effect in psychological research.

Realism of confidence in sensory discrimination: the underconfidence phenomenon.

This paper documents a very pervasive underconfidence bias in the area of sensory discrimination. In order to account for this phenomenon, a subjective distance theory of confidence in sensory discrimination is proposed. This theory, based on the law of comparative judgment and the assumption of confidence as an increasing function of the perceived distance between stimuli, predicts underconfidence--that is, that people should perform better than they express in their confidence assessments. Due to the fixed sensitivity of the sensory system, this underconfidence bias is practically impossible to avoid. The results of Experiment 1 confirmed the prediction of underconfidence with the help of present-day calibration methods and indicated a good quantitative fit of the theory. The results of Experiment 2 showed that prolonged experience of outcome feedback (160 trials) had no effect on underconfidence. It is concluded that the subjective distance theory provides a better explanation of the underconfidence phenomenon than do previous accounts in terms of subconscious processes.

A theory for individual preference of designing the sound field in a concert hall.

The scale value S [Y. Ando, ConcertHallAcoustics (Springer-Verlag, New York, 1985)], which is obtained by the law of comparative judgment and confirmed by the test of goodness of fit, may be inversely transformed into the original probability P(B≳A). Consider first of all the simplest case consisting of only two kinds of sound fields A and B in a concert hall, and if S=−1.0, then P(B≳A)=0.84 and P(B<A)=0.16. This means that 84% of listeners judge that sound field B is preferred over A, and 16% of listeners judge A as more preferred. For the purpose of obtaining preferred conditions for different listeners, such a relationship should be satisfied by the number of seats. In order to determine the number of seats in reference to that of the most preferred condition, for the majority of listeners, probability index (PI) is defined by PI=P(B<A)/P(B≳A), 0≤PI≤1.0. If P(B≳A)=0.5, then PI=1.0, which implies the most preferred condition. After construction of the concert hall, a seat selection system for each listener is used to enhance the listener’s satisfaction.

A Probabilistic IRT Model for Unfolding Preference Data

A probabilistic model is developed for the pair- comparison design in which the unfolding principle that governs the choice process uses a discriminal pro cess analogous to Thurstone's Law of Comparative Judgment. However, this process is governed by the square of the distance between the location of the per son and the stimulus, rather than controlled by the lo cation of the stimulus as in Thurstone's formulation. A simulation study demonstrates the feasibility of esti mation, and two examples use real data to show the implications of the unfolding models for psychological research. Index terms: choice data, item response theory, pair comparisons, preference data, unfolding.

The Application of an Unfolding Model of the PIRT Type to the Measurement of Attitude

Unfolding data for unidimensional variables con structed from direct responses (e.g., agreement or dis agreement) are characterized by single peaked func tions involving the locations of each person and each stimulus. A continuous discrirninal process, of the form postulated by Thurstone when he proposed his Law of Comparative Judgment, is suggested. This process is transformed to a qualitative dichotomous re sponse in which the probability of endorsement is governed by the square of the distance between the lo cations of the person and the stimulus. Maximum like lihood estimates of the parameters are derived, and it is shown that the information associated with any re sponse is a bimodal function of the difference between the person and stimulus locations. The feasibility of parameter estimation is demonstrated with a limited simulation study. The model is applied to a set of statements designed to measure attitudes toward capi tal punishment and scaled by the methods of Thur stone. The responses conformed to the unfolding mechanism, and the scale values of the statements are statistically equivalent to those obtained by Thur stone's methods. Index terms: Attitude measure ment, Developmental data, Discriminal process, Item response theory, Person response theory, Thurstone scaling, Unfolding data, Unidimensional scaling.

On the Use of Unreplicated Pairing in Forestry Planning

ABSTRACT The assessment of judgemental information belongs traditionally to a branch of psychology which is known as psychometrics or psychophysics. One of the most well-known psychometric “laws” is Thurstone's Law of Comparative Judgement which is based on the assumption that relative preferences among different criteria (“stimuli”) can be established if the criteria are lined up in pairs. Thurstone was a statistician and his approach requires replicated pairing. Foresters who use replicated pairing correctly assume that there is a population of judgements in a given set of criteria, i.e. that an individual's preferences are usually not consistent. Replicated pairing is, in theory, most desirable, but it can be very time consuming, and is therefore likely to be rejected by management staff. Unreplicated pairing, if used with circumspection and refinement, can be a realistic alternative for obtaining foresters preferences. Examples are presented to demonstrate the method.

Measuring Price and Quality Competition

Portfolio management requires precise measures of price and quality competition across relevant lines. A methodology is proposed that uses the Law of Comparative Judgment to transform ordinal comparisons between firms into reliable price and quality scales. Results are presented for five broadline manufacturers and fifteen subsidiaries competing in eight product classes in major appliances. The scale values and subsequent interpretations are substantiated by two major trade studies commissioned by the appliance industry. Implications for integrating quality considerations into the strategic planning process are discussed.

Prioritizing computer literacy topics

This article presents a process of prioritizing selected computer literacy topics as an initial step in implementing computer literacy programs for secondary schools. Ninety-five Nebraska educators formed the sample. A 45-item questionnaire was developed as the instrument. Educators were asked to choose more preferable topics among other topics for teaching secondary school students about computers. Thurstone's Law of Comparative Judgment was used to scale or locate educator's opinions about computer literacy topics on an interval level continuum. A general knowledge about computers was emphasized by educators surveyed. Hands-on experiences were viewed as necessary for students who want to be competent in using computers. Programming skills were suggested but not essential for computer literate students. Findings of this study provide implications of integrating computer literacy with existing curriculum. Results also provide guidelines for curriculum planners and persons responsible for the acquisition of hardware, software and staff development.

A probabilistic model for the discrimination of visual number.

This paper proposes a probabilistic model of how humans identify the number of dots within a briefly presented visual display. The model is an application of Thurstone’s law of comparative judgment, and it is assumed that the internal representation of numerosity consists of log-spaced random variables. The discrimination between any two different numerosities is consequently described as a function of max/rain, where max and rain are the larger and smaller numbers, respectively. The model was tested in two experiments in which the Weber fraction for numerosity, corresponding with the critical ratio of max and min, was found to have the value of 162. It was concluded that the classical span of subitizing numerosity is but a special case of the span of discrimination.

On the Law of Comparative Judgment: Scaling with Intransitive Observers and Multidimensional Stimuli

Intransitivities in observers' preference judgments for multidimensional natural landscape scenes are examined. Thurstone's Law of Comparative Judgment (LCJ) psychophysical scaling routine is used to scale landscape preference. Both the method of Rank Ordering (RO) and the method of Paired Comparisons (PC) are used to gather the raw data necessary for scaling. The preference metric derived from the PC data which allows intransitivities to occur, is compared to the metric derived from the RO data, which does not allow observer intransitivity. Analysis indicates that an observer's expressed ordinal preferences derived from his PC data differs from that obtained from his RO data. Analysis also indicates that intransitivity exist to a moderate degree in each individual's expressed preferences and that this intransitivity is related to the differences between the two sets of ordinal preferences. However, very little difference exists between the two, LCJ derived, interval preference metrics obtained from group PC data and group RO data.

The Use of Bayes' Estimates in the Law of Comparative Judgment

The use of Bayes' estimates for proportions in the Law of Comparative Judgment is suggested to avoid sample proportions of zero and one.

Bimodal cues for speech loudness.

This paper presents a bimodal (audio-visual) study of speech loudness. The same acoustic stimuli (three sustained vowels of the articulatory qualities "effort" and "noneffort") are first presented in isolation, and then simultaneously together with an appropriate optical stimulus (the speaker's face on a video screen, synchronously producing the vowels). By the method of paired comparisons (law of comparative judgment) subjective loudness differences could be represented by different intervals between scale values. By this method previous results of effort-dependent speech loudness could be verified. In the bimodal study the optical cues have a measurable effect, but the acoustic cues are still dominant. Visual cues act most effectively if they are presented naturally, i.e., if acoustic and optical effort cues vary in the same direction. The experiments provide some evidence that speech loudness can be influenced by other than acoustic variables.

Relationships Between the Thurstone and Rasch Approaches to Item Scaling

When the logistic function is substituted for the normal, Thurstone's Case V specialization of the law of comparative judgment for paired comparison responses gives an identical equation for the esti mation of item scale values as does the Rasch formulation for direct responses. The law of com parative judgment must be modified to include a subject parameter; but this parameter, which is eliminated statistically with respect to the direct re sponse design, is eliminated experimentally in the paired comparison design. Some comparisons and contrasts are made between the two approaches to item scaling, and it is shown that greater gen eralizability for item scaling is possible when the two approaches are juxtaposed appropriately.