Direct Rate Research Articles

Can holistic declaration of preferences improve a negotiation offer scoring system?

In this paper, we analyse the problem of determining a negotiation offer scoring system using an alternative approach to the classic direct rating (DR). We examine the effectiveness of the prenegotiation preference elicitation on the basis of holistic judgments, supported by a software decision support tool. This approach is based on rank-ordering of examples of complete offers, which is then disaggregated using the Utilités Additives (UTA) method. In a series of studies, we analyse the accuracy of the scoring systems obtained from these approaches, as well as the negotiators’ subjective evaluation of their use and usefulness. The technical capability of the various setups of the UTA-based disaggregation models to produce accurate scoring systems is verified by simulation in Study 1. The empirical applicability of the most promising UTA-based models is studied in two experiments, in which the negotiators used examples of both predefined and self-declared sets of offers (Study 2), or applied an enhanced UTA algorithm (Study 3). The enhanced algorithm used a predefined set of offers, implemented certain elements of DR, and allowed for iterative improvements of the scoring system. The results show that the UTA-based approach works in a technical sense, but empirically its performance is worse than DR unless the set of example offers is predefined. The enhanced algorithm produced a better scoring system, but users' subjective evaluations were mixed.

VASMA Weighting: Survey-Based Criteria Weighting Methodology that Combines ENTROPY and WASPAS-SVNS to Reflect the Psychometric Features of the VAS Scales

Data symmetry and asymmetry might cause difficulties in various areas including criteria weighting approaches. Preference elicitation is an integral part of the multicriteria decision-making process. Weighting approaches differ in terms of accuracy, ease of use, complexity, and theoretical foundations. When the opinions of the wider audience are needed, electronic surveys with the matrix questions consisting of the visual analogue scales (VAS) might be employed as the easily understandable data collection tool. The novel criteria weighting technique VASMA weighting (VAS Matrix for the criteria weighting) is presented in this paper. It respects the psychometric features of the VAS scales and analyzes the uncertainties caused by the survey-based preference elicitation. VASMA weighting integrates WASPAS-SVNS for the determination of the subjective weights and Shannon entropy for the calculation of the objective weights. Numerical example analyzing the importance of the criteria that affect parents’ decisions regarding the choice of the kindergarten institution was performed as the practical application. Comparison of the VASMA weighting and the direct rating (DR) methodologies was done. It revealed that VASMA weighting is able to overcome the main disadvantages of the DR technique—the high biases of the collected data and the low variation of the criteria weights.

Comparing the validity of numerical judgements elicited by direct rating and point allocation: Insights from objectively verifiable perceptual tasks

Two popular methods for assigning numerical values to a set of to-be-judged objects in order to capture their relative standing are Direct Rating (DR) and Point Allocation (PA). People using PA distribute a fixed sum of 100 points among the objects, while people using DR rate each object on a fixed scale, typically 0–10, later rescaled to sum to 100. Prior research shows that these methods exhibit distinct profiles when values are ranked from largest to smallest, with DR being more test–retest reliable. But which method best translates people’s inner judgments into outer numerical values (is more valid)? Instead of examining subjective or abstract stimuli, we use objectively verifiable perceptual tasks, namely judgments of line length presented using bar charts. We show that (i) DR is more inter-rater reliable than PA; (ii) DR is more accurate than PA at the individual level; (iii) but there is no difference in accuracy when individual judgments are combined to form group-level estimates; and (iv) DR judgments were improved by using prior knowledge of method bias, whereas PA judgments were not.

A comparison of three weight elicitation methods: good, better, and best

This paper compares the properties and performance of three weight elicitation methods. It is in effect a “second round contest” in which the Bottomley et al. (2000) champion, direct rating (DR), locks horns with two new challengers. People using DR rate each attribute in turn on a scale of 0–100, whilst people using Max100 first assign to the most important attribute(s) a rating of 100, and then rate the other attributes relative to it/them. People using Min10 first assign the least important attribute(s) a rating of 10, and then rate the other attributes relative to it/them. The weights produced by Max100 were somewhat more test–retest reliable than DR. Both methods were considerably more reliable than Min10. Using people's test–retest data as attribute weights on simulated alternative values in a multi-attribute choice scenario, the same alternative would be chosen on 91% of occasions using Max100, 87% of occasions using DR, but only 75% of occasions using Min10. Moreover, the three methods are shown to have very distinct “signatures”, that is profiles relating weights to rank position. Finally, people actually preferred using Max100 and DR rather than Min10, an important pragmatic consideration.

Testing the Reliability of Weight Elicitation Methods: Direct Rating versus Point Allocation

Two commonly used methods of assigning numerical judgments (i.e., importance weights) to attributes in order to signify their relative importance are point allocation (PA) and direct rating (DR). These methods may seem to be minor variants of each other, yet they produce very different profiles of attribute weights when rank ordered from most to least important. The weights elicited by DR were more reliable than those elicited by PA in a test–retest situation. An important practical implication of this is for multicriteria decision making. Using people's test–retest data as attribute weights on simulated alternative values revealed that the same alternative would be chosen on 88% of occasions with DR, but only 74% of occasions with PA. Moreover, subjects reacted more favorably to DR than to PA.