Judgment Aggregation Research Articles

Assessment Framework of Green Intelligent Transformation of Small Hydropower in China

With the comprehensive promoted construction of the establishment of green small hydropower, the defects of existing small hydropower station are gradually emerging, and it is necessary to implement green intelligent transformation to promote the construction of energy internet in China. This study focuses on constructing a green intelligent planning and transforming assessment framework, and assists management department to filtrate the small hydropower stations which can be transformed reasonably. Firstly, power station economy, ecological environment, technical safety management and social benefits are involved in the assessment index system. Secondly, multi-expert judgment aggregation based on fuzzed comparison scale is put forward to calculate the index value, and evidence synthesis is used to comprehensively assess the feasibility of green and intelligent planning and transformation for several small hydropower stations. The simulation case analysis shows the constructed assessment framework can reflect the actual situation of objectives properly and would provide decision-making basis for green and intelligent planning and transformation of small hydropower stations.

Convex combinations in judgment aggregation

Judgments are the basis for almost all decisions. They often come from different models and multiple experts. This information is typically aggregated using simple averages, which leads to the well-known shared information problem. A weighted average of the individual judgments based on empirically estimated sophisticated weights is commonly discarded in practice, because the sophisticated weights have large estimation errors. In this paper, I explore mixture weights, which are convex combinations of sophisticated and naïve weights. I show analytically that if the data generation process is stable, there always exists a mixture weight which aggregates judgments better than the naïve weights. I thus offer a path to alleviate the shared information problem. In contrast to other proposed solutions, it does not require any control over the judgment process. I demonstrate the utility of mixture weights in numerical analyses and in two empirical applications. I also offer heuristic selection algorithms for the correct mixture weight and analyze them in my numerical and empirical settings.

Judgment Aggregation in Creative Production: Evidence from the Movie Industry

We study a novel, low-cost approach to aggregating judgment from a large number of industry experts on ideas that they encounter in their normal course of business. Our context is the movie industry, in which customer appeal is difficult to predict and investment costs are high. The Black List, an annual publication, ranks unproduced scripts based on anonymous nominations from film executives. This approach entails an inherent trade-off: Low participation costs enable high response rates, but nominations lack standard criteria, and which voters see which ideas is unobservable and influenced by various factors. Despite these challenges, we find that such aggregation is predictive: Listed scripts are substantially more likely to be released than observably similar, but unlisted, scripts, and, conditional on release and investment levels, listed scripts generate higher box-office revenues. We also find that this method mitigates entry barriers for less-experienced writers, as (i) their scripts are more likely to be listed than those by experienced writers and to rank higher if listed and (ii) within scripts by less-experienced writers, being listed is associated with a higher release rate. Yet, the gap in release probabilities relative to experienced writers remains large, even for top-ranked scripts. These results can be explained by the premise that scripts from less-experienced writers are more visible among eligible voters than scripts from experienced writers. This highlights idea visibility as an important determinant of votes and surfaces the trade-offs, as well as potential limitations, associated with such methods. This paper was accepted by Ashish Arora, entrepreneurship and innovation.

Consistent rights on property spaces

In many aggregation problems, subgroups of agents have the right to predetermine certain properties of the aggregate. Yet, such rights may be inconsistent. In preference aggregation, for example, the ‘liberal paradox’ refers to the incompatibility of minimal liberal rights with the Pareto principle (a right to society as a whole). We show that, in general, rights to properties are consistent if and only if the following simple condition holds. Whenever rights are given to a critical (i.e., minimally inconsistent) combination of properties, the respective rights holding groups must intersect to at least one common member. Rights are consistent with monotone independent aggregation (voting by properties) if and only if this condition holds under a suitable generalization of criticality. Our property formulation allows us to study a wide range of applications in social choice and judgment aggregation theory.

Strategic manipulation in judgment aggregation under higher-level reasoning

We analyse the incentives of individuals to misrepresent their truthful judgments when engaged in collective decision-making. Our focus is on scenarios in which individuals reason about the incentives of others before choosing which judgments to report themselves. To this end, we introduce a formal model of strategic behaviour in logic-based judgment aggregation that accounts for such higher-level reasoning as well as the fact that individuals may only have partial information about the truthful judgments and preferences of their peers. We find that every aggregation rule must belong to exactly one of three possible categories: it is either (i) immune to strategic manipulation for every level of reasoning, or (ii) manipulable for every level of reasoning, or (iii) immune to manipulation only for every kth level of reasoning, for some natural number k greater than 1.

Judgment Aggregation by a Boundedly Rational Decision-Maker

This study identifies a new optimal decision rule in a general pairwise choice framework, taking into account behavioral aspects where subjective probabilities are assumed. The optimal rule under such a setting is compared to the one identified where behavioral considerations are not incorporated into the model.

Accounting for groups: the dynamics of intragroup deliberation

In a highly influential work, List and Pettit (Group Agency: The Possibility, Design, and Status of Corporate Agents, Oxford University Press, 2011) draw upon the theory of judgement aggregation to offer an argument for the existence of nonreductive group agents; they also suggest that nonreductive group agency is a widespread phenomenon. In this paper, we argue for the following two claims. First, that the axioms they consider cannot naturally be interpreted as either descriptive characterisations or normative constraints upon group judgements, in general. This makes it unclear how the List and Pettit argument is to apply to real world group behaviour. Second, by examining empirical data about how group judgements are made by a powerful international regulatory board, we show how each of the List and Pettit axioms can be violated in ways which are straightforwardly explicable at the level of the individual. This suggests that group agency may best be understood as a pluralistic phenomenon, where close inspection of the dynamics of intragroup deliberation can reveal that what prima facie appears to be a nonreductive group agent is, in fact, reducible.

What is a Good Calibration Question?

Weighted aggregation of expert judgments based on their performance on calibration questions may improve mathematically aggregated judgments relative to equal weights. However, obtaining validated, relevant calibration questions can be difficult. If so, should analysts settle for equal weights? Or should they use calibration questions that are easier to obtain but less relevant? In this article, we examine what happens to the out-of-sample performance of weighted aggregations of the classical model (CM) compared to equal weighted aggregations when the set of calibration questions includes many so-called "irrelevant" questions, those that might ordinarily be considered to be outside the domain of the questions of interest. We find that performance weighted aggregations outperform equal weights on the combined CM score, but not on statistical accuracy (i.e., calibration). Importantly, there was no appreciable difference in performance when weights were developed on relevant versus irrelevant questions. Experts were unable to adapt their knowledge across vastly different domains, and in-sample validation did not accurately predict out-of-sample performance on irrelevant questions. We suggest that if relevant calibration questions cannot be found, then analysts should use equal weights, and draw on alternative techniques to improve judgments. Our study also indicates limits to the predictive accuracy of performance weighted aggregation, and the degree to which expertise can be adapted across domains. We note limitations in our study and urge further research into the effect of question type on the reliability of performance weighted aggregations.

The median rule in judgement aggregation

A judgement aggregation rule takes the views of a collection of voters over a set of interconnected issues and yields a logically consistent collective view. The median rule is a judgement aggregation rule that selects the logically consistent view which minimizes the average distance to the views of the voters (where the “distance” between two views is the number of issues on which they disagree). In the special case of preference aggregation, this is called the Kemeny rule. We show that, under appropriate regularity conditions, the median rule is the unique judgement aggregation rule which satisfies three axioms: Ensemble Supermajority Efficiency, Reinforcement, and Continuity. Our analysis covers aggregation problems in which the consistency restrictions on input and output judgements may differ. We also allow for issues to be weighted, and provide numerous examples in which issue weights arise naturally.

Towards a unified aggregation framework for preferences and judgments

In their foundational work, List and Pettit formalized the judgment aggregation framework and showed that the preference aggregation framework from social choice theory can be mapped into it, arguing that the reverse was not possible. We show that a natural extension of a graph-theoretic representation of the preference aggregation framework indeed allows us to embed also the judgment aggregation framework. Moreover, we show that many concepts from the two original frameworks match up under the new one, show that it is possible to detect “logical consistency” with graph-theoretical properties, and give a new nuanced comparison between the doctrinal paradox and the Condorcet paradox.

Convex Combinations in Optimal Judgment Aggregation

Judgments are the basis for almost all decisions. They often come from different models and multiple experts. This information is typically aggregated using simple averages, which leads to the well-known shared information problem. A weighted average of the individual judgments based on empirically estimated sophisticated weights is commonly discarded in practice, because the sophisticated weights have large estimation errors. In this paper, we explore mixture weights, which are convex combinations of sophisticated and naive weights. We show analytically that if the data generation process is stable, there always exists a mixture weight which aggregates judgments better than the naive weights. We thus offer a path to alleviate the shared information problem. In contrast to other proposed solutions, we do not require any control over the judgment process. We demonstrate the utility of mixture weights in numerical analyses and in two empirical applications. We also offer heuristic selection algorithms for the correct mixture weight and analyze them in our numerical and empirical settings.

Dynamically Rational Judgment Aggregation

Judgment-aggregation theory has always focused on the attainment of rational collective judgments. But so far, rationality has been understood in static terms: as “coherence” of judgments at a given time, construed as consistency, completeness, and/or deductive closure. By contrast, this paper discusses whether collective judgments can be dynamically rational, so that they change rationally in response to new information. Formally, a judgment aggregation rule is dynamically rational with respect to a given revision operator if, whenever all individuals revise their judgments in light of some information (a learnt proposition), then the new aggregate judgments are the old ones revised in light of this information, i.e., aggregation and revision commute. We prove a general impossibility theorem: if the propositions on the agenda are sufficiently interconnected, no judgment aggregation rule with standard properties is dynamically rational with respect to any revision operator satisfying some mild conditions (familiar from belief revision theory). Our theorem is the dynamic-rationality analogue of some well-known impossibility theorems for static rationality. We also explore how dynamic rationality might be achieved by relaxing some of the conditions on the aggregation rule and/or the revision operator. In particular, we show that premise-based aggregation is dynamically rational with respect to premise-based revision.

An update on combinatorial method for aggregation of expert judgments in AHP

Paper aims the paper aims to demonstrate the advantages of several modifications of combinatorial method of expert judgment aggregation in AHP. Modifications are based on 1) weighting of spanning trees; 2) sorting of spanning trees by graph diameter during aggregation. Originality Both the method and its modifications are developed and improved by the authors. We propose to 1) weight spanning trees, based on quality of respective expert estimates, and 2) sort them by diameter in order to reduce the impact of expert errors and the method’s computational complexity. Research method we focus on theoretical and empirical studies of several modifications of combinatorial method of aggregation of expert judgments within AHP. Main findings modified combinatorial method has several conceptual advantages over ordinary method. It is also less sensitive to perturbations of initial data. Additionally, selection of spanning trees with smaller diameters allows us to reduce computational complexity of the method and minimize the impact of expert errors. Implications for theory and practice Combinatorial method is a universal instrument of expert judgment aggregation, applicable to additive/multiplicative, complete/incomplete, individual/group pair-wise comparisons, provided in different scales. It is used in the original strategic planning technology, which has recently found several important applications.

Game Form Representation for Judgement and Arrovian Aggregation Rules

Game Form Representation for Judgement and Arrovian Aggregation Rules

A Regularized Weighting Model of Judgment Aggregation

A Regularized Weighting Model of Judgment Aggregation

The Complexity Landscape of Outcome Determination in Judgment Aggregation

We provide a comprehensive analysis of the computational complexity of the outcome determination problem for the most important aggregation rules proposed in the literature on logic-based judgment aggregation. Judgment aggregation is a powerful and flexible framework for studying problems of collective decision making that has attracted interest in a range of disciplines, including Legal Theory, Philosophy, Economics, Political Science, and Artificial Intelligence. The problem of computing the outcome for a given list of individual judgments to be aggregated into a single collective judgment is the most fundamental algorithmic challenge arising in this context. Our analysis applies to several different variants of the basic framework of judgment aggregation that have been discussed in the literature, as well as to a new framework that encompasses all existing such frameworks in terms of expressive power and representational succinctness.

Wisdom of crowds and collective decision-making in a survival situation with complex information integration

BackgroundThe wisdom of crowds and collective decision-making are important tools for integrating information between individuals, which can exceed the capacity of individual judgments. They are based on different forms of information integration. The wisdom of crowds refers to the aggregation of many independent judgments without deliberation and consensus, while collective decision-making is aggregation with deliberation and consensus. Recent research has shown that collective decision-making outperforms the wisdom of crowds. Additionally, many studies have shown that metacognitive knowledge of subjective confidence is useful for improving aggregation performance. However, because most of these studies have employed relatively simple problems; for example, involving general knowledge and estimating values and quantities of objects, it remains unclear whether their findings can be generalized to real-life situations involving complex information integration. This study explores the performance and process of the wisdom of crowds and collective decision-making by applying the wisdom of crowds with weighted confidence to a survival situation task commonly used in studies of collective decision-making.ResultsThe wisdom of crowds and collective decision-making outperformed individual judgment. However, collective decision-making did not outperform the wisdom of crowds. Contrary to previous studies, weighted confidence showed no advantage from comparison between confidence-weighted and non-weighted aggregations; a simulation analysis varying in group size and sensitivity of confidence weighting revealed interaction between group size and sensitivity of confidence weighting. This reveals that it is because of small group size and not the peculiarity of the survival task that results in no advantage of weighted confidence.ConclusionsThe study’s findings suggest that the wisdom of crowds could be applicable to complex problem-solving tasks, and interaction between group size and sensitivity of confidence weighting is important for confidence-weighted aggregation effects.

Elaborating or Aggregating? The Joint Effects of Group Decision-Making Structure and Systematic Errors on the Value of Group Interactions

We explore when group interactions will have a positive effect on the accuracy of quantitative judgments. The results from three laboratory experiments revealed that the value of group interactions, compared with a statistical aggregation of individual judgments, differed strongly between groups in which decisions were made based on consensus, compared with groups with a randomly assigned group leader, and that this effect was moderated by the level of systematic error among group members. In particular, when systematic errors were low, group interactions generally provided little value, and groups’ decision-making structure (consensus versus leader based) did not have a significant effect on the value of group interactions. However, when the level of systematic error was high, the value of interactions in groups with a randomly designated leader was strongly positive and significantly higher than in groups with a consensus decision-making structure where interactions still provided only little value. Moreover, we found that this effect was mediated by information elaboration—which was higher in leader groups but only had a significant effect on the value of group interactions when there was a high level of systematic error. Finally, we also identified two important boundary conditions for these results. In particular, leader groups did not provide more value than consensus groups in the presence of strong systematic errors when leaders were assigned based on prior individual performance rather than randomly or when group members had only little expertise in the task. This paper was accepted by Yuval Rottenstreich, decision analysis.

Modeling Annotator Perspective and Polarized Opinions to Improve Hate Speech Detection

In this paper we propose an approach to exploit the fine-grained knowledge expressed by individual human annotators during a hate speech (HS) detection task, before the aggregation of single judgments in a gold standard dataset eliminates non-majority perspectives. We automatically divide the annotators into groups, aiming at grouping them by similar personal characteristics (ethnicity, social background, culture etc.). To serve a multi-lingual perspective, we performed classification experiments on three different Twitter datasets in English and Italian languages. We created different gold standards, one for each group, and trained a state-of-the-art deep learning model on them, showing that supervised models informed by different perspectives on the target phenomena outperform a baseline represented by models trained on fully aggregated data. Finally, we implemented an ensemble approach that combines the single perspective-aware classifiers into an inclusive model. The results show that this strategy further improves the classification performance, especially with a significant boost in the recall of HS prediction.

Enhancing the Wisdom of the Crowd With Cognitive-Process Diversity: The Benefits of Aggregating Intuitive and Analytical Judgments

Drawing on dual-process theory, we suggest that the benefits that arise from combining several quantitative individual judgments will be heightened when these judgments are based on different cognitive processes. We tested this hypothesis in three experimental studies in which participants provided estimates for the dates of different historical events (Study 1, N = 152), made probabilistic forecasts for the outcomes of soccer games (Study 2, N = 98), and estimated the weight of individuals on the basis of a photograph (Study 3, N = 3,695). For each of these tasks, participants were prompted to make judgments relying on an analytical process, on their intuition, or (in a control condition) on no specific instructions. Across all three studies, our results show that an aggregation of intuitive and analytical judgments provides more accurate estimates than any other aggregation procedure and that this advantage increases with the number of aggregated judgments.