Abstract
We develop a cognitive modeling approach, motivated by classic theories of knowledge representation and judgment from psychology, for combining people's rankings of items. The model makes simple assumptions about how individual differences in knowledge lead to observed ranking data in behavioral tasks. We implement the cognitive model as a Bayesian graphical model, and use computational sampling to infer an aggregate ranking and measures of the individual expertise. Applications of the model to 23 data sets, dealing with general knowledge and prediction tasks, show that the model performs well in producing an aggregate ranking that is often close to the ground truth and, as in the “wisdom of the crowd” effect, usually performs better than most of individuals. We also present some evidence that the model outperforms the traditional statistical Borda count method, and that the model is able to infer people's relative expertise surprisingly well without knowing the ground truth. We discuss the advantages of the cognitive modeling approach to combining ranking data, and in wisdom of the crowd research generally, as well as highlighting a number of potential directions for future model development.
Highlights
People have all sorts of different knowledge, and are able to express their knowledge in many ways
The wisdom of the crowd effect involves combining or aggregating the knowledge expressed by different people, and considering how the aggregate and individual expressions perform relative to some goal or criterion
We describe a corpus of data sets in which people rank order items, dealing with various domains, collected in a variety of ways, and involving both general knowledge and prediction tasks
Summary
People have all sorts of different knowledge, and are able to express their knowledge in many ways. One ubiquitous form and expression of knowledge involved ranking or ordering items to produce a structured list, giving the relative positions of a set of items with respect to a criterion of interest. We consider the well-known wisdom of the crowd effect [1] applied to rankings. The wisdom of the crowd effect involves combining or aggregating the knowledge expressed by different people, and considering how the aggregate and individual expressions perform relative to some goal or criterion. Our focus is on combining the rankings provided by different people in those situations where there is a (current or future) ground truth against which an aggregate ranking can be assessed
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
