Wisdom Of Crowds Research Articles

When Social Influence Promotes the Wisdom of Crowds

Whether, and under what conditions, groups exhibit has been a major focus of research across the social and computational sciences. Much of this work has focused on the role of social influence in promoting the wisdom of the crowd versus leading the crowd astray, resulting in conflicting conclusions about how the social network structure determines the impact of social influence. Here, we demonstrate that it is not enough to consider the network structure in isolation. Using theoretical analysis, numerical simulation, and reanalysis of four experimental datasets (totaling 2,885 human subjects), we find that the wisdom of crowds critically depends on the interaction between (i) the centralization of the social influence network and (ii) the distribution of the initial, individual estimates. By adopting a framework that integrates both the structure of the social influence and the distribution of the initial estimates, we bring previously conflicting results under one theoretical framework and clarify the effects of social influence on the wisdom of crowds.

The wisdom of the crowd in dynamic economies

The wisdom of the crowd applied to financial markets asserts that prices represent a consensus belief that is more accurate than individual beliefs. However, a market selection argument implies that prices eventually reflect only the beliefs of the most accurate agent. In this paper, we show how to reconcile these alternative points of view. In markets in which agents naively learn from equilibrium prices, a dynamic wisdom of the crowd holds. Market participation increases agents' accuracy, and equilibrium prices are more accurate than the most accurate agent.

Identify Experts through Revealed Confidence: Application to Wisdom of Crowds

Identifying the experts among the crowds may help us further improve the wisdom of crowds. However, due to overconfidence, conventional confidence elicitation can't reliably identify experts. To better reflect the relative expertise among agents, we propose the Revealed Confidence (RC) algorithm, which uses belief-update under an influence as a measure for confidence. We highlight the distinction between first-order and second-order uncertainty. We demonstrate that neglecting second-order uncertainty, as in the case of conventional self-reported confidence, may lead a perfectly rational agent to exhibit over-precision. From our belief-updating model, we prove that RC reveals both first-order and second-order uncertainty, as well as clarify the implications of the distance effect (the distance between an initial answer and its influence affects belief-updating) to the revelation of uncertainty. Our empirical analysis test, and confirm, several of our model predictions that (1) RC is able to improve upon conventional wisdom of crowds methods by overweighting the more accurate agents (2) the distance effect indeed affects belief-updating henceforth RC algorithm's performance and should be carefully controlled for. We also compare RC algorithm with the state-of-the-art minimal Pivoting Method and show RC still has an edge.

An emergent taxonomy for operational risk: capturing the wisdom of crowds

There has been a substantial change in the operational risks faced by financial services firms over the last fifteen years. Risks such as conduct, cyber and third party have risen in importance and now dominate boardroom agendas. How organizations think about this expanding portfolio of threats and manage them in a consistent way is underpinned by their risk taxonomy. This changing risk profile, combined with a recent shift of focus away from capital measurement toward risk management, means that many organizations are actively revising their operational risk taxonomies. In doing so, they are deviating from Basel Committee on Banking Supervision event types, but without a common standard for which to aim. In this paper, we take a data-driven approach and combine the individual active taxonomies of sixty large financial institutions (fifty-eight for construction and two for validation) to create a coherent new reference taxonomy: the ORX reference taxonomy for operational and nonfinancial risk. This combines both the theoretical and the practical wisdom of the crowd. The resultant taxonomy is a two-layered hierarchy of risks, with sixteen risks at level 1 and sixty-one risks at level 2. As part of this work, we look at some of the contemporary concerns that have shaped how institutions think about their risks. Constructing a mutually exclusive and collectively exhaustive taxonomy driven by data is not without its challenges, and the careful narrowing of scope and separation of risks, in partnership with industry experts, was sometimes needed. The Basel event types provide a central point of comparison, as they are universally adopted, have longevity and provide a basis for the ways in which institutions share data and provide insights into operational risks. The desired contribution of this ORX initiative was to create a common point of reference and thereby solid ground for industry discussion about developing operational risk taxonomies with the aim of laying the foundations that will allow consistent industry sharing of insights and data over the coming years.

Collective Intelligence and Wisdom of Crowds

Collective Intelligence and Wisdom of Crowds

TIMCC: On Data Freshness in Privacy-Preserving Incentive Mechanism Design for Continuous Crowdsensing Using Reverse Auction

As an emerging paradigm that leverages the wisdom and efforts of the crowd, mobile crowdsensing has shown its great potential to collect distributed data. The crowd may incur such costs and risks as energy consumption, memory consumption, and privacy leakage when performing various tasks, so they may not be willing to participate in crowdsensing tasks unless they are well-paid. Hence, a proper privacy-preserving incentive mechanism is of great significance to motivate users to join, which has attracted a lot of research efforts. Most of the existing works regard tasks as one-shot tasks, which may not work very well for the type of tasks that requires continuous monitoring, e.g., WIFI signal sensing, where the WiFi signal may vary over time, and users are required to contribute continuous efforts. The incentive mechanism for continuous crowdsensing has yet to be investigated, where the corresponding tasks need continuous efforts of users, and the freshness of the sensed data is very important. In this paper, we design TIMCC, a privacy-preserving incentive mechanism for continuous crowdsensing. In contrast to most existing studies that treat tasks as one-shot tasks, we consider the tasks that require users to contribute continuous efforts, where the freshness of data is a key factor impacting the value of data, which further determines the rewards. We introduce a metric named age of data that is defined as the amount of time elapsed since the generation of the data to capture the freshness of data. We adopt the reverse auction framework to model the connection between the platform and the users. We prove that the proposed mechanism satisfies individual rationality, computational efficiency, and truthfulness. Simulation results further validate our theoretical analysis and the effectiveness of the proposed mechanism.

Supplemental Material for How the “Wisdom of the Inner Crowd” Can Boost Accuracy of Confidence Judgments

Supplemental Material for How the “Wisdom of the Inner Crowd” Can Boost Accuracy of Confidence Judgments

Expert Selection Within a Single Judgment Problem

Combining point estimates from multiple judges often provides a more accurate aggregate estimate than using a point estimate from a single judge, a phenomenon called wisdom of crowds. However, if the judges use shared information when forming their estimates, the simple average will end up over-emphasizing this common component at the expense of the judges’ private information. A decision maker could in theory obtain a more accurate estimate by appropriately combining all information in the judges’ opinions. Although this information is embedded within the judges’ individual estimates, it is typically unobservable and thus cannot be directly aggregated by a decision maker. In this article, we propose a weighting of judges’ individual estimates that appropriately combines their collective information within a single estimation problem. Judges are asked to provide both a point estimate of the quantity of interest and a prediction of the average estimate that will be given by all other judges. Predictions of others are then used as part of a criterion to determine weights that are applied to each judge’s estimate to form an aggregate estimate. Our weighting procedure is robust to noise in the judges’ responses and can be expressed in closed form. We use both simulation and data from six experimental studies to illustrate that our procedure outperforms existing averaging-like methods.

The Disproportion of Crowd Wisdom: The Impact of Status Seeking on Yelp Reviews

This study shows that while status seeking motivates people to participate in crowd-sourcing platforms, it also negatively impacts the bedrock of crowd-sourcing – wisdom of crowds. Using Yelp restaurant reviews in 6 cities, we found that motivations of status seeking lead people to review a greater variety of restaurants, and achieving status further encourages this variety seeking as well as the targeting of more expensive restaurants for review. The impact of this individual-level tendency is confirmed by our aggregate-level analysis which shows that restaurants with higher price levels, higher uniqueness levels, and a larger percentage of elite reviews tend to obtain enough reviews to generate wisdom of crowds sooner than other restaurants. This leads to a different kind of distortion to crowd wisdom: an over-representation of status- conferring products and an under-representation of products that are not status-worthy. The findings suggest the importance of studying sources of distortion that are endemic to crowd-sourcing itself.

CONAN: A framework for detecting and handling collusion in crowdsourcing

In contrast to the traditional view that individuals should work independently to realize the crowd wisdom, crowdsourcing workers often collaborate with each other in task processing either explicitly or implicitly. Some may even collude for obtaining rewards easily, for example, by plagiarizing others’ answers. Collusion behavior sabotages the independency among workers, and will subvert the benefits of task redundancy that is commonly adopted in crowdsourcing. Therefore, dealing with collusion is critical for ensuring the quality of crowdsourcing. Existing work usually treats all the collusive answers as being harmful, thus simply filters away them once they are detected. However, it is not always the best strategy in practice. In particular, when the collusive answers are plagiarized from a worker with good ability, utilizing them instead of simple elimination can benefit the result quality. In this work, we first propose a collusion-aware framework for detecting and handling collusion in crowdsourcing properly. Second, we design a collusion detection method based on the statistical test of the consistency of workers’ answers across tasks. Third, we provide a theoretical means to determine when collusive answers should be kept and utilized, then we design a collusion-aware answer aggregation method. Finally, we conducted thorough evaluation with both synthetic and real-world datasets, and the results demonstrate the effectiveness of our approach.

Fusion of Multi-RSMOTE With Fuzzy Integral to Classify Bug Reports With an Imbalanced Distribution

With the help of automated classification, severe bugs can be rapidly identified so that the latent damage to software projects can be minimized. However, bug report datasets commonly suffer from disproportionate number of category samples. When presented with the situation of class imbalance, most standard classification learning approaches fail to properly learn the distributive characteristics of the samples and tend to result in unfavorable performance to predict class label. In this case, imbalanced learning becomes critical to advance classification algorithms. In this paper, we propose an improved synthetic minority oversampling technique to avoid the degraded performance caused by class imbalance in bug report datasets. Moreover, to lessen the chance of occasionalities in random sampling process, we propose a repeated sampling technique to train different, but related classifiers. Finally, an ensemble algorithm based on Choquet fuzzy integral is employed to combine the wisdom of crowds and make better decisions. We conduct comprehensive experiments on several bug report datasets from real-world bug repositories. The results demonstrate that the proposed method boosts the classification performance across the classes of the data. Specifically, compared with various ensemble learning techniques, the Choquet fuzzy integral achieves outstanding results on integrating multiple random oversampling techniques.

Quantification of predictive uncertainty in hydrological modelling by harnessing the wisdom of the crowd: A large-sample experiment at monthly timescale

Predictive hydrological uncertainty can be quantified by using ensemble methods. If properly formulated, these methods can offer improved predictive performance by combining multiple predictions. In this work, we use 50-year-long monthly time series observed in 270 catchments in the United States to explore the performances provided by an ensemble learning post-processing methodology for issuing probabilistic hydrological predictions. This methodology allows the utilization of flexible quantile regression models for exploiting information about the hydrological model's error. Its key differences with respect to basic two-stage hydrological post-processing methodologies using the same type of regression models are that (a) instead of a single point hydrological prediction it generates a large number of “sister predictions” (yet using a single hydrological model), and that (b) it relies on the concept of combining probabilistic predictions via simple quantile averaging. A major hydrological modelling challenge is obtaining probabilistic predictions that are simultaneously reliable and associated to prediction bands that are as narrow as possible; therefore, we assess both these desired properties of the predictions by computing their coverage probabilities, average widths and average interval scores. The results confirm the usefulness of the proposed methodology and its larger robustness with respect to basic two-stage post-processing methodologies. Finally, this methodology is empirically proven to harness the “wisdom of the crowd” in terms of average interval score, i.e., the average of the individual predictions combined by this methodology scores no worse –usually better− than the average of the scores of the individual predictions.

Evolutionary dynamics of traveling behavior in social networks

In this paper, the topology of complex network is used to characterize the interactions among travelers. Based on the assumption of travelers’ limited rationality and the wisdom of crowds, an evolutionary game model of travelers was established. Combining these research findings, we study commuters’ travel behavior rules during peak traffic periods and make some recommendations that have the potential to relieve traffic congestion. In the first section, we analyze the effect of heterogeneity on travel choice behavior by Monte Carlo simulations, which explains why the total amount of traffic during peak hours is hard to decrease. The experimental results indicate that increasing fees would not reduce traffic flow during peak hours. Moreover, controlling price for the management of travel demand needs to be coordinated with the management of traffic system under the condition of limited road resources. On this foundation, the second section proposes an environmental travel mode for the transfer between cycle highways and rail transit. We verify the effectiveness of this travel mode in alleviating traffic congestion. Furthermore, considering the interactions between different travel modes in transportation system, we suggest some management recommendations with reference to the characteristics of different travel modes.

Quantification of predictive uncertainty in hydrological modelling by harnessing the wisdom of the crowd: Methodology development and investigation using toy models

We introduce an ensemble learning post-processing methodology for probabilistic hydrological modelling. This methodology generates numerous point predictions by applying a single hydrological model, yet with different parameter values drawn from the respective simulated posterior distribution. We call these predictions “sister predictions”. Each sister prediction extending in the period of interest is converted into a probabilistic prediction using information about the hydrological model's errors. This information is obtained from a preceding period for which observations are available, and is exploited using a flexible quantile regression model. All probabilistic predictions are finally combined via simple quantile averaging to produce the output probabilistic prediction. The idea is inspired by the ensemble learning methods originating from the machine learning literature. The proposed methodology offers larger robustness in performance than basic post-processing methodologies using a single hydrological point prediction. It is also empirically proven to “harness the wisdom of the crowd” in terms of average interval score, i.e., the obtained quantile predictions score no worse –usually better− than the average score of the combined individual predictions. This proof is provided within toy examples, which can be used for gaining insight on how the methodology works and under which conditions it can optimally convert point hydrological predictions to probabilistic ones. A large-scale hydrological application is made in a companion paper.

A deeper look at the collective intelligence phenomenon

For the upcoming conference on Intelligence Studies at ICI 2020 in Bad Nauheim, Germany the focus of this issue of JISIB is on collective intelligence and foresight. The first two papers by Søilen and Almedia and Lesca deal with collective intelligence from an intelligence studies perspective. It may be said that the Internet itself is a gigantic collective intelligence effort, the largest in human history. Open source is a prerequisite for this system to work for everyone. The article by Černý et al. is on open source. All other contributions are on the connection between the Internet, software and intelligence. This issue consists of seven articles to compensate for two articles that were taken out by editors in the last issue. The first article by Søilen entitled “Making sense of the collective intelligence field: a review” is a historical review of the field of collective intelligence. The paper shows how collective intelligence is an interdisciplinary field and argues there is a flaw in the notion of “wisdom of crowds”. Collective intelligence can be understood in terms of social systems theory and as such this approach has been fruitful for the social sciences, although so far not very popular. It also bares relevance for the study of business and economics. The second article by Almeida and Lesca is entitled “Collective intelligence process to interpret weak signals and early warnings”. Early warning and the detection of weak signals is a vital topic for any intelligence organization. Two aspects are discussed in the paper, the importance of new technology and collective sense making or interpretation The third article by Shaikh and Singhal entitled “Study on the various intellectual property management strategies used and implemented by ICT firms for business intelligence” deals with intellectual property rights and patenting strategies. The authors identify a number of defensive and offensive IP strategies applied to ICT companies. The results have a bearing on patent acquisitions. The fourth article by Lamrhari et al. is entitled “Web intelligence for understanding customer satisfaction: application of Latent Dirichlet Allocation (LDA) and the Kano model”. Customer satisfaction today is mostly measured with data from the internet, using different business intelligence techniques. The Kano model is still valuablei,ii, but the way we gather information to assess the different levels in the model has changed. The authors use Latent Dirichlet Allocation to analyze the voice of customer (VOC) in online reviews. They suggest that BI techniques and a fuzzy-Kano model can enable companies to better understand their customers’ online reviews. The fifth article by Nahili et al. is entitled “A new corpus-based convolutional neutral network for big data text analysis”. Companies need efficient ways to analyze everything that is said about them on the internet (reviews, comments). The paper suggests a convolutional neural network (CNN) as it has been successfully used for text classification. IMDB movie reviews and Reuters datasets were used for the experiment. The sixth article by Černý et al. is entitled “Using open data and google search data for competitive intelligence analysis”. Taking the Czech antidepressant market as an example, the authors show how competitive intelligence can be obtained using Google Search data, Google Trend and other OSINT sources. The seventh article by Dadkhah et al. is entitled “The potential of business intelligence tools for expert findings”. The paper suggests a way for researchers to find experts using business intelligence tools. The same method may also be used by any business or person looking for experts on a specific topic. As always, we would above all like to thank the authors for their contributions to this issue of JISIB. Thanks to Dr. Allison Perrigo for reviewing English grammar and helping with layout design for all articles and to the Swedish Research Council for continuous financial support. We hope to see you all at the ICI 2020 on the 16-17 March, 2020. The deadline for the two-page abstract submission is March 1st, 2020.

Making sense of the collective intelligence field: A review

The problem we want to solve is to find out what is new in the collective intelligence literature and how it is to be understood alongside other social science disciplines. The reason it is important is that collective intelligence and problems of collaboration seem familiar in the social sciences but do not necessarily fit into any of the established disciplines. Also, collective intelligence is often associated with the notion of wisdom of crowds, which demands scrutiny. We found that the collective intelligence field is valuable, truly interdisciplinary, and part of a paradigm shift in the social sciences. However, the content is not new, as suggested by the comparison with social intelligence, which is often uncritical and lacking in the data it shows and that the notion of the wisdom of crowds is misleading (RQ1). The study of social systems is still highly relevant for social scientists and scholars of collective intelligence as an alternative methodology to more traditional social science paradigms as found, for example, in the study of business or management (RQ2).

What's in a Review

As video game press ("experts") and casual gamers ("amateurs") have different motivations when writing video game reviews, discrepancies in their reviews may arise. To study such potential discrepancies, we conduct a large-scale investigation of more than 1 million reviews on the Metacritic review platform. In particular, we assess the existence and nature of discrepancies in video game appraisal by experts and amateurs, and how they manifest in ratings, over time, and in review language. Leveraging these insights, we explore the predictive power of early expert vs. amateur reviews in forecasting video game reputation in the short- and long-term. We find that amateurs, in contrast to experts, give more polarized ratings of video games, rate games surprisingly long after game release, and are positively biased towards older games. On a textual level, we observe that experts write rather complex, less readable texts than amateurs, whose reviews are more emotionally charged. While in the short-term amateur reviews are remarkably predictive of game reputation among other amateurs (achieving 91% ROC AUC in a binary classification), both expert and amateur reviews are equally well suited for long-term predictions. Overall, our work is the first large-scale comparative study of video game reviewing behavior, with practical implications for amateurs when deciding which games to play, and for game developers when planning which games to design, develop, or continuously support. More broadly, our work contributes to the discussion of wisdom of the few vs. wisdom of the crowds, as we uncover the limits of experts in capturing the views of amateurs in the particular context of video game reviews.

Woc-Bots: An Agent-Based Approach to Decision-Making

We present a flexible, robust approach to predictive decision-making using simple, modular agents (WoC-Bots) that interact with each other socially and share information about the features they are trained on. Our agents form a knowledge-diverse crowd, allowing us to use Wisdom of the Crowd (WoC) theories to aggregate their opinions and come to a collective conclusion. Compared to traditional multi-layer perceptron (MLP) networks, WoC-Bots can be trained more quickly, more easily incorporate new features, and make it easier to determine why the network gives the prediction that it does. We compare our predictive accuracy with MLP networks to show that WoC-Bots can attain similar results when predicting the box office success of Hollywood movies, while requiring significantly less training time.

The Wisdom of the Market: Using Human Factors to Design Prediction Markets for Collective Intelligence

There is an ever-growing literature on the power of prediction markets to harness “the wisdom of the crowd” from large groups of people. However, traditional prediction markets are not designed in a human-centered way, often restricting their own potential. This creates the opportunity to implement a cognitive science perspective on how to enhance the collective intelligence of the participants. Thus, we propose a new model for prediction markets that integrates human factors, cognitive science, game theory and machine learning to maximize collective intelligence. We do this by first identifying the connections between prediction markets and collective intelligence, to then use human factors techniques to analyze our design, culminating in the practical ways with which our design enables artificial intelligence to complement human intelligence.

Studying the “Wisdom of Crowds” at Scale

In a variety of problem domains, it has been observed that the aggregate opinions of groups are often more accurate than those of the constituent individuals, a phenomenon that has been dubbed the “wisdom of the crowd”. However, due to the varying contexts, sample sizes, methodologies, and scope of previous studies, it has been difficult to gauge the extent to which conclusions generalize. To investigate this question, we carried out a large online experiment to systematically evaluate crowd performance on 1,000 questions across 50 topical domains. We further tested the effect of different types of social influence on crowd performance. For example, in one condition, participants could see the cumulative crowd answer before providing their own. In total, we collected more than 500,000 responses from nearly 2,000 participants. We have three main results. First, averaged across all questions, we find that the crowd indeed performs better than the average individual in the crowd—but we also find substantial heterogeneity in performance across questions. Second, we find that crowd performance is generally more consistent than that of individuals; as a result, the crowd does considerably better than individuals when performance is computed on a full set of questions within a domain. Finally, we find that social influence can, in some instances, lead to herding, decreasing crowd performance. Our findings illustrate some of the subtleties of the wisdom-of-crowds phenomenon, and provide insights for the design of social recommendation platforms.