CrEx-Wisdom Framework for Fusion of Crowd and Experts in Crowd Voting Environment – Machine Learning Approach

Abstract

In recent years crowd-voting and crowd-sourcing systems are attracting increased attention in research and industry. As a part of computational social choice (COMSOC) crowd-voting and crowd-sourcing address important societal problems (e.g. participatory budgeting), but also many industry problems (e.g. sentiment analyses, data labeling, ranking and selection, etc.). Consequently, decisions that are based on aggregation of crowd votes do not guarantee high-quality results. Even more, in many cases majority of crowd voters may not be satisfied with final decisions if votes have high heterogeneity. On the other side in many crowd voting problems and settings it is possible to acquire and formalize knowledge and/or opinions from domain experts. Integration of expert knowledge and “Wisdom of crowd” should lead to high-quality decisions that satisfy crowd opinion. In this research, we address the problem of integration of experts domain knowledge with “Wisdom of crowds” by proposing machine learning based framework that enables ranking and selection of alternatives as well as quantification of quality of crowd votes. This framework enables weighting of crowd votes with respect to expert knowledge and procedures for modeling trade-off between crowd and experts satisfaction with final decisions (ranking or selection).