Reputation Systems Evaluation Survey

Abstract

Various reputation systems have been proposed for a broad range of distributed applications, such as peer-to-peer, ad-hoc, and multiagent systems. Their evaluation has been mostly based on proprietary methods due to the lack of widely acceptable evaluation measures and methodologies. Differentiating factors in various evaluation approaches include the evaluation metrics, the consideration of the dynamic behavior of peers, the use of social networks, or the study of resilience to specific threat scenarios. The lack of a generally accepted common evaluation framework hinders the objective evaluation and comparison of different reputation systems. Aiming at narrowing the gap in the research area of objective evaluation of reputation systems, in this article, we study the various approaches to evaluating and comparing reputation systems, present them in a taxonomy, and analyze their strengths and limitations, with special focus on works suggesting a Common Evaluation Framework (CEF). We identify the challenges for a widely accepted CEF that enables testing and benchmarking of reputation systems, and we present the required properties for such a CEF; we also present an analysis of current CEF-related works in the context of the identified properties and our related proposals.