Evaluation of recommender systems: A new approach

Abstract

It is difficult to deny that comparison between recommender systems requires a common way for evaluating them. Nevertheless, at present, they have been evaluated in many, often incompatible, ways. We affirm this problem is mainly due to the lack of a common framework for recommender systems, a framework general enough so that we may include the whole range of recommender systems to date, but specific enough so that we can obtain solid results. In this paper, we propose such a framework, attempting to extract the essential features of recommender systems. In this framework, the most essential feature is the objective of the recommender system. What is more, in this paper, recommender systems are viewed as applications with the following essential objective. Recommender systems must: (i) choose which (of the items) should be shown to the user, (ii) decide when and how the recommendations must be shown. Next, we will show that a new metric emerges naturally from this framework. Finally, we will conclude by comparing the properties of this new metric with the traditional ones. Among other things, we will show that we may evaluate the whole range of recommender systems with this single metric.