An Evaluation Framework for Content Recommender Systems The Industry Perspective

Abstract

Recommender systems are a reality today. Evaluating recommender systems is difficult because of their extreme diversity. Many aspects need to be considered to be able to benchmark recommender systems against each other. This paper proposes an evaluation framework for content recommender systems which goes beyond traditional prediction accuracy. The first aspects to be considered relate to the input required for the correct functioning of the recommender system, to the output it produces and the usage of this output. Other aspects relate to how suitable the content recommender system is for the one deploying it, for the ones using it, as well as in today's world, inherently multidevice and with multiple sources of content. The quality of recommendations and the user experience they enable are key to the evaluation. Deployment aspects of content recommender systems, usually forgotten, complete this framework. The proposed evaluation framework provides a complete picture of the strenghts and weaknesses of content recommender systems from the industry perspective.