Evaluating expertise recommendations

Abstract

Finding a person who has the expertise to solve a specific problem is an important application of recommender systems to a difficult organizational problem. Prior systems have made attempts to implement solutions to this problem, but few systems have undergone systematic user evaluation. This work describes a systematic evaluation of the Expertise Recommender (ER), a system that recommends people who are likely to have expertise in a specific problem. ER and the organizational context for which it was designed are described to provide a basis for understanding this evaluation. Prior to conducting the evaluation, a baseline experiment showed that people are relatively good at judging coworkers' expertise when given an appropriate context. This finding provides a way to demonstrate the effectiveness of ER by comparing ER's performance to ratings by coworkers. The evaluation, the design, and results are described in detail. The results suggest that the participants agree with the recommendations made by ER, and that ER significantly outperforms other expertise recommender systems when compared using similar metrics.