Cloud-based learning system for answer ranking

Abstract

Community question answering (Q&A) is a new knowledge-sharing model where a large number of questions and answers are accumulated through the user’s submission. When the user submits a new question, the Q&A system can provide the accurate answers list by the learning model. The traditional ranking algorithm mainly uses a large number of labeled data to train the model. However, a ranking model trained in the source domain may lead to poor performance in the target domain because of the lack of labeled training samples in the new domain. To address this challenge, this paper proposes a transfer learning algorithm based on feature selection for ranking. Suppose that the source domain and the target domain share the low-dimensional feature representation, and due to the user features exist share knowledge in source domain and target, so we use the user features are integrated into the answer space. Then the features of the target domain are shared for knowledge transfer. Furthermore, to improve the computational efficiency for the huge amount of data in the community Q&A, the learning model is distributed and processed by the Spark technology. Experimental results show that the proposed method could effectively exploit the cross-domain knowledge to enhance the effect of ranking.