Semantic models for answer re-ranking in question answering

Abstract

The task of Question Answering (QA) is to find correct answers to users' questions expressed in natural language. In the last few years non-factoid QA received more attention. It focuses on causation, manner and reason questions, where the expected answer has the form of a passage of text. The presence of question and answers corpora allows the adoption of Learning to Rank (MLR) algorithms in order to out- put a sensible ranking of the candidate answers. The importance and effectiveness of linguistically motivated features, obtained from syntax, lexical semantics and semantic role labeling, was shown in literature [2-4], but there are still several different possible semantic features that have not been taken into account so far and our goal is to find out if their use could lead to performance improvement. In particular features coming from Semantic Models (SM) like Distributional Semantic Models (DSMs), Explicit Semantic Analysis (ESA), Latent Dirichlet Allocation (LDA) induced topics have never been applied to the task so far. Based on the usefulness that those models show in other tasks, we think that SM can have a significant role in improving current state-of-the-art systems' performance in answer re-ranking. The questions this research wants to answer are: 1) Do semantic features bring information that is not present in the bag-of-words and syntactic features? 2) Do they bring different information or does it overlap with that of other features? 3) Are additional semantic features useful for answer re-ranking? Does their adoption improve systems' performance? 4) Which of them is more effective and under which circumstances? We performed a preliminary evaluation of DSMs on the ResPubliQA 2010 Dataset. We built a DSM based answer scorer that represents the question and the answer as the sums of the vectors of their terms taken term-term co-occurrence matrix and calculates their cosine similarity. We replaced the term-term matrix with the ones obtained by Random Indexing (RI), Latent Semantic Analysis (LSA) and LSA over the RI. Considering each DSM on its own, the results prove that all the DSMs are better than the baseline (the standard term-term co-occurrence matrix), and the improvement is always significant. The best improvement for the MRR in English is obtained by LSA (+180%), while in Italian by LSARI (+161%). We also showed that combining the DSMs with overlap based measures via CombSum the ranking is significantly better than the baseline obtained by the overlap measures alone. For English we have obtained an improvement in MRR of about 16% and for Italian, we achieve a even higher improvement in MRR of 26%. Finally, adopting RankNet for combining the overlap features and the DSMs features, improves the MRR of about 13%. More details can be found in [1]. In order to investigate the effectiveness of the semantic features, we still need to incorporate other semantic features, such as ESA, LDA and other state-of-the-art linguistic features. Other operators for semantic compositionality, like product, tensor product and circular convolution, will also be investigated. Moreover we will experiment on different datasets, focus- ing mainly on non-factoid QA. The Yahoo! Answers Manner Questions datasets are a good starting point. A new dataset will also be collected with questions from the users of Wikiedi (a QA system over Wikipedia articles, www.wikiedi.it) and answers in the form of paragraphs from Wikipedia pages.