Feature Extraction for Token Based Word Alignment for Question Answering Systems

Abstract

Mapping between the source words and the target words in a set of parallel sentences are a crucial part of Question Answering (QA) systems. I fan accurate aligner is used in QA systems then the efficiency of these systems also gets increased. We purpose the aligner which despite using very less lexical resources gives very good results in terms of precision, recall and F1. Previous aligners either uses more lexical resources or uses very less lexical resources. Hence, we have used POS TAG and WordNet as lexical resources. But some words whose meaning we maynot know but these occur in a similar distributionand by observing their distribution these words aresimilar. Consider two sentences "Lambodar is theson of Parvati" and "Ganesha is the son of Parvati". Here we will not find the meaning of Lambodar and Ganesha in Wordnet but since they have similar distributions so they should be aligned. For these words, we used Distribution Similarity Feature in our word aligner. This distributional similarity helps our alignerin broader coverage of words. Previous aligners were having recall in the range of 75-86 but this aligner has recall in the range of 88.4-93.3. Similarly, Exact match of previous aligners was in the range of 21-35.3 but the proposed aligner’s exact match range is 46.1-58.6. Similarly F-measure and precision have also increased.