Combining lexical and statistical translation evidence for cross‐language information retrieval

Abstract

This article explores how best to use lexical and statistical translation evidence together for cross‐language information retrieval (CLIR). Lexical translation evidence is assembled from Wikipedia and from a large machine‐readable dictionary, statistical translation evidence is drawn from parallel corpora, and evidence from co‐occurrence in the document language provides a basis for limiting the adverse effect of translation ambiguity. Coverage statistics for NII Testbeds and Community for Information Access Research (NTCIR) queries confirm that these resources have complementary strengths. Experiments with translation evidence from a small parallel corpus indicate that even rather rough estimates of translation probabilities can yield further improvements over a strong technique for translation weighting based on using Jensen–Shannon divergence as a term‐association measure. Finally, a novel approach to posttranslation query expansion using a random walk over the Wikipedia concept link graph is shown to yield further improvements over alternative techniques for posttranslation query expansion. Evaluation results on the NTCIR‐5 English–Korean test collection show statistically significant improvements over strong baselines.