Sentence alignment using local and global information

Abstract

Parallel corpora are essential resources for statistical machine translation (SMT) and cross language information retrieval (CLIR) systems. Creating parallel corpora is highly expensive in terms of both time and cost. In this paper, we propose a novel approach to automatically extract parallel sentences from aligned documents. To do so, we first train a Maximum Entropy binary classifier to compute the local similarity between each two sentences in different languages. To consider global information (e.g., the position of sentence pairs in the aligned documents), we define an objective function to penalize the cross alignments and then propose an integer linear programming approach to optimize the objective function. In our experiments, we focus on English and Persian Wikipedia articles. The experimental results on manually aligned test data indicate that the proposed method outperforms the baselines, significantly. Furthermore, the extrinsic evaluations of the corpus extracted from Wikipedia on both SMT and CLIR systems demonstrate the quality of the extracted parallel sentences. In addition, Experiments on the English–German language pair demonstrate that the proposed ILP method is a language-independent sentence alignment approach. The extracted English–Persian parallel corpus is freely available for research purposes.