Exploring Web-Based Translation Resources Applied to Hindi-English Cross-Lingual Information Retrieval

Abstract

Internet users perceive a multilingual web but are unfamiliar with it due to communication in their regional language called Cross-Lingual Information Retrieval (CLIR). In CLIR, a translation technique is used to translate the user queries into the target document’s language. Conventional translation techniques are based on either a manual dictionary or a parallel corpus, whereas the trending Statistical Machine Translation (SMT) and Neural Machine Translation (NMT) techniques are trained on a parallel corpus. NMT is not so mature for Hindi-English translation, according to the literature, and SMT performs better than the NMT. SMT provides a static translation due to the limited vocabularies in the available parallel corpus. It may not provide the translations for missing or unseen words, whereas the web provides a dynamic interface where multiple users are updating information at the same time. The web may provide the translations for missing or unseen words, and therefore the web is effectively used for technically developed languages like English, German, Spanish, Russian, and Chinese. In this article, different web resources such as Wikipedia, Hindi WordNet and Indo WordNet, ConceptNet, and online dictionary based translation techniques are proposed and applied to Hindi-English CLIR. Wikipedia-based translation approach incorporates three modules—exactly matched, partially matched, and disambiguation—to address the issues of wrong inter-wiki links, partially matched terms, and ambiguous articles. Hindi WordNet and Indo WorNet attribute “English synset” and ConceptNet attributes “Related term” & “Synonymy” are used for obtaining translations. Further, WordNet path similarity is used to disambiguate translations. Various online dictionaries are available that return multiple relevant and irrelevant translations. The proposed approaches are compared to the SMT where the Wikipedia-based approach achieves approximately similar mean average precision to SMT.