Cross-language Information Retrieval Research Articles

Using Lasso RCCA for cross-language information retrieval

ABSTRACTRestricted canonical correlation analysis and the lasso shrinkage method were paired together for canonical correlation analysis with non-negativity restrictions on datasets, where a sample size is much smaller than the number of variables. The method was implemented in an alternating least-squares algorithm and applied to cross-language information retrieval on a dataset with aligned documents in eight languages. A set of experiments was ran to evaluate the method and compare it to other methods in the field.

English-Marathi Cross Language Information Retrieval System

Nowadays, different types of content in different languages are available on World Wide Web and their usage is increasing rapidly. Cross Language Information Retrieval (CLIR) deals with retrieval of documents in another language than the language of the requested query. Various researchers worked on Cross Language Information Retrieval systems for Indian languages using different translation approaches. There is still CLIR system to be developed which allow user to retrieve Marathi documents when English query is given. In the proposed English to Marathi Cross Language Information Retrieval system, translation is based on query translation approach. The proposed system retrieves Marathi documents depending on matching terms in query. The performance of the proposed system is improved by query pre-processing and query expansion using WordNet.

ENGLISH TO HINDI TRANSLITERATION SYSTEM USING COMBINATION-BASED APPROACH

Transliteration plays a very significant role in machine translation, which has many applications such as cross-lingual information retrieval, communication, question-answering etc. The main objective of this research paper is to provide a method for transliteration of named entities from English to Hindi language. The proposed method consists of two modules, both of which apply phoneme-based approach to transliterate named entities. For transliteration, Module-I utilizes CMU Pronouncing dictionary, which is a collection of 133270 words along with their pronunciation. If the word to be transliterated is not found in CMU Pronouncing dictionary, Module-II is used. Module-II is based on 5-gram model, in which a maximum of five letters (two left, two right and one target letter) are used to generate transliterated target letter. The system has been tested on a database of 2408 North-Indian names. Google Input tool for Windows has been used for comparative study of the proposed transliteration system. The word accuracy of the transliteration system has been found to be 70.22% against 58.73% of Google Input tool.

Recognizing Named Entities in Text based User Reviews

Named Entity Recognition is a subtask in Information Extraction. In Named Entity Recognition (NER) we try to identify each of the words provided into some categories predefined by us. These categories or classes can be organization, name, time, place etc. Named Entity Recognition is a part of Natural Language Processing which is aims at making human and computer interactions more meaningful and efficient. NLP is nowadays being effectively being used in the area of automatic text summarization, cross language information retrieval, speech recognition and query named entity recognition. This paper focuses on Named entity recognition, how it is performed and a how apply NER to web search queries so that correct and more user personalized results are displayed by the web search engines..

An empirical analysis of user behaviour on multilingual information retrieval

PurposeThis paper aims to understand user information behaviours when they perform multilingual information retrieval. It also offers reference for the development of multilingual information retrieval systems and relevant service platforms.Design/methodology/approachThe authors designed an experiment on multilingual information retrieval with WorldWideScience, utilized Camtasia studio7 (a screen capturing and recording tool) to record overall operational processes of subjects and collected participants’ thought processes with think-aloud protocols. Meanwhile, a questionnaire survey and interviews were used to examine the subjects’ background information, their feelings for the experiment and their ideas about the experimental platform, respectively. Thirty-two valid data points were obtained by 41 subjects.FindingsThe users preferred their own language for retrieval. Most users from social science chose general search or advanced search freely according to the tasks. The majority of the participants selected key words directly from the tasks as search terms. Doctoral candidates were more likely to construct a search query with logic symbols. Translation tools were utilized for assisting retrieval and solving doubts of translation. When facing obstacles, users stayed on the original web page to explore continually, followed by back to homepage.Originality/valueThis paper provides a study of user behaviour through investigating how users behave on the whole process of retrieving multilingual information. The findings offer advice for optimizing the function of multilingual information retrieval systems and service platforms.

Study on Tibetan-Chinese Comparable Corpus Extraction

Tibetan-Chinese comparable corpus extraction is a basis work for Tibetan-Chinese cross language question answering system, information retrieval, machine translation and other researches. This paper is an exploration to solve the scarcity of Tibetan-Chinese comparable corpus. It will promote the knowledge sharing between different languages. In this paper, we propose a method to extract Tibetan-Chinese comparable corpus. The main work is in the following: (1) Tibetan-Chinese comparable corpus extraction model based on multi-feature of bilingual websites. (2) Extraction method based on entity link from naturally annotated resources. Finally, the experimental results show our approach is effective.

Towards a new possibilistic query translation tool for cross-language information retrieval

Approaches of query translation in Cross-Language Information Retrieval (CLIR) have frequently used dictionaries which suffer from translation ambiguity. Besides, a word-by-word query translation is not sufficient. In this paper, we propose, evaluate and compare a new possibilistic approach for query translation in order to improve the previous dictionary-based ones. This approach uses a probability-to-possibility transformation as a mean to introduce further tolerance in query translation process. Firstly, we identify noun phrases (NPs) in the source query and translate them as units using translation patterns and a language model. Secondly, source query terms which are not included in any selected NPs are translated word-by-word using our new possibilistic approach of single word translation. Indeed, we take into account all query words and their translations when we choose the suitable translation of a given word. We start from the idea that the correct suitable translations of query terms have a tendency to co-occur in the target language documents unlike unsuitable ones. Finally, to increase the coverage of the bilingual dictionary, additional words and their translations are automatically generated from a parallel bilingual corpus. We tested our approach using the French-English parallel text corpus Europarl and the CLEF-2003 French-English CLIR test collection. The reported experiments showed the performance of the probability-to-possibility transformation-based approach compared to the probabilistic one and to some state-of-the-art CLIR tools.

Using restricted CCA for cross-language information retrieval

ABSTRACTCanonical correlation analysis is a method of correlating linear relationship between two sets of variables. When not any linear combination of variables is allowed, restricted canonical correlation analysis is appropriate. The method was implemented with alternating least-squares and applied to the cross-language information retrieval on a dataset with officially translated and aligned documents in eight European languages.

An expectation-maximization algorithm for query translation based on pseudo-relevant documents

Query translation in cross-language information retrieval (CLIR) can be done by employing dictionaries, aligned corpora, or machine translators. Scarcity of aligned corpora for various domains in many language pairs intensifies the importance of dictionary-based CLIR which motivates us to use only a bilingual dictionary and two independent collections in source and target languages for query translation. We exploit pseudo-relevant documents for a given query in the source language and pseudo-relevant documents for a translation of the query in the target language with a proposed expectation-maximization algorithm for improving query translation. The proposed method (called EM4QT) assumes that each target term either is translated from the source pseudo-relevant documents or has come from a noisy collection. Since EM4QT does not directly consider term coherency, which is defined as fluency of the target translation, we investigate a crucial question: can EM4QT be improved using either coherency-based methods or token-to-token translation ones? To address this question, we combine different translation models via simple linear interpolation and a proposed divergence minimization method. Evaluations over four CLEF collections in Persian, French, Spanish, and German indicate that EM4QT significantly outperforms competitive baselines in all the collections. Our experiments also reveal that since EM4QT indirectly considers term coherency, combining the method with coherency-based models cannot significantly improve the retrieval performance. On the other hand, investigating the query-by-query results supports the view that EM4QT usually gives a relatively high weight to one translation and its combination with the proposed token-to-token translation model, which is obtained by running EM4QT for each query term separately, soothes the effect and reaches better results for many queries. Comparing the method with a competitive word-embedding baseline reveals the superiority of the proposed model.

Continuous space models for CLIR

We present and evaluate a novel technique for learning cross-lingual continuous space models to aid cross-language information retrieval (CLIR). Our model, which is referred to as external-data composition neural network (XCNN), is based on a composition function that is implemented on top of a deep neural network that provides a distributed learning framework. Different from most existing models, which rely only on available parallel data for training, our learning framework provides a natural way to exploit monolingual data and its associated relevance metadata for learning continuous space representations of language. Cross-language extensions of the obtained models can then be trained by using a small set of parallel data. This property is very helpful for resource-poor languages, therefore, we carry out experiments on the English-Hindi language pair. On the conducted comparative evaluation, the proposed model is shown to outperform state-of-the-art continuous space models with statistically significant margin on two different tasks: parallel sentence retrieval and ad-hoc retrieval.

An Approach to Construct a Named Entity Annotated English-Vietnamese Bilingual Corpus

Manually constructing an annotated Named Entity (NE) in a bilingual corpus is a time-consuming, labor--intensive, and expensive process, but this is necessary for natural language processing (NLP) tasks such as cross-lingual information retrieval, cross-lingual information extraction, machine translation, etc. In this article, we present an automatic approach to construct an annotated NE in English-Vietnamese bilingual corpus from a bilingual parallel corpus by proposing an aligned NE method. Basing this corpus on a bilingual corpus in which the initial NEs are extracted from its own language separately, the approach tries to correct unrecognized NEs or incorrectly recognized NEs before aligning the NEs by using a variety of bilingual constraints. The generated corpus not only improves the NE recognition results but also creates alignments between English NEs and Vietnamese NEs, which are necessary for training NE translation models. The experimental results show that the approach outperforms the baseline methods effectively. In the English-Vietnamese NE alignment task, the F-measure increases from 68.58% to 79.77%. Thanks to the improvement of the NE recognition quality, the proposed method also increases significantly: the F-measure goes from 84.85% to 88.66% for the English side and from 75.71% to 85.55% for the Vietnamese side. By providing the additional semantic information for the machine translation systems, the BLEU score increases from 33.04% to 45.11%.

FaDA: Fast Document Aligner using Word Embedding

Abstract FaDA is a free/open-source tool for aligning multilingual documents. It employs a novel crosslingual information retrieval (CLIR)-based document-alignment algorithm involving the distances between embedded word vectors in combination with the word overlap between the source-language and the target-language documents. In this approach, we initially construct a pseudo-query from a source-language document. We then represent the target-language documents and the pseudo-query as word vectors to find the average similarity measure between them. This word vector-based similarity measure is then combined with the term overlap-based similarity. Our initial experiments show that s standard Statistical Machine Translation (SMT)- based approach is outperformed by our CLIR-based approach in finding the correct alignment pairs. In addition to this, subsequent experiments with the word vector-based method show further improvements in the performance of the system.

Using Word Embeddings for Query Translation for Hindi to English Cross Language Information Retrieval

Cross-Language Information Retrieval (CLIR) has become an important problem to solve in the recent years due to the growth of content in multiple languages in the Web. One of the standard methods is to use query translation from source to target language. In this paper, we propose an approach based on word embeddings, a method that captures contextual clues for a particular word in the source language and gives those words as translations that occur in a similar context in the target language. Once we obtain the word embeddings of the source and target language pairs, we learn a projection from source to target word embeddings, making use of a dictionary with word translation pairs. We then propose various methods of query translation and aggregation. The advantage of this approach is that it does not require the corpora to be aligned (which is difficult to obtain for resource-scarce languages), a dictionary with word translation pairs is enough to train the word vectors for translation. We experiment with Forum for Information Retrieval and Evaluation (FIRE) 2008 and 2012 datasets for Hindi to English CLIR. The proposed word embedding based approach outperforms the basic dictionary based approach by 70% and when the word embeddings are combined with the dictionary, the hybrid approach beats the baseline dictionary based method by 77%. It outperforms the English monolingual baseline by 15%, when combined with the translations obtained from Google Translate and Dictionary.

A Comparison of Methods for Identifying the Translation of Words in a Comparable Corpus: Recipes and Limits

Identifying translations in comparable corpora is a challenge that has attracted many researchers since a long time. It has applications in several applications including Machine Translation and Cross-lingual Information Retrieval. In this study we compare three state-of-the-art approaches for these tasks: the so-called context-based projection method, the projection of monolingual word embeddings, as well as a method dedicated to identify translations of rare words. We carefully explore the hyper-parameters of each method and measure their impact on the task of identifying the translation of English words in Wikipedia into French. Contrary to the standard practice, we designed a test case where we do not resort to heuristics in order to pre-select the target vocabulary among which to find translations, therefore pushing each method to its limit. We show that all the approaches we tested have a clear bias toward frequent words. In fact, the best approach we tested could identify the translation of a third of a set of frequent test words, while it could only translate around 10% of rare words.

3.5K runs, 5K topics, 3M assessments and 70M measures: What trends in 10 years of Adhoc-ish CLEF?

Multilingual information access and retrieval is a key concern in today global society and, despite the considerable achievements over the past years, it still presents many challenges. In this context, experimental evaluation represents a key driver of innovation and multilinguality is tackled in several evaluation initiatives worldwide, such as CLEF in Europe, NTCIR in Japan and Asia, and FIRE in India. All these activities have run several evaluation cycles and there is a general consensus about their strong and positive impact on the development of multilingual information access systems. However, a systematic and quantitative assessment of the impact of evaluation initiatives on multilingual information access and retrieval over the long period is still missing.Therefore, in this paper we conduct the first systematic and large-scale longitudinal study on several CLEF Adhoc-ish tasks – namely the Adhoc, Robust, TEL, and GeoCLEF labs – in order to gain insights on the performance trends of monolingual, bilingual and multilingual information access systems, spanning several European and non-European languages, over a range of 10 years.We learned that monolingual retrieval exhibits a stable positive trend for many of the languages analyzed, even though the performance increase is not always steady from year to year due to the varying interests of the participants, who may not always be focused on just increasing performances. Bilingual retrieval demonstrates higher improvements in recent years – probably due to the better language resources now available – and it also outperforms monolingual retrieval in several cases. Multilingual retrieval shows improvements over the years and performances are comparable to those of bilingual and monolingual retrieval, and sometimes even better. Moreover, we have found evidence that the rule-of-thumb of a 3-year duration for an evaluation task is typically enough since top performances are usually reached by the third year and sometimes even by the second year, which then leaves room for research groups to investigate relevant research issues other than top performances.Overall, this study provides quantitative evidence that CLEF has achieved the objective which led to its establishment, i.e. making multilingual information access a reality for European languages. However, the outcomes of this paper not only indicate that CLEF has steered the community in the right direction, but they also highlight the many open challenges for multilinguality. For instance, multilingual technologies greatly depend on language resources and targeted evaluation cycles help not only in developing and improving them, but also in devising methodologies which are more and more language-independent. Another key aspect concerns multimodality, intended not only as the capability of providing access to information in multiple media, but also as the ability of integrating access and retrieval over different media and languages in a way that best fits with user needs and tasks.

Comparative analysis of book tags: a cross-lingual perspective

Purpose In the era of social media, users all over the world annotate books with social tags to express their preferences and interests. The purpose of this paper is to explore different tagging behaviours by analysing the book tags in different languages. Design/methodology/approach This investigation collected nearly 56,000 tags of 1,200 books from one Chinese and two English online bookmarking systems; it combined content analysis and machine-processing methods to evaluate the similarities and differences between different tagging systems from a cross-lingual perspective. Jaccard’s coefficient was adopted to evaluate the similarity level. Findings The results show that the similarity between mono-lingual tags of the same books is higher than that of cross-lingual tags in different systems and the similarity between tags of books written for specialties is higher than that of books written for the general public. Research limitations/implications Those who have more in common annotate books with more similar tags. The similarity between users in tagging systems determines the similarity of the tag sets. Practical implications The results and conclusion of this study will benefit users’ cross-lingual information retrieval and cross-lingual book recommendation for online bookmarking systems. Originality/value This study may be one of the first to compare cross-lingual tags. Its methodology can be applied to tag comparison between any two languages. The insights of this study will help develop cross-lingual tagging systems and improve information retrieval.

Cross-Language Information Retrieval with Incorrect Query Translations

In this paper, we present a Cross Language Information Retrieval (CLIR) approach using corpus driven query suggestion. We have used corpus statistics to gather a clue on selecting the right query terms when the translation of a specific query is missing or incorrect. The derived set of queries are ranked to select the top ranked queries. These top ranked queries are further used to perform query formulation. Using the re-formulated weighted query, we perform cross language information retrieval. The results are compared with the results of CLIR system with Google translation of user queries and CLIR with the proposed query suggestion approach. We have English and Tamil corpus of FIRE 2012 dataset and analyzed the effects of the proposed approach. The experimental results show that the proposed approach performs well with the incorrect translation of the queries.

A study of user profile representation for personalized cross-language information retrieval

Purpose– With an increase in the amount of multilingual content on the World Wide Web, users are often striving to access information provided in a language of which they are non-native speakers. The purpose of this paper is to present a comprehensive study of user profile representation techniques and investigate their use in personalized cross-language information retrieval (CLIR) systems through the means of personalized query expansion.Design/methodology/approach– The user profiles consist of weighted terms computed by using frequency-based methods such as tf-idf and BM25, as well as various latent semantic models trained on monolingual documents and cross-lingual comparable documents. This paper also proposes an automatic evaluation method for comparing various user profile generation techniques and query expansion methods.Findings– Experimental results suggest that latent semantic-weighted user profile representation techniques are superior to frequency-based methods, and are particularly suitable for users with a sufficient amount of historical data. The study also confirmed that user profiles represented by latent semantic models trained on a cross-lingual level gained better performance than the models trained on a monolingual level.Originality/value– Previous studies on personalized information retrieval systems have primarily investigated user profiles and personalization strategies on a monolingual level. The effect of utilizing such monolingual profiles for personalized CLIR remains unclear. The current study fills the gap by a comprehensive study of user profile representation for personalized CLIR and a novel personalized CLIR evaluation methodology to ensure repeatable and controlled experiments can be conducted.

Improved Query Translation for English to Hindi Cross Language Information Retrieval

Improved Query Translation for English to Hindi Cross Language Information Retrieval

Improving Performance of Hindi-English based Cross Language Information Retrieval using Selective Documents Technique and Query Expansion

Improving Performance of Hindi-English based Cross Language Information Retrieval using Selective Documents Technique and Query Expansion