Cross Language Information Retrieval System Research Articles

Exploiting Parallel Sentences and Cosine Similarity for Identifying Target Language Translation

In recent times, The Internet has become a huge information resource which contains information in multiple languages. Users are not acquainted with all languages and this language diversity becomes a great barrier for world communication. Cross-Language Information Retrieval (CLIR) provides a solution for this language barrier where a user can search the required information in his regional language. In this paper, a CLIR system is proposed based on Parallel Corpus (PC). A set of parallel sentences are extracted from PC which are based on query words. Term frequency matrix and cosine similarity measure are used for identifying target language translation. The proposed Term Frequency Method (TFM) approach is compared with Probabilistic Lexicon Method (PLM) approach and result analysis shows that proposed TFM approach performs better than the PLM approach.

Building Proximity Models for Cross Language Information Retrieval

In information retrieval systems, the proximity of query terms has been employed to enable ranking models to go beyond the ”bag of words” assumption and it can promote scores of documents where the matched query terms are close to each other. In this article, we study the integration of proximity models into cross-language information retrieval systems. The new proximity models are proposed and incorporated into existing cross-language information systems by combining the proximity score and the original score to re-rank retrieved documents. The experiment results show that the proposed models can help to improve the retrieval performance by 4%-7%, in terms of Mean Average Precision.

Building Structured Query in Target Language for Vietnamese – English Cross Language Information Retrieval Systems

Abstract — Query translation is the most important component in Cross Language Information Retrieval systems using dictionary-based approach. In this paper, we present a method to build structured query in target language from a given query in source language. The method is based on constructing bi-lingual dictionaries, keyword extraction from source query, getting translation candidates for each keyword using mutual information and finally building structured query in target language. By combining several translations for each query term in target language, we overcome shortcomings of selecting only one translation and therefore improve system performance. Results for Vietnamese-English cross-lingual retrieval show improvements of building structured query over other methods using dictionary to produce query translation in target language by combining single selection translation for each keyword. Keywords — CLIR; dictionary-based; Vietnamese; keyword extraction; structured query; disambiguation; mutual information

A Modified Cosine Similarity for Cross Language Information Retrieval

Since millions of documents are available on the Internet, some documents contain similar content but they are written in different languages by various authors. Unfortunately, the existing search engines do not support to all documents that are relevant to a single language query. Therefore, several researchers have put a huge effort to overcome such a problem. The major problems of a cross language search engine include 1) how to store information in a unify model and represent information of multiple languages documents effectively and 2) how to rank the retrieved multiple language documents and present to a user in the right order. This paper overcomes the first problem using an ontology model and we present a new ranking technique for a cross language information retrieval system (CLIR). Keyword weighting scheme in an ontology and document sections are introduced. Cosine similarity formula is modified to particularly support CLIR. The experimental results show the modified formula obtains more efficient ranking results than the existing method.

Cross-language information retrieval

AbstractThe article reviews the motivation behind the development of cross-language information retrieval (CLIR) – a relatively new area of information retrieval in multilingual textual databases, defines its objectives and position among the research disciplines dealing with various aspects of processing electronic texts. A short historical overview is followed by a description of the most important methodologies (document translation and query translation) and language resources used in connection with them. Regarding the resources, attention is focused on the two- and multilingual ontologies (thesauri, transfer lexicons and similarity thesauri), corpora, their construction and use with the CLIR experiments. The article primarily aims at illustrating the various methodological approaches, while the functioning of particular systems is less prominent. There are no references to the condition of CLIR in Slovenia or to the existence of language resources suitable for processing Slovenian texts in CLIR systems, since this topic calls for a separate review.

Improving Performance of English-Hindi Cross Language Information Retrieval using Transliteration of Query Terms

The main issue in Cross Language Information Retrieval (CLIR) is the poor performance of retrieval in terms of average precision when compared to monolingual retrieval performance. The main reasons behind poor performance of CLIR are mismatching of query terms, lexical ambiguity and un-translated query terms. The existing problems of CLIR are needed to be addressed in order to increase the performance of the CLIR system. In this paper, we are putting our effort to solve the given problem by proposed an algorithm for improving the performance of English-Hindi CLIR system. We used all possible combination of Hindi translated query using transliteration of English query terms and choosing the best query among them for retrieval of documents. The experiment is performed on FIRE 2010 (Forum of Information Retrieval Evaluation) datasets. The experimental result show that the proposed approach gives better performance of English-Hindi CLIR system and also helps in overcoming existing problems and outperforms the existing English-Hindi CLIR system in terms of average precision.

Text Document Retrieval In English Using Keywords of Indonesian Dictionary Based

Abstract— Cross language information regtrieval at the present moment is crucial to the amount of available information use the language variety. Mean while the user who have mastering another language is very less. So with the appointment system of cross language information retrieval is expected.Information retrieval applications on this system to search for text documents in English using keyword of Indonesian dictionary-based. This system was built by several stages starting from collection of documents data by using crawler, entering documents data into a corpus table, punctuation removal, tokenization of documents, stopword removal and term weighting.The ability of this system can add documents data, dictionaries data and stopword data. It is enable to retrieval text documents in English that contains keyword of Indonesian with dictionary-based. Results of documents retrieval was ranked based on the calculation of term weights. Able to see the contents of the document and open the original document. The development of this system can be used as a cross language information retrieval system on dictionary based.Keywords— retrieval information, dictionary based, cross language.

A Flexible Rule-Based Approach to Learn Medical English-Chinese OOV Term Translations from the Web

Out-of-vocabulary (OOV) terms, which do not exist in most dictionaries, usually cause failures in a cross language information retrieval (CLIR) system. Most existing approaches achieve a high performance when using web-mining to translate name entity type OOV terms. However, these methods gain a low performance when they are applied to medical OOV terms because they contain non-Chinese characters which are normally ignored by existing approaches, such as symbols, Roman alphabets and Arabic numbers. This paper presents a flexible rule-based approach towards the acquisition of medical OOV term translation. Our method uses a combination of a novel rule-based pattern extraction and brute force generation to identify the part of non-Chinese characters. To cope with the time-consuming task of ranking list and human extraction of OOV term translation, this paper presents a machine learning method to select correct translations automatically. In the method, twenty-one different features for each Chinese translation candidate are extracted, and the correct Chinese translations are selected by machine learning with our newly proposed statistics filter. By testing our method with 1,654 English ICD9 medical OOV terms, our proposed method (SF+F+W+B+P+S with the base machine learning algorithm SVM) outperforms the existing methods with a recall and precision value of 83.05% and 79.72%, respectively.

Enhanced Named Entity Transliteration Model Using Machine Learning Algorithm

Named entities are important elements in Cross Language Information Retrieval (CLIR) system for locating relevant documents. For translating named entities, compression word format model was designed in the literature, particularly on person names. This model only maps the given source name with the target name stored in the database but it does not generates the target name. The same model is applicable only for the source names which have equivalents in the database. To overcome these problems, a new model called mapping and generation with dynamic learning is proposed. This model transliterates the named entities, particularly person names, places and organization names. The process in the proposed model is to compress the given source name and compare with the target database names. The model accurately maps the source name with the target database names, if the source name has a right equivalent target name and retrieves the equivalent and relevant target names. Otherwise it transliterates the source name and retrieves the relevant target names from the web. These new words need to be updated in the database automatically. To automate the updation process, a dynamic learning algorithm is designed and deployed. The accuracy of the proposed system is improved using learning algorithm when compared with manual updation process.

Cross Language Information Retrieval using Multilingual Ontology as Translation and Query Expansion Base (pp. 13-16)

Abstract —This paper reports an experiment to evaluate a Cross Language Information Retrieval (CLIR) system that uses a multilingual ontology to improve query translation in the travel domain. The ontology-based approach significantly outperformed the Machine Readable Dictionary translation baseline using Mean Average Precision as a metric in a user-centered experiment. Index terms —Ontology, multilingual, cross language information retrieval. I. I NTRODUCTION HE growing requirement on the Internet for users to access information expressed in language other than their own has led to Cross Language Information Retrieval (CLIR) becoming established as a major topic in IR. One approach to CLIR uses different translation approaches to translate queries to documents and indexes in other languages. As queries submitted to search engines suffer lack of context, translation approaches have great problems with resolving query ambiguity. In our approach, we built a multilingual ontology to be used as a translation base for CLIR. In this paper we evaluate our proposed query translation methodology and compare it with a base line system that uses a Machine Readable Dictionary (MRD) as translation base in a user-centered experiment. II. B

Chinese text segmentation: A hybrid approach using transductive learning and statistical association measures

Chinese text segmentation (CTS) is a fundamental step in building any Chinese or cross-language information retrieval system. This paper identifies and proposes solutions to two main challenges facing today’s CTS systems: segmenting words longer than the context window and identifying words not derived from affixation or composition. Our methods exploit unlabeled data, making them scalable at little extra cost. To tackle the first problem, we use a transductive learning approach to automatically construct a dictionary, and then refine it by improving its test set coverage while reducing its over-fitting tendency. In addition, we incorporate frequency information to discriminate overlapping matching words. For the second problem, we employ statistical association measures non-parametrically through a natural but novel feature representation scheme. To demonstrate the generality of our approach, we verify our system on the most reputable CTS evaluation standard – the SIGHAN bakeoff, which contains datasets in both traditional and simplified Chinese. These datasets are provided by representative academic or industrial research institutes. The experimental results show that with only training data and unlabeled test data and with no external dictionaries, our approach effectively overcomes the above-mentioned problems and reduces segmentation errors by an average of 27.8% compared with the traditional approach. Notably, our approach improves the recall of new words, the most informative words, by 4.7% on average. Also, our approach outperforms the best SIGHAN CTS system, which requires many external resources. Additional analysis shows that our approach has the potential to gain accuracy as the test data increases.

Proper nouns in English–Arabic cross language information retrieval

AbstractOut of vocabulary words, mostly proper nouns and technical terms, are one main source of performance degradation in Cross Language Information Retrieval (CLIR) systems. Those are words not found in the dictionary. Bilingual dictionaries in general do not cover most proper nouns, which are usually primary keys in the query. As they are spelling variants of each other in most languages, using an approximate string matching technique against the target database index is the common approach taken to find the target language correspondents of the original query key. N‐gram technique proved to be the most effective among other string matching techniques. The issue arises when the languages dealt with have different alphabets. Transliteration is then applied based on phonetic similarities between the languages involved. In this study, both transliteration and the n‐gram technique are combined to generate possible transliterations in an English–Arabic CLIR system. We refer to this technique as Transliteration N‐Gram (TNG). We further enhance TNG by applying Part Of Speech disambiguation on the set of transliterations so that words with a similar spelling, but a different meaning, are excluded. Experimental results show that TNG gives promising results, and enhanced TNG further improves performance.

Prediction of performance of cross-language information retrieval using automatic evaluation of translation

This study develops regression models for predicting the performance of cross-language information retrieval (CLIR). The model assumes that CLIR performance can be explained by two factors: (1) the ease of search inherent in each query and (2) the translation quality in the process of CLIR systems. As operational variables, monolingual information retrieval (IR) performance is used for measuring the ease of search, and the well-known evaluation metric BLEU is used to measure the translation quality. This study also proposes an alternative metric, weighted average for matched unigrams (WAMU), which is tailored to gauging translation quality for special IR purposes. The data for regression analysis are obtained from a retrieval experiment of English-to-Italian bilingual searches using the CLEF 2003 test collection. The CLIR and monolingual IR performances are measured by average precision score. The result shows that the proposed regression model can explain about 60% of the variation in CLIR performance, and WAMU has more predictive power than BLEU. A back translation method for applying the regression model to operational CLIR systems in real situations is discussed.

Using Web resources to construct multilingual medical thesaurus for cross-language medical information retrieval

Due to the language barrier, non-English users are unable to retrieve the most updated medical information from the U.S. authoritative medical websites, such as PubMed and MedlinePlus. However, currently, there is no any cross-language medical information retrieval (CLMIR) system that can help Chinese-speaking consumers cross the language barrier in finding useful English medical information. A few CLMIR systems utilize MeSH (Medical Subject Headings) to help overcome the language barrier. Unfortunately, the traditional Chinese version of MeSH is currently unavailable. In this paper, we employ a semi-automatic term translation method to construct a Chinese–English MeSH by exploiting abundant multilingual Web resources, including Web anchor texts and search–result pages. Through this method, we have developed a Chinese–English Mesh Compilation System to assist knowledge engineers in compiling a Chinese–English medical thesaurus with more than 19,000 entries. Furthermore, this thesaurus has been used to develop a prototypical system for cross-language medical information retrieval, MMODE, which can help consumers retrieve top-quality English medical information using Chinese terms.

On the role of user-centred evaluation in the advancement of interactive information retrieval

This paper discusses the role of user-centred evaluations as an essential method for researching interactive information retrieval. It draws mainly on the work carried out during the Clarity Project where different user-centred evaluations were run during the lifecycle of a cross-language information retrieval system. The iterative testing was not only instrumental to the development of a usable system, but it enhanced our knowledge of the potential, impact, and actual use of cross-language information retrieval technology. Indeed the role of the user evaluation was dual: by testing a specific prototype it was possible to gain a micro-view and assess the effectiveness of each component of the complex system; by cumulating the result of all the evaluations (in total 43 people were involved) it was possible to build a macro-view of how cross-language retrieval would impact on users and their tasks. By showing the richness of results that can be acquired, this paper aims at stimulating researchers into considering user-centred evaluations as a flexible, adaptable and comprehensive technique for investigating non-traditional information access systems.

Defining and relating biomedical terms: Towards a cross-language morphosemantics-based system

This paper addresses the issue of how semantic information can be automatically assigned to compound terms, i.e. both a definition and a set of semantic relations. This is particularly crucial when elaborating multilingual databases and when developing cross-language information retrieval systems. The paper shows how morphosemantics can contribute in the constitution of multilingual lexical networks in biomedical corpora. It presents a system capable of labelling terms with morphologically related words, i.e. providing them with a definition, and grouping them according to synonymy, hyponymy and proximity relations. The approach requires the interaction of three techniques: (1) a language-specific morphosemantic parser, (2) a multilingual table defining basic relations between word roots and (3) a set of language-independent rules to draw up the list of related terms. This approach has been fully implemented for French, on an about 29,000 terms biomedical lexicon, resulting to more than 3000 lexical families. A validation of the results against a manually annotated file by experts of the domain is presented, followed by a discussion of our method.

A merging strategy proposal: The 2-step retrieval status value method

A usual strategy to implement CLIR (Cross-Language Information Retrieval) systems is the so-called query translation approach. The user query is translated for each language present in the multilingual collection in order to compute an independent monolingual information retrieval process per language. Thus, this approach divides documents according to language. In this way, we obtain as many different collections as languages. After searching in these corpora and obtaining a result list per language, we must merge them in order to provide a single list of retrieved articles. In this paper, we propose an approach to obtain a single list of relevant documents for CLIR systems driven by query translation. This approach, which we call 2-step RSV (RSV: Retrieval Status Value), is based on the re-indexing of the retrieval documents according to the query vocabulary, and it performs noticeably better than traditional methods. The proposed method requires query vocabulary alignment: given a word for a given query, we must know the translation or translations to the other languages. Because this is not always possible, we have researched on a mixed model. This mixed model is applied in order to deal with queries with partial word-level alignment. The results prove that even in this scenario, 2-step RSV performs better than traditional merging methods.

MorphoSaurus

We propose an interlingua-based indexing approach to account for the particular challenges that arise in the design and implementation of cross-language document retrieval systems for the medical domain. Documents, as well as queries, are mapped to a language-independent conceptual layer on which retrieval operations are performed. We contrast this approach with the direct translation of German queries to English ones which, subsequently, are matched against English documents. We evaluate both approaches, interlingua-based and direct translation, on a large medical document collection, the OHSUMED corpus. A substantial benefit for interlingua-based document retrieval using German queries on English texts is found, which amounts to 93% of the (monolingual) English baseline. Most state-of-the-art cross-language information retrieval systems translate user queries to the language(s) of the target documents. In contra-distinction to this approach, translating both documents and user queries into a language-independent, concept-like representation format is more beneficial to enhance cross-language retrieval performance.

Dictionary-based techniques for cross-language information retrieval

Cross-language information retrieval (CLIR) systems allow users to find documents written in different languages from that of their query. Simple knowledge structures such as bilingual term lists have proven to be a remarkably useful basis for bridging that language gap. A broad array of dictionary-based techniques have demonstrated utility, but comparison across techniques has been difficult because evaluation results often span only a limited range of conditions. This article identifies the key issues in dictionary-based CLIR, develops unified frameworks for term selection and term translation that help to explain the relationships among existing techniques, and illustrates the effect of those techniques using four contrasting languages for systematic experiments with a uniform query translation architecture. Key results include identification of a previously unseen dependence of pre- and post-translation expansion on orthographic cognates and development of a query-specific measure for translation fanout that helps to explain the utility of structured query methods.

Translation events in cross-language information retrieval

Cross-Language Information Retrieval (CLIR) systems enable users to formulate queries in their native language to retrieve documents in foreign languages. Because queries and documents in CLIR do not necessarily share the same language, translation is needed before matching can take place. This translation step tends to cause a reduction in the retrieval performance of CLIR as compared to monolingual information retrieval. To understand the events occurring in crosslanguage retrieval query translation and the relation of these events to retrieval performance, the study explored the following research questions: