Cross-language Information Retrieval Research Articles

Spoken language identification based on the transcript analysis

Abstract Language identification is a great challenge in language engineering, which arises along with the tasks of speech recognition, machine translation, cross-language information retrieval, intelligent dialogue system creation, etc. The presented article introduces the intelligent language identification technology, which is based on speech recognition and statistical methods of spectrogram analysis. The approach to the automatic identification of the spoken language sample uploaded to the system, in particular from video streaming services such as YouTube, is put forward. The article focuses on the automatic identification of spoken language, taking into account several speech recognition solutions for correct or incorrect speech recognition and its conversion into correct or incorrect text. The obtained algorithm is demonstrated in the Ukrainian and Russian languages. The identification quality of the language of an utterance, which lasts >30 s is almost 100%, and for the utterance of a duration of 30 s, the quality is 98%, and for the 5-s utterance, it reaches 89.6%. In addition to that, the system performance is contingent on the streaming speed, so it is a real-time system.

A Comparative Optimization Model of Japanese Literature Characteristics for Cognitive Retrieval of Cross-Language Information.

Cross-language information cognitive retrieval has grown in importance as a study area due to the multilingual character of Internet resources and the diversifying languages spoken by users. This paper analyses and illustrates the key characteristics of Japanese literature from three aspects: ideological structure, structural form, and emotional expression, and makes a straightforward comparison with Chinese literature using a number of well-known Japanese literary masterpieces as examples. An important aspect of this literature is that it is full of lingering feelings, leaving room for the readers to imagine. The lingering charm is endless, obscure, and meaningful, which is the traditional style of Japanese literature. It also focuses on the depoliticization of Japanese literature, that is, the relative separation between literature and politics. The subjectivity of Japanese writing, which goes beyond just conveying feelings but involves less objective description, is related to this. The process of cross-language information retrieval in foreign nations is primarily separated into three stages, according to the changes in research objects. Currently, adding a language conversion mechanism to a monolingual information retrieval system serves as the primary language information retrieval solution. Currently, nearly 40% of the global popularity of literature comes from Japan. In this article, the background and concept of cross-language information retrieval are introduced, and its types, system models, and several key cross-language information retrieval methods are explained, and some solutions to the factors influencing the cross-language information retrieval effect are suggested.

An Ontology based Smart Management of Linguistic Knowledge

Natural language processing provides a very significant contribution to various application areas such as multilingual big data, information retrieval, data integration and multilingual web. However, handling linguistic knowledge to develop such lingware applications is a crucial issue, especially for linguistic novice users. To deal with this issue, a "smart" linguistic knowledge management may help the users to understand the meaning, scope and especially the use of related techniques and algorithms. In this paper, (1) we propose a semantic processing of linguistic knowledge based on a multilingual linguistic domain ontology, called LingOnto. Compared to related work, LingOnto does not only handles linguistic data, but also linguistic processing functionalities and linguistic processing features. Besides, it allows, via a reasoning engine, inferring new linguistic knowledge and assisting in the process of proposing lingware applications. This is particularly useful for novice users, but can also provide new perspectives for the expert ones. LingOnto covers the French, English and Arabic languages. (2) We propose also an assisted user friendly ontology visualization tool called LingGraph. It facilitates the interaction with LingOnto. It offers an easy to use interface for users not familiar with ontologies. It is based on a SPARQL pattern-based approach to allow a smart search interaction functionality to visualize only the ontological view corresponding to the user’s needs and preferences. In order to evaluate LingOnto, we apply it to a framework of identifying valid natural language processing pipelines. Finally, we give the results of the carried-out experiments.

Mind the Gap: Cross-Lingual Information Retrieval with Hierarchical Knowledge Enhancement

Cross-Lingual Information Retrieval (CLIR) aims to rank the documents written in a language different from the user’s query. The intrinsic gap between different languages is an essential challenge for CLIR. In this paper, we introduce the multilingual knowledge graph (KG) to the CLIR task due to the sufficient information of entities in multiple languages. It is regarded as a “silver bullet” to simultaneously perform explicit alignment between queries and documents and also broaden the representations of queries. And we propose a model named CLIR with HIerarchical Knowledge Enhancement (HIKE) for our task. The proposed model encodes the textual information in queries, documents and the KG with multilingual BERT, and incorporates the KG information in the query-document matching process with a hierarchical information fusion mechanism. Particularly, HIKE first integrates the entities and their neighborhood in KG into query representations with a knowledge-level fusion, then combines the knowledge from both source and target languages to further mitigate the linguistic gap with a language-level fusion. Finally, experimental results demonstrate that HIKE achieves substantial improvements over state-of-the-art competitors.

Neural topic-enhanced cross-lingual word embeddings for CLIR

Cross-lingual information retrieval (CLIR) methods have quickly made the transition from translation-based approaches to semantic-based approaches. In this paper, we examine the limitations of current unsupervised neural CLIR methods, especially those leveraging aligned cross-lingual word embedding (CLWE) spaces. At the moment, CLWEs are normally constructed on the monolingual corpus of bilingual texts through an iterative induction process. Homonymy and polysemy have become major obstacles in this process. On the other hand, contextual text representation methods often fail to outperform static CLWE methods significantly for CLIR. We propose a method utilizing a novel neural generative model with Wasserstein autoencoders to learn neural topic-enhanced CLWEs for CLIR purposes. Our method requires minimal or no supervision at all. On the CLEF test collections, we perform a comparative evaluation of the state-of-the-art semantic CLWE methods along with our proposed method for neural CLIR tasks. We demonstrate that our method outperforms the existing CLWE methods and multilingual contextual text encoders. We also show that our proposed method obtains significant improvements over the CLWE methods based upon representative topical embeddings.

Mining an English-Chinese parallel Dataset of Financial News

Parallel text datasets are a valuable for educational purposes, machine translation, and cross-language information retrieval, but few are domain-oriented. We have created a Chinese–English parallel dataset in the domain of finance technology, using the <em>Financial Times</em> website, from which we grabbed 60,473 news items from between 2007 and 2021. This dataset is a bilingual Chinese–English parallel dataset of news in the domain of finance. It is open access in its original state without transformation, and has been made not for machine translation as has been used, but for intelligent mining, in which we conducted many experiments using up-to-date text mining techniques: clustering (topic modeling, community detection, <em>k</em>-means), topic prediction (naive Bayes, SVM, LSTM, Bert), and pattern discovery (dictionary based, time series). We present the usage of these techniques as a framework for other studies, not only as an application but with an interpretation.

Learning Bilingual Word Embedding Mappings with Similar Words in Related Languages Using GAN

ABSTRACT Cross-lingual word embeddings display words from different languages in the same vector space. They provide reasoning about semantics, compare the meaning of words across languages and word meaning in multilingual contexts, necessary to bilingual lexicon induction, machine translation, and cross-lingual information retrieval. This paper proposes an efficient approach to learn bilingual transform mapping between monolingual word embeddings in language pairs. We choose ten different languages from three different language families and downloaded their last update Wikipedia dumps1 1. https://dumps.wikimedia.org. Then, with some pre-processing steps and using word2vec, we produce word embeddings for them. We select seven language pairs from chosen languages. Since the selected languages are relative, they have thousands of identical words with similar meanings. With these identical dictation words and word embedding models of each language, we create training, validation and, test sets for the language pairs. We then use a generative adversarial network (GAN) to learn the transform mapping between word embeddings of source and target languages. The average accuracy of our proposed method in all language pairs is 71.34%. The highest accuracy is achieved for the Turkish-Azerbaijani language pair with the accuracy 78.32%., which is noticeably higher than prior methods.

Transliteration: A Magnetic Analysis

Abstract: Machine transliteration is an important problem in an increasingly multilingual world as it plays a critical role in many downstream applications such as machine translation or cross-lingual information retrieval systems. There is now a vast amount of information accessible via the Internet where a lot of regional and cultural information is put on the World Wide Web in different languages and scripts. There are more that six thousand living languages in the world. Adding to the diversity is the fact that some languages are written in different scripts in different regions of the world. The multitude of foreign languages and mutually incomprehensible scripts of the same language pose a barrier to information exchange as we cannot all learn every language or script in use worldwide. Therefore, if we can get around the language barrier or at least the script barrier, we can access much more of the world's culture and can explore its abundant richness. Keywords: Transliteration, Translation. Cross-lingual, Multilingual, Language, Script

A Review on Indexing Techniques and its application in Multilingual Information Retrieval System

To implement the indexing in multilingual dataset, the indexing process must know. This paper gives the brief about indexing and presents role of indexing, logical view of indexing, types of indexing, indexing process, data structure of indexing, representation of indexing and indexing framework. Indexing is an important concept to implement in retrieval process. By the indexing, retrieval process will reduce the time. The indexing part performs the complete article pool to shape index constructions that permit for resourceful search. These are typically inverted tables. Each article is so preprocessed and recorded to a vector demonstration. At the end indexing method is applied to the multilingual information retrieval system and shows the results.

English-Vietnamese Cross-Lingual Paraphrase Identification Using MT-DNN

Paraphrase identification is a crucial task in natural language understanding, especially in cross-language information retrieval. Nowadays, Multi-Task Deep Neural Network (MT-DNN) has become a state-of-the-art method that brings outstanding results in paraphrase identification [1]. In this paper, our proposed method based on MT-DNN [2] to detect similarities between English and Vietnamese sentences, is proposed. We changed the shared layers of the original MT-DNN from original the BERT [3] to other pre-trained multi-language models such as M-BERT [3] or XLM-R [4] so that our model could work on cross-language (in our case, English and Vietnamese) information retrieval. We also added some tasks as improvements to gain better results. As a result, we gained 2.3% and 2.5% increase in evaluated accuracy and F1. The proposed method was also implemented on other language pairs such as English – German and English – French. With those implementations, we got a 1.0%/0.7% improvement for English – German and a 0.7%/0.5% increase for English – French.

Characteristics recognition and soft multimedia system for Japanese machine translation and edge-driven hardware implementations

With the development of recent economic globalization, international exchanges and cooperation are increasingly frequent and in-depth. In this process, there is a serious obstacle, that is, language differences. Therefore, the development of high-quality and practical machine translation system is of great significance. Japan has a close relationship with China, so it is necessary to acquire and process Japanese information. Japanese translation is the basis of Japanese information processing, which plays an important role in cross language information retrieval, machine translation, information extraction and other practical applications. In recent years, machine translation has made remarkable progress, but there is still much room for improvement in the quality of translation. Multimedia assisted instruction is an important application of computer technology. Therefore, this paper realizes the feature recognition and hardware structure of Japanese machine translation; for the efficient implementation of the model, the edge-driven model is combined. The proposed model is implemented through the edge devices and the performance of the model is validated through the testing. The recognition accuracy is much higher than the traditional models and the robustness is higher.

Efficient bilingual lexicon extraction from comparable corpora based on formal concepts analysis

AbstractBilingual corpora are an essential resource used to cross the language barrier in multilingual natural language processing tasks. Among bilingual corpora, comparable corpora have been the subject of many studies as they are both frequent and easily available. In this paper, we propose to make use of formal concept analysis to first construct concept vectors which can be used to enhance comparable corpora through clustering techniques. We then show how one can extract bilingual lexicons of improved quality from these enhanced corpora. We finally show that the bilingual lexicons obtained can complement existing bilingual dictionaries and improve cross-language information retrieval systems.

Exploring Topic-language Preferences in Multilingual Swahili Information Retrieval in Tanzania

Habitual switching of languages is a common behaviour among polyglots when searching for information on the Web. Studies in information retrieval (IR) and multilingual information retrieval (MLIR) suggest that part of the reason for such regular switching of languages is the topic of search. Unlike survey-based studies, this study uses query and click-through logs. It exploits the querying and results selection behaviour of Swahili MLIR system users to explore how topic of search (query) is associated with language preferences—topic-language preferences. This article is based on a carefully controlled study using Swahili-speaking Web users in Tanzania who interacted with a guided multilingual search engine. From the statistical analysis of queries and click-through logs, it was revealed that language preferences may be associated with the topics of search. The results also suggest that language preferences are not static; they vary along the course of Web search from query to results selection. In most of the topics, users either had significantly no language preference or preferred to query in Kiswahili and changed their preference to either English or no preference for language when selecting/clicking on the results. The findings of this study might provide researchers with more insights in developing better MLIR systems that support certain types of users and in certain scenarios.

English Audio Language Retrieval Based on Adaptive Speech-Adjusting Algorithm

For the purpose of language retrieval for English listening, this paper designs and implements a cross-language information retrieval system for English listening. Different implementation methods of cross-language information retrieval, query, and translation are analyzed. The system adopts cross-language information retrieval technology based on bilingual dictionaries. According to the cross-language retrieval system of the existing bilingual dictionaries and monolingual dictionaries, based on the design and implementation of the fuzzy search dictionary lookup mechanism, the existing dictionary lookup mechanism is constructed and analyzed. Aiming at the problem of translation ambiguity in information retrieval systems based on bilingual dictionaries, a disambiguities elimination algorithm based on cooccurrence technology is proposed. In continuous speech, the speed of different speakers in different contexts is very different. Deviation from normal speech speed often leads to recognition errors, which makes recognition performance decline. Considering that the influence of speech speed on the length of speech units increases or decreases synchronously, and there is a strong correlation between the lengths of adjacent speech units, an adaptive speech speed algorithm is proposed based on the framework of implicit Markov model based on the information of the length of speech units. Experiments on number string and large vocabulary continuous speech recognition show that the algorithm is effective.

A Learning to rank framework based on cross-lingual loss function for cross-lingual information retrieval

A Learning to rank framework based on cross-lingual loss function for cross-lingual information retrieval

Semantic morphological variant selection and translation disambiguation for cross-lingual information retrieval

Semantic morphological variant selection and translation disambiguation for cross-lingual information retrieval

Effective preprocessing based neural machine translation for English to Telugu cross-language information retrieval

&lt;span id="docs-internal-guid-5b69f940-7fff-f443-1f09-a00e5e983714"&gt;&lt;span&gt;In cross-language information retrieval (CLIR), the neural machine translation (NMT) plays a vital role. CLIR retrieves the information written in a language which is different from the user's query language. In CLIR, the main concern is to translate the user query from the source language to the target language. NMT is useful for translating the data from one language to another. NMT has better accuracy for different languages like English to German and so-on. In this paper, NMT has applied for translating English to Indian languages, especially for Telugu. Besides NMT, an effort is also made to improve accuracy by applying effective preprocessing mechanism. The role of effective preprocessing in improving accuracy will be less but countable. Machine translation (MT) is a data-driven approach where parallel corpus will act as input in MT. NMT requires a massive amount of parallel corpus for performing the translation. Building an English - Telugu parallel corpus is costly because they are resource-poor languages. Different mechanisms are available for preparing the parallel corpus. The major issue in preparing parallel corpus is data replication that is handled during preprocessing. The other issue in machine translation is the out-of-vocabulary (OOV) problem. Earlier dictionaries are used to handle OOV problems. To overcome this problem the rare words are segmented into sequences of subwords during preprocessing. The parameters like accuracy, perplexity, cross-entropy and BLEU scores shows better translation quality for NMT with effective preprocessing.&lt;/span&gt;&lt;/span&gt;

Kelantan and Sarawak Malay Dialects: Parallel Dialect Text Collection and Alignment Using Hybrid Distance-Statistical-Based Phrase Alignment Algorithm

Parallel texts corpora are essential resources especially in translation and multilingual information retrieval. However, the publicly available parallel text corpora are limited to certain types and domains. Besides, Malay dialects are not standardized in term of writing. The existing alignment algorithms that is used to analayze the writing will require a large training data to obtain a good result. The paper describes our methodology in acquiring a parallel text corpus of Standard Malay and Malay dialects, particularly Kelantan Malay and Sarawak Malay. Second, we propose a hybrid of distance-based and statistical-based alignment algorithm to align words and phrases of the parallel text. The proposed approach has a better precision and recall than the state-of-the-art GIZA++. In the paper, the alignment obtained were also compared to find out the lexical similarities and differences between SM and the two dialects.

CLOUD BASED MULTI-LANGUAGE INDEXING USING CROSS LINGUAL INFORMATION RETRIEVAL APPROACHES

The exponential growth of data sizes created by digital media (video/audio/images), physicalsimulations, scientific instruments and web authoring joins the new growth of interest in cloud computing. The options for distribution and parallelization of information in clouds make the retrieval and storage processes very complicated, especially when faced with real-time data management. The quantity of Web Users getting access to data over Internet is expanding step by step. An enormous measure of data on Internet is accessible in various languages which could be accessed by anyone whenever. The Information Retrieval (IR) manages finding valuable data from a huge assortment of unorganized, organized and semi-organized information. In the present situation, the variety of data and language boundaries are the difficult challenges for communication and social trade over the world. To tackle such obstructions, CLIR, the cross-language information retrieval frameworks, are these days in solid interest. The Query Expansion (Q.E.) is the way toward adding related and important terms to original inquiry to upgrade its indexing ability to improve the significance of recovered files in CLIR. In this exploration work, Q.E. has been investigated for a Hindi-English and Kannada-English CLIR in that Hindi and Kannada queries are utilized to look through English docs. After the interpretation of query, recovered outcomes are positioned making use of OkapiBM25 to organize the most important doc at the top for expanding the significance of recovered docs using QE. We proposed architecture for Hindi-English and Kannada-English CLIR making use of QE. to improve the importance of recovered reports. In the primary investigation, QE. is performed with and without OkapiBM25 ranking. The outcomes show that the pertinence of recovered archives is higher with OKapiBM25 as contrast with the one without positioning. The work docs plainly demonstrate that the presentation of Hindi-English and Kannada-English CLIR framework can be improved altogether with query development using fitting terms located at suitable place and the recovered Snippets can incredibly fill in as the continuous test collection.

Parallel sentence extraction to improve cross-language information retrieval from Wikipedia

Translation language resources, such as bilingual word lists and parallel corpora, are important factors affecting the effectiveness of cross-language information retrieval (CLIR) systems. In particular, when large domain-appropriate parallel corpora are not available, developing an effective CLIR system is particularly difficult. Furthermore, creating a large parallel corpus is costly and requires considerable effort. Therefore, we here demonstrate the construction of parallel corpora from Wikipedia as well as improved query translation, wherein the queries are used for a CLIR system. To do so, we first constructed a bilingual dictionary, termed WikiDic. Then, we evaluated individual language resources and combinations of them in terms of their ability to extract parallel sentences; the combinations of our proposed WikiDic with the translation probability from the Web’s bilingual example sentence pairs and WikiDic was found to be best suited to parallel sentence extraction. Finally, to evaluate the parallel corpus generated from this best combination of language resources, we compared its performance in query translation for CLIR to that of a manually created English–Korean parallel corpus. As a result, the corpus generated by our proposed method achieved a better performance than did the manually created corpus, thus demonstrating the effectiveness of the proposed method for automatic parallel corpus extraction. Not only can the method demonstrated herein be used to inform the construction of other parallel corpora from language resources that are readily available, but also, the parallel sentence extraction method will naturally improve as Wikipedia continues to be used and its content develops.