Language-independent Algorithm Research Articles

A Deep Neural Network Framework for English Hindi Question Answering

In this article, we propose a unified deep neural network framework for multilingual question answering (QA). The proposed network deals with the multilingual questions and answers snippets. The input to the network is a pair of factoid question and snippet in the multilingual environment (English and Hindi), and output is the relevant answer from the snippet. We begin by generating the snippet using a graph-based language-independent algorithm, which exploits the lexico-semantic similarity between the sentences. The soft alignment of the question words from the English and Hindi languages has been used to learn the shared representation of the question. The learned shared representation of question and attention-based snippet representation are passed as an input to the answer extraction layer of the network, which extracts the answer span from the snippet. Evaluation on a standard multilingual QA dataset shows the state-of-the-art performance with 39.44 Exact Match (EM) and 44.97 F1 values. Similarly, we achieve the performance of 50.11 Exact Match (EM) and 53.77 F1 values on Translated SQuAD dataset.

Machine Learning Methods for Detecting and Monitoring Extremist Information on the Internet

In this paper, we employ machine learning methods to solve the problem of countering terrorism and extremism by using information from the Internet. This problem involves retrieving electronic messages, documents, and web resources that potentially contain information of terrorist or extremist nature, identifying the structure of user groups and online communities that disseminate this information, monitoring and modeling information flows in these communities, as well as assessing threats and predicting risks based on monitoring results. We propose some original language-independent algorithms for pattern-based information retrieval, thematic modeling, and prediction of message flow characteristics, as well as assessment and prediction of potential risk coming from members of online communities by using data on the structure of relations in these communities, which makes it possible to detect potentially dangerous users even without full access to the content they distribute, e.g., through private channels and chat rooms.

Real-time Language-Independent Algorithm for Dialogue Agents

Real-time Language-Independent Algorithm for Dialogue Agents

Novel harmony search-based algorithms for part-of-speech tagging

As a fast and high-quality tagger algorithm is a crucial task in natural language processing, this paper presents novel language-independent algorithms based on harmony search (HS) optimization met...

APPLYING SIMILARITY MEASURES FOR AUTOMATIC LEMMATIZATION: A CASE STUDY FOR MODERN GREEK AND ENGLISH

This paper addresses the problem of automatic induction of the normalized form (lemma) of regular and mildly irregular words with no direct supervision using language-independent algorithms. More specifically, two string distance metric models (i.e. the Levenshtein Edit Distance algorithm and the Dice Coefficient similarity measure) were employed in order to deal with the automatic word lemmatization task by combining two alignment models based on the string similarity and the most frequent inflectional suffixes. The performance of the proposed model has been evaluated quantitatively and qualitatively. Experiments were performed for the Modern Greek and English languages and the results, which are set within the state-of-the-art, have showed that the proposed model is robust (for a variety of languages) and computationally efficient. The proposed model may be useful as a pre-processing tool to various language engineering and text mining applications such as spell-checkers, electronic dictionaries, morphological analyzers etc.

Unlimited vocabulary speech recognition with morph language models applied to Finnish

In the speech recognition of highly inflecting or compounding languages, the traditional word-based language modeling is problematic. As the number of distinct word forms can grow very large, it becomes difficult to train language models that are both effective and cover the words of the language well. In the literature, several methods have been proposed for basing the language modeling on sub-word units instead of whole words. However, to our knowledge, considerable improvements in speech recognition performance have not been reported. In this article, we present a language-independent algorithm for discovering word fragments in an unsupervised manner from text. The algorithm uses the Minimum Description Length principle to find an inventory of word fragments that is compact but models the training text effectively. Language modeling and speech recognition experiments show that n-gram models built over these fragments perform better than n-gram models based on words. In two Finnish recognition tasks, relative error rate reductions between 12% and 31% are obtained. In addition, our experiments suggest that word fragments obtained using grammatical rules do not outperform the fragments discovered from text. We also present our recognition system and discuss how utilizing fragments instead of words affects the decoding process.

The role of trace abstractions in program specialization algorithms

We summarise the role of concepts such as neighbourhoods, characteristic trees, characteristic atoms, and trace terms in the design of program specialisation algorithms. These are techniques, developed in different programming languages, of describing abstractions that can be termed the shape of computations. They have been introduced into program specialisation algorithms mainly for the control of generalisation and polyvariance (the number of different specialised versions of a procedure). We argue that these methods can play a more fundamental role and point the way towards language-independent algorithms for program specialisation, which rely only only a general notion of a program trace.