Rare Words Research Articles

The Rare Word Issue in Natural Language Generation: A Character-Based Solution

In this paper, we analyze the problem of generating fluent English utterances from tabular data, focusing on the development of a sequence-to-sequence neural model which shows two major features: the ability to read and generate character-wise, and the ability to switch between generating and copying characters from the input: an essential feature when inputs contain rare words like proper names, telephone numbers, or foreign words. Working with characters instead of words is a challenge that can bring problems such as increasing the difficulty of the training phase and a bigger error probability during inference. Nevertheless, our work shows that these issues can be solved and efforts are repaid by the creation of a fully end-to-end system, whose inputs and outputs are not constrained to be part of a predefined vocabulary, like in word-based models. Furthermore, our copying technique is integrated with an innovative shift mechanism, which enhances the ability to produce outputs directly from inputs. We assess performance on the E2E dataset, the benchmark used for the E2E NLG challenge, and on a modified version of it, created to highlight the rare word copying capabilities of our model. The results demonstrate clear improvements over the baseline and promising performance compared to recent techniques in the literature.

From context-aware to knowledge-aware: Boosting OOV tokens recognition in slot tagging with background knowledge

Neural-based context-aware models for slot tagging tasks in language understanding have achieved state-of-the-art performance, especially deep contextualized models, such as ELMo, BERT. However, the presence of out-of-vocab (OOV) words significantly degrades the performance of neural-based models, especially in a few-shot scenario. In this paper, we propose a novel knowledge-aware slot tagging model to integrate contextual representation of input text and the large-scale lexical background knowledge. Besides, we use multi-level graph attention to explicitly reason via lexical relations. We aim to leverage both linguistic regularities covered by deep language models (LM) and high-quality background knowledge derived from curated knowledge bases (KB). Consequently, our model could infer rare and unseen words in the test dataset by incorporating contextual semantics learned from the training dataset and lexical relations from ontology. The experiments show that our proposed knowledge integration mechanism achieves consistent improvements across settings with different sizes of training data on two public benchmark datasets. We also show through detailed analysis that incorporating background knowledge effectively alleviates issues of data scarcity.

Two halves of a meaningful text are statistically different

Which statistical features distinguish a meaningful text (possibly written in an unknown system) from a meaningless set of symbols? Here we answer this question by comparing features of the first half of a text to its second half. This comparison can uncover hidden effects, because the halves have the same values of many parameters (style, genre, etc). We found that the first half has more different words and more rare words than the second half. Also, words in the first half are distributed less homogeneously over the text. These differences hold for the significant majority of several hundred relatively short texts we studied. Differences disappear after a random permutation of words that destroys the linear structure of the text. The differences reveal a temporal asymmetry in meaningful texts, which is confirmed by showing that texts are much better compressible in their natural way (i.e. along the narrative) than in the word-inverted form. We conjecture that these results connect the semantic organization of a text (defined by the flow of its narrative) to its statistical features.

A hybrid naïve Bayes based on similarity measure to optimize the mixed-data classification

In this paper, a hybrid method has been introduced to improve the classification performance of naive Bayes (NB) for the mixed dataset and multi-class problems. This proposed method relies on a similarity measure which is applied to portions that are not correctly classified by NB. Since the data contains a multi-valued short text with rare words that limit the NB performance, we have employed an adapted selective classifier based on similarities (CSBS) classifier to exceed the NB limitations and included the rare words in the computation. This action has been achieved by transforming the formula from the product of the probabilities of the categorical variable to its sum weighted by numerical variable. The proposed algorithm has been experimented on card payment transaction data that contains the label of transactions: the multi-valued short text and the transaction amount. Based on K-fold cross validation, the evaluation results confirm that the proposed method achieved better results in terms of precision, recall, and F-score compared to NB and CSBS classifiers separately. Besides, the fact of converting a product form to a sum gives more chance to rare words to optimize the text classification, which is another advantage of the proposed method.

WITHDRAWN: Improving feature engineering by fine tuning the parameters of Skip gram model

Word Embedding is the primary task in text mining, a subfield of Natural Language Processing. Feature Engineering also referred as feature extraction deals with building features out of existing data. This process involves constructing right features from the dataset and feeding relevant and best features into the Machine learning model for training. Feature engineering leads to greater improvements on many NLP tasks. The various aspects of feature engineering includes feature selection, feature addition, feature filtering and feature scaling. The Skip-gram architecture works better than other feature extraction models like CBOW, TD-IDF etc. This unsupervised learning technique used to find context word for a given target word. The skip-gram word2vec model outclasses the shortcomings with word-embedding matrix like loss of information from former cell, noisy information and over-fitting problems. The Skip gram algorithm trains the neural network and each time the output will be a softmax vector and it computes the cross entropy loss between output vector and true one-hot encoder. Skip-gram word2vec model shows good representation over finding out rare words or phrases. To improve the computational speed and accuracy, the parameters of skip gram model are fine tuned. Experimental results proves the efficiency of the model.

The newspaper “Babochka” (1829–1832) as a linguistic source

The article deals with the newspaper “Babochka” as an important source for philological research, which objectively reflects the linguistic processes of its time. Translated articles selected from leading periodicals in Europe and America, creatively revised by the authors, included the Russian reader in the world media space. A differential approach is used in the article that focuses primarily on the dynamic elements of the lexical and semantic system. The newspaper presents the innovations of the early 19th century, including borrowings, foreign language inclusions, complex adjectives formed in Russian, and dialect words. As a result of the analysis of the source, the emergence of new meanings for words already in use was noted, the dating of a number of new lexemes was clarified, and contexts for their semantization were identified. The work concludes that the rare words and rare word usage recorded in the texts of the newspaper “Babochka” can be considered as valuable materials for historical lexicology.

Burstiness-Aware Web Search Analysis on Different Levels of Evidences

Personalizing the analysis for web search potentially improves the search experience. A good analytical model for web search should leverage not only collective wisdom but also individual characteristics. Most of the existing analytical models, however, focus on how to utilize the collective wisdom, from a crowd, for instance. In this paper, we address the problem of user-specific web search analysis by considering the so-called burstiness in web search, which captures the behavior of rare words appearing many times in a single document. We go beyond click graph and propose two probabilistic topic models, Topic Independence Model and Topic Dependence Model. The former adopts the assumption that the generation of query terms and URLs are topically independent, and the latter captures the coupling between search queries and URLs. We also capture the temporal burstiness of topics by utilizing continuous Beta distribution. Through a large-scale analysis of a real-life search query log, we observe that each user's web search trail enjoys multiple kinds of user-based unique characteristics. On a massive search query log, the new models achieve a better held-out likelihood than existing baselines, and they can also effectively reveal the latent evolution of topics on the corpus level and user-based level.

Re-Transformer: A Self-Attention Based Model for Machine Translation

Machine translation is one of the most popular and hardest tasks in Natural Language Processing. This paper proposes a self-attention based model for machine translation, named Re-Transformer, by transforming the Transformer [1]. Different from prevailing approach through module or GPU stacking to improve the system performance, Re-Transformer modifies the basic architecture; there are four refinements as follows. First, Re-Transformer adopts sub-word Tokenization in corpus preprocessing to overcome rare words. In the encoder layer, dual Self-Attention stacks and less Point Wise Feed Forward layer are adopted to obtain better comprehension of an input sentence. In decoder, reduced the stack of “Decoder” are used to speedup training and inferring speed meanwhile keeps the same level of BLEU. The experiment results show that Re-Transformer with BLEU metric score 31.36, 38.45 (four-layer decoder) and 32.14, 55.62 (two-layer decoder) improves around 4 and 17 points of BLEU metric against the Transformer over the WMT 2014 English-German and English-French Translation Corpus.

On some sources of Sagang Sechen’s Teachings (1662)

AbstractNine of the seventy-nine alliterative quatrains of Sagang Sechen’s great gnomic poem are revisited, their possible literary sources suggested, their interpretation revised. Seven of them go back, entirely or partially, to Sa-skya Paṇḍita’s Subhāṣitaratnanidhi, one to the Janapoṣanabindhu, one of Nāgārjuna’s nītiśāstras, and one uses a comparison known from the Secret History. Parallels are quoted from Sonom Gara’s and the Oirat Zaya Paṇḍita’s prose translations of the Subhāṣitaratnanidhi. Also discussed are the rare word küčigei and the possible identity of Sonom Gara and Suonanqilo.

Topic Modeling in Embedding Spaces

Topic modeling analyzes documents to learn meaningful patterns of words. However, existing topic models fail to learn interpretable topics when working with large and heavy-tailed vocabularies. To this end, we develop the embedded topic model (etm), a generative model of documents that marries traditional topic models with word embeddings. More specifically, the etm models each word with a categorical distribution whose natural parameter is the inner product between the word’s embedding and an embedding of its assigned topic. To fit the etm, we develop an efficient amortized variational inference algorithm. The etm discovers interpretable topics even with large vocabularies that include rare words and stop words. It outperforms existing document models, such as latent Dirichlet allocation, in terms of both topic quality and predictive performance.

Analyzing Subword Techniques to Improve English to Sinhala Neural Machine Translation

Neural machine translation (NMT) is a remarkable approach which performs much better than the Statistical machine translation (SMT) models when there is an abundance of parallel corpus. However, vanilla NMT is primarily based upon word-level with a fixed vocabulary. Therefore, low resource morphologically rich languages such as Sinhala are mostly affected by the out of vocabulary (OOV) and Rare word problems. Recent advancements in subword techniques have opened up opportunities for low resource communities by enabling open vocabulary translation. In this paper, we extend our recently published state-of-the-art EN-SI translation system using the transformer and explore standard subword techniques on top of it to identify which subword approach has a greater effect on English Sinhala language pair. Our models demonstrate that subword segmentation strategies along with the state-of-the-art NMT can perform remarkably when translating English sentences into a rich morphology language regardless of a large parallel corpus.

Budowa i zastosowania korpusu monitorującego MoncoPL

This paper introduces the methodology of compiling and maintaining MoncoPL, a large monitor corpus of web-based Polish. Furthermore, an overview of the search engine of the same name is provided to show how the size and composition of the corpus, currently reaching over 5.6 billion word tokens, facilitates research on distributional properties of rare words, neologisms and phraseological units. Finally, the article exemplifies some advantages of using a densely-sampled diachronic corpus for the purposes of observing frequency trends and cycles of various constructions in online media discourse.

How the Bible Is Written

How the Bible Is Written

A Neural-Network-Based Approach to Chinese–Uyghur Organization Name Translation

The recognition and translation of organization names (ONs) is challenging due to the complex structures and high variability involved. ONs consist not only of common generic words but also names, rare words, abbreviations and business and industry jargon. ONs are a sub-class of named entity (NE) phrases, which convey key information in text. As such, the correct translation of ONs is critical for machine translation and cross-lingual information retrieval. The existing Chinese–Uyghur neural machine translation systems have performed poorly when applied to ON translation tasks. As there are no publicly available Chinese–Uyghur ON translation corpora, an ON translation corpus is developed here, which includes 191,641 ON translation pairs. A word segmentation approach involving characterization, tagged characterization, byte pair encoding (BPE) and syllabification is proposed here for ON translation tasks. A recurrent neural network (RNN) attention framework and transformer are adapted here for ON translation tasks with different sequence granularities. The experimental results indicate that the transformer model not only outperforms the RNN attention model but also benefits from the proposed word segmentation approach. In addition, a Chinese–Uyghur ON translation system is developed here to automatically generate new translation pairs. This work significantly improves Chinese–Uyghur ON translation and can be applied to improve Chinese–Uyghur machine translation and cross-lingual information retrieval. It can also easily be extended to other agglutinative languages.

Authorship Attribution of Short Historical Arabic Texts using Stylometric Features and a KNN Classifier with Limited Training Data

Authorship Attribution (AA) is a task that aims to recognize the authorship of unknown texts based on writing style. Out of the various approaches to solve the AA problem, Stylometry is a promising one. This paper explores the use of a K-Nearest Neighbor (KNN) classifier combined with stylometry features to perform AA. This study indicates the robustness of KNN in performing AA on short historical Arabic texts written by different authors. To classify the texts according to the author, KNN was trained with a set of stylometry features including rare words, count characters and 2-, 3- and 4-grams character levels. Various feature set sizes ranging from 34 to 2000 were tested in the experiment. The experiments were conducted on limited training data with datasets consisting of 3 short texts per the author’s book. This method proved to be at least as effective as Information Gain (IG) when selecting the most significant n-grams. Moreover, the KNN classifier achieved high accuracy results with the best classification accuracy of up to 90%, except for the 5-KK using the 4-gram character level. This work contributes towards utilizing KNN for identifying the distinctive stylometry feature for robust AA identification in short historical Arabic texts.

Occasional and Rare Lexemes in the Poetic Language of V. A. Zhukovsky (Methodology of Dictionary Interpretation)

The article deals with the question of the lexicographic interpretation of the poetic language of V.A. Zhukovsky (1783—1852) in the form of the author’s dictionary, which became possible after the publication of the initial volumes of the “Complete Works and Letters” of the writer. Special attention is paid to poetic neologisms in the works of the first Russian romanticist. The question is raised about the typology of individual author’s lexemes. Occasionalisms properly, rare words and poetry, which became such as a result of the “replication” by many writers of successful and historically promising author’s neologisms of their predecessors are distinguished. The occasional units that are found in Zhukovsky’s translation of Homer’s poem “The Odyssey” and perform specific aesthetic functions are analyzed. The results of a comparative analysis of Zhukovsky’s poetic language with the lexicon of poets of the 18th — 20th centuries (N. M. Karamzin, I. I. Dmitriev, D. V. Davydov, K. N. Batyushkov, P. A. Vyazemsky, N. M. Yazykov , A. I. Polezhaev, D. V. Venevitinov, M. Yu. Lermontov and others) are presented in the article. The novelty of the research is seen in the fact that for the first time the neological layer of Zhukovsky’s poetic language is comprehended. The relevance of the study is due to the possibility of using the observations, facts and conclusions stated in it in the theory and practice of lexicography — when compiling explanatory, historical and author’s dictionaries.

Rare Feature Selection in High Dimensions

It is common in modern prediction problems for many predictor variables to be counts of rarely occurring events. This leads to design matrices in which many columns are highly sparse. The challenge posed by such “rare features” has received little attention despite its prevalence in diverse areas, ranging from natural language processing (e.g., rare words) to biology (e.g., rare species). We show, both theoretically and empirically, that not explicitly accounting for the rareness of features can greatly reduce the effectiveness of an analysis. We next propose a framework for aggregating rare features into denser features in a flexible manner that creates better predictors of the response. Our strategy leverages side information in the form of a tree that encodes feature similarity. We apply our method to data from TripAdvisor, in which we predict the numerical rating of a hotel based on the text of the associated review. Our method achieves high accuracy by making effective use of rare words; by contrast, the lasso is unable to identify highly predictive words if they are too rare. A companion R package, called rare, implements our new estimator, using the alternating direction method of multipliers. Supplementary materials for this article are available online.

Sacred, Profane, Troublesome, Adventurous

This article is concerned with a gigantic unpublished dictionary of Ancient Greek, most probably compiled at Alexandria during the first half of the sixth century ad. The dictionary is ascribed to Patriarch Cyril of Alexandria, an ascription strongly doubted. It is the first Greek dictionary which unites entries (usually rare ancient Greek words) found in Christian as well as pagan writers. The article investigates the ideology of the lexicon, which is strongly Christian, but also displays a warm acceptance of the classical literary past. The lexicon became the most influential in the history of Greek lexicography, having influenced almost all medieval Greek lexica (Hesychius, Synagoge, Photius, Suda, Zonaras and others). The article assembles all the information available today concerning the complicated history of scholarship on the lexicon, whose 200 preserved manuscripts and different surviving receptions have long puzzled scholars.

Jargon use in Public Understanding of Science papers over three decades.

Public Understanding of Science is an interdisciplinary journal serving the scholarly community and practitioners. This article reports an analysis of the readability and jargon in articles published in Public Understanding of Science throughout its almost three decades of existence to examine trends in accessibility to diverse audiences. The accessibility of Public Understanding of Science articles published in 1999/2000 (47), 2009 (49) and 2019 (65) was assessed in terms of readability and use of jargon. Readability decreased and use of jargon increased between 1999 and 2000 and the two following decades for empirical and non-empirical papers, and all parts including the abstracts. An analysis of rare words shows that most are not part of the general academic vocabulary or disciplinary jargon, but rather words that appeared only in one article. Public Understanding of Science has moved away from everyday language. This does not mean it is incomprehensible to its scholarly readership, but may have consequences to other audiences such as practitioners.

Morphological Segmentation to Improve Crosslingual Word Embeddings for Low Resource Languages

Crosslingual word embeddings developed from multiple parallel corpora help in understanding the relationships between languages and improving the prediction quality of machine translation. However, in low resource languages with complex and agglutinative morphologies, inducing good-quality crosslingual embeddings becomes challenging due to the problem of complex morphological forms and rare words. This is true even for languages that share common linguistic structure. In our work, we have shown that performing a simple morphological segmentation upon the corpora prior to the generation of crosslingual word embeddings for both roots and suffixes greatly improves the prediction quality and captures semantic similarities more effectively. To exhibit this, we have chosen two related languages: Telugu and Kannada of the Dravidian language family. We have also tested our method upon a widely spoken North Indian language, Hindi, belonging to the Indo-European language family, and have observed encouraging results.