Building Language Models Research Articles

On Using Self-Report Studies to Analyze Language Models

We are at a curious point in time where our ability to build language models (LMs) has outpaced our ability to analyze them. We do not really know how to reliably determine their capabilities, biases, dangers, knowledge, and so on. The benchmarks we have are often overly specific, do not generalize well, and are susceptible to data leakage. Recently, I have noticed a trend of using self-report studies, such as various polls and questionnaires originally designed for humans, to analyze the properties of LMs. I think that this approach can easily lead to false results, which can be quite dangerous considering the current discussions on AI safety, governance, and regulation. To illustrate my point, I will delve deeper into several papers that employ self-report methodologies and I will try to highlight some of their weaknesses.

Fast Exploring Literature by Language Machine Learning for Perovskite Solar Cell Materials Design

Making computers automatically extract latent scientific knowledge from literature is highly desired for future materials and chemical research in the artificial intelligence era. Herein, the natural language processing (NLP)‐based machine learning technique to build language models and automatically extract hidden information regarding perovskite solar cell (PSC) materials from 29 060 publications is employed. The concept that there are light‐absorbing materials, electron‐transporting materials, and hole‐transporting materials in PSCs is successfully learned by the NLP‐based machine learning model without a time‐consuming human expert training process. The NLP model highlights a hole‐transporting material that receives insufficient attention in the literature, which is then elaborated via density functional theory calculations to provide an atomistic view of the perovskite/hole‐transporting layer heterostructures and their optoelectronic properties. Finally, the above results are confirmed by device experiments. The present study demonstrates the viability of NLP as a universal machine learning tool to extract useful information from existing publications.

Ethical issues raised by incorporating personalized language models into brain-computer interface communication technologies: a qualitative study of individuals with neurological disease

Purpose To examine the views of individuals with neurodegenerative diseases about ethical issues related to incorporating personalized language models into brain-computer interface (BCI) communication technologies. Methods Fifteen semi-structured interviews and 51 online free response surveys were completed with individuals diagnosed with neurodegenerative disease that could lead to loss of speech and motor skills. Each participant responded to questions after six hypothetical ethics vignettes were presented that address the possibility of building language models with personal words and phrases in BCI communication technologies. Data were analyzed with consensus coding, using modified grounded theory. Results Four themes were identified. (1) The experience of a neurodegenerative disease shapes preferences for personalized language models. (2) An individual’s identity will be affected by the ability to personalize the language model. (3) The motivation for personalization is tied to how relationships can be helped or harmed. (4) Privacy is important to people who may need BCI communication technologies. Responses suggest that the inclusion of personal lexica raises ethical issues. Stakeholders want their values to be considered during development of BCI communication technologies. Conclusions With the rapid development of BCI communication technologies, it is critical to incorporate feedback from individuals regarding their ethical concerns about the storage and use of personalized language models. Stakeholder values and preferences about disability, privacy, identity and relationships should drive design, innovation and implementation. IMPLICATIONS FOR REHABILITATION Individuals with neurodegenerative diseases are important stakeholders to consider in development of natural language processing within brain-computer interface (BCI) communication technologies. The incorporation of personalized language models raises issues related to disability, identity, relationships, and privacy. People who may one day rely on BCI communication technologies care not just about usability of communication technology but about technology that supports their values and priorities. Qualitative ethics-focused research is a valuable tool for exploring stakeholder perspectives on new capabilities of BCI communication technologies, such as the storage and use of personalized language models.

Arabic Text Steganography Based on Deep Learning Methods

Steganography is one of the oldest methods for securely sending and transferring secret information between two people without raising suspicion. Recently, the use of Artificial Intelligence (AI) has become simpler and more widely used. Since the emergence of natural language processing (NLP), building language models using deep learning has become more. Furthermore, because of the importance of concealing secret information in delivered messages, Artificial Intelligence theories along with Natural Language Processing algorithms were employed to conceal secret information within the text cover. The Arabic language was used because of its large number of words, vocabulary, and linguistic meanings, and its most significant feature is Arabic poetry. This study discovered a new way to hide secret data inside newly formulated Arabic poetry based on previous Arabic poetic texts and a database of a number of Arab poets from the ancient and modern eras using Artificial Intelligence and Long Short-Term Memory (LSTM) theories to increase storage capacity by 45 percent. The linguistic accuracy and volume of secret data hidden within the formulated poetry were increased using a Baudot Code algorithm, where the secret data is hidden at the level of letters rather than words, and the linguistic accuracy and volume of secret data hidden within the formulated poetry were increased to eliminate the drawbacks found in previous studies.

Automated Source Code Generation and Auto-Completion Using Deep Learning: Comparing and Discussing Current Language Model-Related Approaches

In recent years, the use of deep learning in language models has gained much attention. Some research projects claim that they can generate text that can be interpreted as human writing, enabling new possibilities in many application areas. Among the different areas related to language processing, one of the most notable in applying this type of modeling is programming languages. For years, the machine learning community has been researching this software engineering area, pursuing goals like applying different approaches to auto-complete, generate, fix, or evaluate code programmed by humans. Considering the increasing popularity of the deep learning-enabled language models approach, we found a lack of empirical papers that compare different deep learning architectures to create and use language models based on programming code. This paper compares different neural network architectures like Average Stochastic Gradient Descent (ASGD) Weight-Dropped LSTMs (AWD-LSTMs), AWD-Quasi-Recurrent Neural Networks (QRNNs), and Transformer while using transfer learning and different forms of tokenization to see how they behave in building language models using a Python dataset for code generation and filling mask tasks. Considering the results, we discuss each approach’s different strengths and weaknesses and what gaps we found to evaluate the language models or to apply them in a real programming context.

Методи та компоненти обробки природної мови

A dramatic change in the abilities of language models to provide state of the artaccuracy in a number of Natural Language Processing tasks is currently observed. These improvements open a lot of possibilities in solving NLP downstream tasks. Such tasks include machine translation, speech recognition, information retrieval, sentiment analysis, summarization, question answering, multilingual dialogue systems development and many more. Language models are one of the most important components in solving each of the mentioned tasks. This paper is devoted to research and analysis of the most adopted techniques and designs for building and training language models that show a state of the art results. Techniques and components applied in creation of language models and its parts are observed in this paper, paying attention to neural networks, embedding mechanisms, bidirectionality, encoder anddecoder architecture, attention and self-attention, as well as parallelization through using Transformer. Results: the most promising techniques imply pre-training and fine-tuning of a language model, attention-based neural network as a part of model design, and a complex ensemble of multidimensional embeddings to build deep context understanding. The latest offered architectures based on these approaches require a lot of computational power for training language model and it is a direction of further improvement.Ref. 49, pic. 13

Crawl and crowd to bring machine translation to under-resourced languages

We present a widely applicable methodology to bring machine translation (MT) to under-resourced languages in a cost-effective and rapid manner. Our proposal relies on web crawling to automatically acquire parallel data to train statistical MT systems if any such data can be found for the language pair and domain of interest. If that is not the case, we resort to (1) crowdsourcing to translate small amounts of text (hundreds of sentences), which are then used to tune statistical MT models, and (2) web crawling of vast amounts of monolingual data (millions of sentences), which are then used to build language models for MT. We apply these to two respective use-cases for Croatian, an under-resourced language that has gained relevance since it recently attained official status in the European Union. The first use-case regards tourism, given the importance of this sector to Croatia’s economy, while the second has to do with tweets, due to the growing importance of social media. For tourism, we crawl parallel data from 20 web domains using two state-of-the-art crawlers and explore how to combine the crawled data with bigger amounts of general-domain data. Our domain-adapted system is evaluated on a set of three additional tourism web domains and it outperforms the baseline in terms of automatic metrics and/or vocabulary coverage. In the social media use-case, we deal with tweets from the 2014 edition of the soccer World Cup. We build domain-adapted systems by (1) translating small amounts of tweets to be used for tuning by means of crowdsourcing and (2) crawling vast amounts of monolingual tweets. These systems outperform the baseline (Microsoft Bing) by 7.94 BLEU points (5.11 TER) for Croatian-to-English and by 2.17 points (1.94 TER) for English-to-Croatian on a test set translated by means of crowdsourcing. A complementary manual analysis sheds further light on these results.

A study of n-gram and decision tree letter language modeling methods

The goal of this paper is to investigate various language model smoothing techniques and decision tree based language model design algorithms. For this purpose, we build language models for printable characters (letters), based on the Brown corpus. We consider two classes of models for the text generation process: the n-gram language model and various decision tree based language models. In the first part of the paper, we compare the most popular smoothing algorithms applied to the former. We conclude that the bottom-up deleted interpolation algorithm performs the best in the task of n-gram letter language model smoothing, significantly outperforming the back-off smoothing technique for large values of n. In the second part of the paper, we consider various decision tree development algorithms. Among them, a K-means clustering type algorithm for the design of the decision tree questions gives the best results. However, the n-gram language model outperforms the decision tree language models for letter language modeling. We believe that this is due to the predictive nature of letter strings, which seems to be naturally modeled by n-grams.