Model Perplexity Research Articles

Easy and effective! Data augmentation for knowledge-aware dialogue generation via multi-perspective sentences interaction

In recent years, knowledge-based dialogue generation has garnered significant attention due to its capacity to produce informative and coherent responses through the integration of external knowledge into models. However, obtaining high-quality knowledge that aligns with the dialogue content poses a considerable challenge, necessitating substantial time and resources. To tackle the issue of limited dialogue data, a majority of research endeavors concentrate on data augmentation to augment the volume of training data. Regrettably, these methods overlook knowledge augmentation, leading to a restricted diversity in input data and yielding enhancements solely in specific metrics. Real-world conversations exhibit a spectrum of characteristics, including repetitions, reversals, and interruptions, demanding a heightened level of data diversity. In this study, we introduce a straightforward yet effective data augmentation technique known as Multi-perspective Sentence Interaction to bolster the connections among sentences from varied viewpoints. Through an examination of target responses from multiple dialogue perspectives, we enhance our comprehension of the relationships between dialogue sentences, thereby facilitating the expansion of knowledge-based dialogue data. Through experiments conducted on various knowledge-based dialogue datasets and utilizing different models, our findings illustrate a notable enhancement in the quality of model generation facilitated by our method. Specifically, we observed a 3.5% enhancement in reply accuracy and a 0.1506 increase in diversity (DIST-2). Moreover, there was a substantial improvement in knowledge selection accuracy by 19.04% and a reduction in model perplexity by 31.48%.

Detecting AI-Generated Code Assignments Using Perplexity of Large Language Models

Large language models like ChatGPT can generate human-like code, posing challenges for programming education as students may be tempted to misuse them on assignments. However, there are currently no robust detectors designed specifically to identify AI-generated code. This is an issue that needs to be addressed to maintain academic integrity while allowing proper utilization of language models. Previous work has explored different approaches to detect AI-generated text, including watermarks, feature analysis, and fine-tuning language models. In this paper, we address the challenge of determining whether a student's code assignment was generated by a language model. First, our proposed method identifies AI-generated code by leveraging targeted masking perturbation paired with comperhesive scoring. Rather than applying a random mask, areas of the code with higher perplexity are more intensely masked. Second, we utilize a fine-tuned CodeBERT to fill in the masked portions, producing subtle modified samples. Then, we integrate the overall perplexity, variation of code line perplexity, and burstiness into a unified score. In this scoring scheme, a higher rank for the original code suggests it's more likely to be AI-generated. This approach stems from the observation that AI-generated codes typically have lower perplexity. Therefore, perturbations often exert minimal influence on them. Conversely, sections of human-composed codes that the model struggles to understand can see their perplexity reduced by such perturbations. Our method outperforms current open-source and commercial text detectors. Specifically, it improves detection of code submissions generated by OpenAI's text-davinci-003, raising average AUC from 0.56 (GPTZero baseline) to 0.87 for our detector.

ChatGPT-Generated Code Assignment Detection Using Perplexity of Large Language Models (Student Abstract)

In the era of large language models like Chatgpt, maintaining academic integrity in programming education has become challenging due to potential misuse. There's a pressing need for reliable detectors to identify Chatgpt-generated code. While previous studies have tackled model-generated text detection, identifying such code remains uncharted territory. In this paper, we introduce a novel method to discern Chatgpt-generated code. We employ targeted masking perturbation, emphasizing code sections with high perplexity. Fine-tuned CodeBERT is utilized to replace these masked sections, generating subtly perturbed samples. Our scoring system amalgamates overall perplexity, variations in code line perplexity, and burstiness. In this scoring scheme, a higher rank for the original code suggests it's more likely to be chatgpt-generated. The underlying principle is that code generated by models typically exhibits consistent, low perplexity and reduced burstiness, with its ranking remaining relatively stable even after subtle modifications. In contrast, human-written code, when perturbed, is more likely to produce samples that the model prefers. Our approach significantly outperforms current detectors, especially against OpenAI's text-davinci-003 model, with the average AUC rising from 0.56 (GPTZero baseline) to 0.87.

A search tool based on language modelling developed for The Index of Middle English Prose

Non-standardised early vernaculars present a problem for search tools due to the high degree of variation. The challenge lies in the variation found in orthography, syntax, and lexicon between titles, incipits, and explicits in manuscript copies of the same work. Traditional search methods relying on exact string matching or regular expressions fail to address these variations comprehensively. This project presents a web-based search tool specifically designed to handle linguistic and textual variation. The software is made available as a part of the Index of Middle English Prose (IMEP). The search tool addresses the issue of variation by utilizing a database of incipits and explicits, character-based n-gram language models (LMs) built with the Stanford Research Institute Language Modelling (SRILM) toolkit, and a fuzzy search script (IMEP: FSS) written in Python. The tool optimizes for recall, retrieving multiple potential matches for a search string, without attempting to identify the ‘correct’ one. The search process involves looking up exact matches in the database while simultaneously using the fuzzy search script to evaluate the incipits and explicits against a model of the search string, followed by a match of the search string against models of the incipits and explicits. This two-step process shortens the processing time, which would otherwise be unreasonably long, because while using SRILM to match the search string against each incipit or explicit in the IMEP for precision could be time-consuming, running a first step where all texts are matched against a single LM built from the search string allows for faster processing. A web application, built using Django and Docker, combines the results of the direct database lookup and the fuzzy search script, presenting them as a list with exact matches followed by fuzzy matches ordered by increasing model perplexity. The tool is made available Open Access and can be adapted to other datasets.

A curious case of retrogenesis in language: Automated analysis of language patterns observed in dementia patients and young children

IntroductionWhile linguistic retrogenesis has been extensively investigated in the neuroscientific and behavioral literature, there has been little work on retrogenesis using computerized approaches to language analysis. MethodsWe bridge this gap by introducing a method based on comparing output of a pre-trained neural language model (NLM) with an artificially degraded version of itself to examine the transcripts of speech produced by seniors with and without dementia and healthy children during spontaneous language tasks. We compare a range of linguistic characteristics including language model perplexity, syntactic complexity, lexical frequency and part-of-speech use across these groups. ResultsOur results indicate that healthy seniors and children older than 8 years share similar linguistic characteristics, as do dementia patients and children who are younger than 8 years. DiscussionOur study aligns with the growing evidence that language deterioration in dementia mirrors language acquisition in development using computational linguistic methods based on NLMs. This insight underscores the importance of further research to refine its application in guiding developmentally appropriate patient care, particularly in early stages.

A search tool based on language modelling developed for The Index of Middle English Prose.

Non-standardised early vernaculars present a problem for search tools due to the high degree of variation. The challenge lies in the variation found in orthography, syntax, and lexicon between titles, incipits, and explicits in manuscript copies of the same work. Traditional search methods relying on exact string matching or regular expressions fail to address these variations comprehensively. This project presents a web-based search tool specifically designed to handle linguistic and textual variation. The software is made available as a part of the Index of Middle English Prose (IMEP). The search tool addresses the issue of variation by utilizing a database of incipits and explicits, character-based n-gram language models (LMs) built with the Stanford Research Institute Language Modelling (SRILM) toolkit, and a fuzzy search script (IMEP: FSS) written in Python. The tool optimizes for recall, retrieving multiple potential matches for a search string, without attempting to identify the 'correct' one. The search process involves looking up exact matches in the database while simultaneously using the fuzzy search script to evaluate the incipits and explicits against a model of the search string, followed by a match of the search string against models of the incipits and explicits. This two-step process shortens the processing time, which would otherwise be unreasonably long, because while using SRILM to match the search string against each incipit or explicit in the IMEP for precision could be time-consuming, running a first step where all texts are matched against a single LM built from the search string allows for faster processing. A web application, built using Django and Docker, combines the results of the direct database lookup and the fuzzy search script, presenting them as a list with exact matches followed by fuzzy matches ordered by increasing model perplexity. The tool is made available Open Access and can be adapted to other datasets.

Transferable adversarial distribution learning: Query-efficient adversarial attack against large language models

It is a challenging task to fool a text classifier based on deep neural networks under the black-box setting where the target model can only be queried. Among the existing black-box attacks, decision-based methods have a large query cost due to exponential perturbation space and greedy search strategy. Transfer-based methods, on the other hand, tend to overfit the surrogate model and thus fail when applied to unknown target models. In this paper, we propose a straightforward yet highly effective adversarial attack framework for black-box transformer-based models, thereby exposing vulnerabilities within large language models. Specifically, we leverage a fine-tuned large language model as a white-box surrogate model and optimize a distribution of adversarial text. This distribution is parameterized by a continuous-valued matrix based upon the surrogate model. To avoid overfitting of the distribution and improve its adversarial transferability, we incorporate an additional causal language model into our framework as a constraint model. Based on this constraint model, we add language model perplexity and semantic consistency as regularization terms during the distribution training process. To further reduce the number of queries to the target model, i.e., improve the threat level of examples drawn from our distribution, we employ a geometric loss strategy to ensure that the distribution training process learns the optimal perturbation. Extensive experimental studies have been carried out on benchmark datasets and the results demonstrate significant improvement on the performance and query efficiency under black-box setting in comparison with well-established approaches. Our approach achieves an 80.98% reduction in BERT model accuracy while consuming only 21.86% of the query times required by prior attacks.

Identifying Anomalous Indus Texts from West Asia Using Markov Chain Language Model

The Indus Valley civilization thrived during its mature period between 2500 BCE and 1800 BCE and traded with civilizations of West Asia such as Mesopotamia and Dilmun through the Persian gulf. During this period, the Indus civilization developed a writing system now called the Indus script which is logosyllabic. It is unclear what language it encodes and is considered one of the biggest unsolved mysteries of historical linguistics. Researchers have collected Indus script texts from various sites in the Indian subcontinent and from West Asia and have built the Indus script text corpus. This research shows that several West Asian Indus texts in the corpus likely use a different language or syntax than the ones from the Indian subcontinent. We built various Markov chain language models based on n-grams from the Indus texts corpus. Using a best-fit language model, we calculated the model perplexity for each of the West Asian Indus texts. Our results show that the model perplexity was high for most West Asian Indus texts and that these texts did not fit in well with the language model built with Indus texts from just the Indian subcontinent. We conclude that the language and/or the syntax in the West Asian Indus texts are different from the Indus texts from the Indian subcontinent. We hope that this research and the statistical models we developed here will aid in quantifying the geographical difference in Indus scripts and contribute to the Indus script decipherment effort.

Aggregating pairwise semantic differences for few-shot claim verification.

As part of an automated fact-checking pipeline, the claim verification task consists in determining if a claim is supported by an associated piece of evidence. The complexity of gathering labelled claim-evidence pairs leads to a scarcity of datasets, particularly when dealing with new domains. In this article, we introduce Semantic Embedding Element-wise Difference (SEED), a novel vector-based method to few-shot claim verification that aggregates pairwise semantic differences for claim-evidence pairs. We build on the hypothesis that we can simulate class representative vectors that capture average semantic differences for claim-evidence pairs in a class, which can then be used for classification of new instances. We compare the performance of our method with competitive baselines including fine-tuned Bidirectional Encoder Representations from Transformers (BERT)/Robustly Optimized BERT Pre-training Approach (RoBERTa) models, as well as the state-of-the-art few-shot claim verification method that leverages language model perplexity. Experiments conducted on the Fact Extraction and VERification (FEVER) and SCIFACT datasets show consistent improvements over competitive baselines in few-shot settings. Our code is available.

Neural Topic Model Training with the REBAR Gradient Estimator

Topic modelling is an important approach of unsupervised machine learning that allows automatically extracting the main “topics” from large collections of documents. In addition, topic modelling is able to identify the topic proportions of each individual document, which can be helpful for organizing the collections. Many topic modelling algorithms have been proposed to date, including several that leverage advanced techniques such as variational inference and deep autoencoders. However, to date topic modelling has made limited use of reinforcement learning, a framework that has obtained vast success in many other unsupervised learning tasks. For this reason, in this article we propose training a neural topic model using a reinforcement learning objective and minimizing the objective with the recently-proposed REBAR gradient estimator. Experiments performed over two probing datasets have shown that the proposed model has achieved improvements over all the compared models in terms of both model perplexity and topic coherence, and produced topics that appear qualitatively informative and consistent.

A Deep Neural Network-Based Yoruba Intelligent Chatbot System

Two Artificial Intelligence software systems, Bot and Chatbot have recently debuted on the internet. This initiate a communication between the user and a virtual agent. The modeling and performance in deep learning (DL) computation for an Assistant Conversational Agent are presented in this research (Chatbot). The deep neural network (DNN) technique is used to respond to a large number of tokens in an input sentence with more appropriate dialogue. The model was created to do Yoruba-to-Yoruba translations. The major goal of this project is to improve the model's perplexity and learning rate, as well as to find a blue score for translation in the same language. Kares is used to run the experiments, which is written in Python. The data was trained using a deep learning-based algorithm. With the use of training examples, a collection of Yoruba phrases with various intentions was produced. The results demonstrate that the system can communicate in basic Yoruba terms and that it would be able to learn simple Yoruba words. The study result when evaluated showed that the system had 80% accuracy rate. Keywords: Chatbot, Natural Language Processing, Deep Learning, Artificial Neural Network, Yoruba Language

Bangla text generation system by incorporating attention in sequence-to-sequence model

In this AI-driven digital era, the pervasive nature of digital data is possible due to the widespread and cheap access to the Internet. Internet is continuously flourishing with data in many forms. Among them, textual data are a great source of information where people share their expressions in written format. Social media, blogs, online newspapers, government documents are some notable textual data sources. Information extraction from this enormous amount of data by manual inspection is time-consuming, cumbersome, and sometimes impossible. Natural Language Processing (NLP) is the computational domain for addressing those limitations by solving human language-related problems. Text summarization, Named entity recognition, Question answering are some of them where a common task for a machine is to generate coherent text. In such scenarios, the input is a sequence of text, and the output is also a sequence, but they differ in length. Sequence-to-Sequence (Seq2Seq) is an algorithmic approach to address that scenario by utilizing layers of recurrent units. However, the simple Seq2Seq model fails to capture the long-term relationship on the input sequence. Research shows that the attention mechanism guides the model to concentrate on specific inputs. Existing literature shows a lack of quality research on this text generation problem in the Bangla language, whereas many languages show excellent results. This work aims to develop such a system by incorporating attention to the Seq2Seq model and justifying its applicability by comparing it with baseline models. The model perplexity shows that the system can generate human-level readable text using a preprocessed dataset.

Placental Social Ethics: Designing for Epistemologies of Resistance

Placental Social Ethics: Designing for Epistemologies of Resistance

A time-series based aggregation scheme for topic detection in Weibo short texts

Discovering hot topics within social network like Twitter and Weibo, has received much attention in recent years. While topic models such as Latent Dirichlet Allocation (LDA) have been successfully applied in topic discovery, they are often less coherent when applied to microblog content which is known as “posts”. In this paper, we propose a time-series based aggregation scheme for topic modeling in Weibo. As Weibo topics are coherent within a time slice, we divide Weibo dataset into groups by time slice. With this scheme, posts in every group are aggregated into several longer pseudo-documents using paragraph-vector based similarity algorithms. While applying this scheme to LDA model, we dramatically decrease the topic model perplexity and increase the clustering quality, which also allows for better discovery of underlying topics in Weibo. Furthermore, we can let other topic models extended on LDA be directly used on such short texts.

A topic-driven language model for learning to generate diverse sentences

Generating diverse sentences under a topic is a meaningful, yet not well-solved task in the field of natural language processing. We present a neural language model for generating diverse sentences conditioned on a given topic distribution. From the perspective of diversity, the proposed model takes the advantages of variational autoencoders with convolutional neural network and long short-term memory architecture. The proposed model is trained end-to-end to learn topic-level Gaussian distributions in the latent space. Then our model decodes the samples of topics obtained from latent space to generate each sentence. Results on Restaurant Dataset and Yahoo! Answers Dataset show that our model outperforms other methods in terms of language model perplexity. Also, our approach can generate a large set of different coherent sentences related to given topics. The diversity of our sentences provides a novel interpretation of topics.

Controlling Global Statistics in Recurrent Neural Network Text Generation

Recurrent neural network language models (RNNLMs) are an essential component for many language generation tasks such as machine translation, summarization, and automated conversation. Often, we would like to subject the text generated by the RNNLM to constraints, in order to overcome systemic errors (e.g. word repetition) or achieve application-specific goals (e.g. more positive sentiment). In this paper, we present a method for training RNNLMs to simultaneously optimize likelihood and follow a given set of statistical constraints on text generation. The problem is challenging because the statistical constraints are defined over aggregate model behavior, rather than model parameters, meaning that a straightforward parameter regularization approach is insufficient. We solve this problem using a dynamic regularizer that updates as training proceeds, based on the generative behavior of the RNNLMs. Our experiments show that the dynamic regularizer outperforms both generic training and a static regularization baseline. The approach is successful at improving word-level repetition statistics by a factor of four in RNNLMs on a definition modeling task. It also improves model perplexity when the statistical constraints are $n$-gram statistics taken from a large corpus.

Microblog Topic Mining Based on FR‐DATM

Microblog has become a major platform for people to release or obtain information. Texts on Microblog are shorter and have scarce co-occurrence information of terms. It is more complicated to discover topics from Microblog. To solve the problems, this paper proposes a dynamic author topic model FR-DATM and uses Gibbs sampling implementation for inference of this model. The FR-DATM model analyzes the relationships between blogs, and connects the related blogs to solve the sparseness of data. It allows blogs to be related to multiple topics, and each author of the blogs is also related to the topics of the blogs. The FR-DATM can also mine the topic evolution of the blogs and the authors. Experiments on Twitter dataset show that FR-DATM outperforms Latent dirichlet allocation (LDA) model and Microblog latent Dirichlet Allocation (MB-LDA) from three different perspectives: The quality of generated latent topics, the model perplexity and FR-DATM can mine the topic that the author are concerned dynamically.

Evaluation of Language Models over Croatian Newspaper Texts

Statistical language modeling involves techniques and procedures that assign probabilities to word sequences or, said in other words, estimate the regularity of the language. This paper presents basic characteristics of statistical language models, reviews their use in the large set of speech and language applications, explains their formal definition and shows different types of language models. Detailed overview of n-gram and class-based models (as well as their combinations) is given chronologically, by type and complexity of models, and in aspect of their use in different NLP applications for different natural languages. The proposed experimental procedure compares three different types of statistical language models: n-gram models based on words, categorical models based on automatically determined categories and categorical models based on POS tags. In the paper, we propose a language model for contemporary Croatian texts, a procedure how to determine the best n-gram and the optimal number of categories, which leads to significant decrease of language model perplexity, estimated from the Croatian News Agency articles (HINA) corpus. Using different language models estimated from the HINA corpus, we show experimentally that models based on categories contribute to a better description of the natural language than those based on words. These findings of the proposed experiment are applicable, except for Croatian, for similar highly inflectional languages with rich morphology and non-mandatory sentence word order. DOI: http://dx.doi.org/10.5755/j01.itc.46.4.18367

From language identification to language distance

In this paper, we define two quantitative distances to measure how far apart two languages are. The distance measure that we have identified as more accurate is based on the perplexity of n-gram models extracted from text corpora. An experiment to compare forty-four European languages has been performed. For this purpose, we computed the distances for all the possible language pairs and built a network whose nodes are languages and edges are distances. The network we have built on the basis of linguistic distances represents the current map of similarities and divergences among the main languages of Europe.

Personalizing Recurrent-Neural-Network-Based Language Model by Social Network

With the popularity of mobile devices, personalized speech recognizers have become more attainable and are highly attractive. Since each mobile device is used primarily by a single user, it is possible to have a personalized recognizer that well matches the characteristics of the individual user. Although acoustic model personalization has been investigated for decades, much less work has been reported on personalizing language models, presumably because of the difficulties in collecting sufficient personalized corpora. In this paper, we propose a general framework for personalizing recurrent-neural-network-based language models (RNNLMs) using data collected from social networks, including the posts of many individual users and friend relationships among the users. Two major directions for this are model-based and feature-based RNNLM personalization. In model-based RNNLM personalization, the RNNLM parameters are fine-tuned to an individual user's wording patterns by incorporating social texts posted by the target user and his or her friends. For the feature-based approach, the RNNLM model parameters are fixed across users, but the RNNLM input features are instead augmented with personalized information. Both approaches not only drastically reduce the model perplexity, but also moderately reduce word error rates in n-best rescoring tests.