Language Model Perplexity Research Articles

Detecting AI-Generated Code Assignments Using Perplexity of Large Language Models

Large language models like ChatGPT can generate human-like code, posing challenges for programming education as students may be tempted to misuse them on assignments. However, there are currently no robust detectors designed specifically to identify AI-generated code. This is an issue that needs to be addressed to maintain academic integrity while allowing proper utilization of language models. Previous work has explored different approaches to detect AI-generated text, including watermarks, feature analysis, and fine-tuning language models. In this paper, we address the challenge of determining whether a student's code assignment was generated by a language model. First, our proposed method identifies AI-generated code by leveraging targeted masking perturbation paired with comperhesive scoring. Rather than applying a random mask, areas of the code with higher perplexity are more intensely masked. Second, we utilize a fine-tuned CodeBERT to fill in the masked portions, producing subtle modified samples. Then, we integrate the overall perplexity, variation of code line perplexity, and burstiness into a unified score. In this scoring scheme, a higher rank for the original code suggests it's more likely to be AI-generated. This approach stems from the observation that AI-generated codes typically have lower perplexity. Therefore, perturbations often exert minimal influence on them. Conversely, sections of human-composed codes that the model struggles to understand can see their perplexity reduced by such perturbations. Our method outperforms current open-source and commercial text detectors. Specifically, it improves detection of code submissions generated by OpenAI's text-davinci-003, raising average AUC from 0.56 (GPTZero baseline) to 0.87 for our detector.

ChatGPT-Generated Code Assignment Detection Using Perplexity of Large Language Models (Student Abstract)

In the era of large language models like Chatgpt, maintaining academic integrity in programming education has become challenging due to potential misuse. There's a pressing need for reliable detectors to identify Chatgpt-generated code. While previous studies have tackled model-generated text detection, identifying such code remains uncharted territory. In this paper, we introduce a novel method to discern Chatgpt-generated code. We employ targeted masking perturbation, emphasizing code sections with high perplexity. Fine-tuned CodeBERT is utilized to replace these masked sections, generating subtly perturbed samples. Our scoring system amalgamates overall perplexity, variations in code line perplexity, and burstiness. In this scoring scheme, a higher rank for the original code suggests it's more likely to be chatgpt-generated. The underlying principle is that code generated by models typically exhibits consistent, low perplexity and reduced burstiness, with its ranking remaining relatively stable even after subtle modifications. In contrast, human-written code, when perturbed, is more likely to produce samples that the model prefers. Our approach significantly outperforms current detectors, especially against OpenAI's text-davinci-003 model, with the average AUC rising from 0.56 (GPTZero baseline) to 0.87.

A curious case of retrogenesis in language: Automated analysis of language patterns observed in dementia patients and young children

IntroductionWhile linguistic retrogenesis has been extensively investigated in the neuroscientific and behavioral literature, there has been little work on retrogenesis using computerized approaches to language analysis. MethodsWe bridge this gap by introducing a method based on comparing output of a pre-trained neural language model (NLM) with an artificially degraded version of itself to examine the transcripts of speech produced by seniors with and without dementia and healthy children during spontaneous language tasks. We compare a range of linguistic characteristics including language model perplexity, syntactic complexity, lexical frequency and part-of-speech use across these groups. ResultsOur results indicate that healthy seniors and children older than 8 years share similar linguistic characteristics, as do dementia patients and children who are younger than 8 years. DiscussionOur study aligns with the growing evidence that language deterioration in dementia mirrors language acquisition in development using computational linguistic methods based on NLMs. This insight underscores the importance of further research to refine its application in guiding developmentally appropriate patient care, particularly in early stages.

Transferable adversarial distribution learning: Query-efficient adversarial attack against large language models

It is a challenging task to fool a text classifier based on deep neural networks under the black-box setting where the target model can only be queried. Among the existing black-box attacks, decision-based methods have a large query cost due to exponential perturbation space and greedy search strategy. Transfer-based methods, on the other hand, tend to overfit the surrogate model and thus fail when applied to unknown target models. In this paper, we propose a straightforward yet highly effective adversarial attack framework for black-box transformer-based models, thereby exposing vulnerabilities within large language models. Specifically, we leverage a fine-tuned large language model as a white-box surrogate model and optimize a distribution of adversarial text. This distribution is parameterized by a continuous-valued matrix based upon the surrogate model. To avoid overfitting of the distribution and improve its adversarial transferability, we incorporate an additional causal language model into our framework as a constraint model. Based on this constraint model, we add language model perplexity and semantic consistency as regularization terms during the distribution training process. To further reduce the number of queries to the target model, i.e., improve the threat level of examples drawn from our distribution, we employ a geometric loss strategy to ensure that the distribution training process learns the optimal perturbation. Extensive experimental studies have been carried out on benchmark datasets and the results demonstrate significant improvement on the performance and query efficiency under black-box setting in comparison with well-established approaches. Our approach achieves an 80.98% reduction in BERT model accuracy while consuming only 21.86% of the query times required by prior attacks.

Aggregating pairwise semantic differences for few-shot claim verification.

As part of an automated fact-checking pipeline, the claim verification task consists in determining if a claim is supported by an associated piece of evidence. The complexity of gathering labelled claim-evidence pairs leads to a scarcity of datasets, particularly when dealing with new domains. In this article, we introduce Semantic Embedding Element-wise Difference (SEED), a novel vector-based method to few-shot claim verification that aggregates pairwise semantic differences for claim-evidence pairs. We build on the hypothesis that we can simulate class representative vectors that capture average semantic differences for claim-evidence pairs in a class, which can then be used for classification of new instances. We compare the performance of our method with competitive baselines including fine-tuned Bidirectional Encoder Representations from Transformers (BERT)/Robustly Optimized BERT Pre-training Approach (RoBERTa) models, as well as the state-of-the-art few-shot claim verification method that leverages language model perplexity. Experiments conducted on the Fact Extraction and VERification (FEVER) and SCIFACT datasets show consistent improvements over competitive baselines in few-shot settings. Our code is available.

A topic-driven language model for learning to generate diverse sentences

Generating diverse sentences under a topic is a meaningful, yet not well-solved task in the field of natural language processing. We present a neural language model for generating diverse sentences conditioned on a given topic distribution. From the perspective of diversity, the proposed model takes the advantages of variational autoencoders with convolutional neural network and long short-term memory architecture. The proposed model is trained end-to-end to learn topic-level Gaussian distributions in the latent space. Then our model decodes the samples of topics obtained from latent space to generate each sentence. Results on Restaurant Dataset and Yahoo! Answers Dataset show that our model outperforms other methods in terms of language model perplexity. Also, our approach can generate a large set of different coherent sentences related to given topics. The diversity of our sentences provides a novel interpretation of topics.

Evaluation of Language Models over Croatian Newspaper Texts

Statistical language modeling involves techniques and procedures that assign probabilities to word sequences or, said in other words, estimate the regularity of the language. This paper presents basic characteristics of statistical language models, reviews their use in the large set of speech and language applications, explains their formal definition and shows different types of language models. Detailed overview of n-gram and class-based models (as well as their combinations) is given chronologically, by type and complexity of models, and in aspect of their use in different NLP applications for different natural languages. The proposed experimental procedure compares three different types of statistical language models: n-gram models based on words, categorical models based on automatically determined categories and categorical models based on POS tags. In the paper, we propose a language model for contemporary Croatian texts, a procedure how to determine the best n-gram and the optimal number of categories, which leads to significant decrease of language model perplexity, estimated from the Croatian News Agency articles (HINA) corpus. Using different language models estimated from the HINA corpus, we show experimentally that models based on categories contribute to a better description of the natural language than those based on words. These findings of the proposed experiment are applicable, except for Croatian, for similar highly inflectional languages with rich morphology and non-mandatory sentence word order. DOI: http://dx.doi.org/10.5755/j01.itc.46.4.18367

Speech-Based Human and Service Robot Interaction: An Application for Mexican Dysarthric People

Dysarthria is a motor speech disorder due to weakness or poor coordination of the speech muscles. This condition can be caused by a stroke, traumatic brain injury, or by a degenerative neurological disease. Commonly, people with this disorder also have muscular dystrophy, which restricts their use of switches or keyboards for communication or control of assistive devices (i.e., an electric wheelchair or a service robot). In this case, speech recognition is an attractive alternative for interaction and control of service robots, despite the difficulty of achieving robust recognition performance. In this paper we present a speech recognition system for human and service robot interaction for Mexican Spanish dysarthric speakers. The core of the system consisted of a Speaker Adaptive (SA) recognition system trained with normal-speech. Features such as on-line control of the language model perplexity and the adding of vocabulary, contribute to high recognition performance. Others, such as assessment and text-to-speech (TTS) synthesis, contribute to a more complete interaction with a service robot. Live tests were performed with two mild dysarthric speakers, achieving recognition accuracies of 90–95% for spontaneous speech and 95–100% of accomplished simulated service robot tasks.

Analysis of Morph-Based Language Modeling and Speech Recognition in Slovak

The inflection of the Slovak language causes a large number of unique word forms, which produces not only a large vocabulary, but also a number of out-of-vocabulary words. Morph-based language models solve this problem by decomposition of inflected word forms into small sub-word units and resolve the general problem of sparsity the training data. In this paper, we present several rule-based and data-driven approaches to the automatic segmentation of words into morphs. These data are later used in the modeling of the Slovak language for large vocabulary continuous speech recognition. Preliminary results show a significant decrease in the number of out-of-vocabulary words and reduction of resultant language model perplexity.

Minimum discrimination information-based language model adaptation using tiny domain corpora for intelligent personal assistants

This paper proposes a novel Language Model (LM) adaptation method based on Minimum Discrimination Information (MDI). In the proposed method, a background LM is viewed as a discrete distribution and an adapted LM is built to be as close as possible to the background LM, while satisfying unigram constraint. This is due to the fact that there is a limited amount of domain corpus available for the adaptation of a natural language-based intelligent personal assistant system. Two unigram constraint estimation methods are proposed: one based on word frequency in the domain corpus, and one based on word similarity estimated from WordNet. In terms of the adapted LM's perplexity using word frequency in tiny domain corpora (ranging from 30~120 seconds in length) the relative performance improvements are measured at 13.9%~16.6%. Further relative performance improvements (1.5%~2.4%) are observed when WordNet is used to generate word similarities. These successes express an efficient ways for re-scaling and normalizing the conditional distribution, which uses an interpolation-based LM.

Communication Interface for Mexican Spanish Dysarthric Speakers

Dysarthria is a motor speech disorder due to weakness or poor coordination of the speechmuscles. This condition can be caused by a stroke, cerebral palsy, or by a traumatic braininjury. For Mexican people with this condition there are few, if any, assistive technologies to improve their social interaction skills. In this paper we present our advances towards the development of a communication interface for dysarthric speakers whose native language is Mexican Spanish. We propose a methodology that relies on (1) special design of a training normal-speech corpus with limited resources, (2) standard speaker adaptation, and (3) control of language model perplexity, to achieve high Automatic Speech Recognition (ASR) accuracy. The interface allows the user and therapist to perform tasks such as dynamic speaker adaptation, vocabulary adaptation, and text-to-speech (TTS) synthesis. Live tests were performed with a mildly dysarthric speaker, achieving accuracies of 93%-95% for spontaneous speech.

An Iterative Relative Entropy Minimization-Based Data Selection Approach for n-Gram Model Adaptation

Performance of statistical n-gram language models depends heavily on the amount of training text material and the degree to which the training text matches the domain of interest. The language modeling community is showing a growing interest in using large collections of text (obtainable, for example, from a diverse set of resources on the Internet) to supplement sparse in-domain resources. However, in most cases the style and content of the text harvested from the web differs significantly from the specific nature of these domains. In this paper, we present a relative entropy based method to select subsets of sentences whose n-gram distribution matches the domain of interest. We present results on language model adaptation using two speech recognition tasks: a medium vocabulary medical domain doctor-patient dialog system and a large vocabulary transcription system for European parliamentary plenary speeches (EPPS). We show that the proposed subset selection scheme leads to performance improvements over state of the art speech recognition systems in terms of both speech recognition word error rate (WER) and language model perplexity (PPL).

Extended probabilistic HAL with close temporal association for psychiatric query document retrieval

Psychiatric query document retrieval can assist individuals to locate query documents relevant to their depression-related problems efficiently and effectively. By referring to relevant documents, individuals can understand how to alleviate their depression-related symptoms according to recommendations from health professionals. This work presents an extended probabilistic Hyperspace Analog to Language ( epHAL ) model to achieve this aim. The epHAL incorporates the close temporal associations between words in query documents to represent word cooccurrence relationships in a high-dimensional context space. The information flow mechanism further combines the query words in the epHAL space to infer related words for effective information retrieval. The language model perplexity is considered as the criterion for model optimization. Finally, the epHAL is adopted for psychiatric query document retrieval, and indicates its superiority in information retrieval over traditional approaches.

Incorporating linguistic structure into maximum entropy language models

In statistical language models, how to integrate diverse linguistic knowledge in a general framework for long-distance dependencies is a challenging issue. In this paper, an improved language model incorporating linguistic structure into maximum entropy framework is presented. The proposed model combines trigram with the structure knowledge of base phrase in which trigram is used to capture the local relation between words, while the structure knowledge of base phrase is considered to represent the long-distance relations between syntactical structures. The knowledge of syntax, semantics and vocabulary is integrated into the maximum entropy framework. Experimental results show that the proposed model improves by 24% for language model perplexity and increases about 3% for sign language recognition rate compared with the trigram model.

Testing the correlation of word error rate and perplexity

Many groups have investigated the relationship of word error rate and perplexity of language models. This issue is of central interest because perplexity optimization can be done independent of a recognizer and in most cases it is possible to find simple perplexity optimization procedures. Moreover, many tasks in language model training such as the optimization of word classes may use perplexity as target function resulting in explicit optimization formulas which are not available if error rates are used as target. This paper first presents some theoretical arguments for a close relationship between perplexity and word error rate. Thereafter the notion of uncertainty of a measurement is introduced and is then used to test the hypothesis that word error rate and perplexity are correlated by a power law. There is no evidence to reject this hypothesis.

Variable-length categoryn-gram language models

This paper presents a language model based onn-grams of word groups (categories). The length of eachn-gram is increased selectively according to an estimate of the resulting improvement in predictive quality. This allows the model size to be controlled while including longer-range dependencies when these benefit performance. The categories are chosen to correspond to part-of-speech classifications in a bid to exploita priorigrammatical information. To account for different grammatical functions, the language model allows words to belong to multiple categories, and implicitly involves a statistical tagging operation which may be used to label new text. Intrinsic generalization by the category-based model leads to good performance with sparse data sets. However word-basedn-grams deliver superior average performance as the amount of training material increases. Nevertheless, the category model continues to supply better predictions for wordn-tuples not present in the training set. Consequently, a method allowing the two approaches to be combined within a backoff framework is presented. Experiments with the LOB, Switchboard and Wall Street Journal corpora demonstrate that this technique greatly improves language model perplexities for sparse training sets, and offers significantly improved size vs. performance tradeoffs when compared with standard trigram models.

Probabilistic word classification based on a context-sensitive binary tree method

A corpus-based statistical-oriented Chinese word classification can be regarded as a fundamental step for automatic or non-automatic, monolingual natural processing systems. Word classification can solve the problems of data sparseness and have far fewer parameters. So far, much relative work about word classification has been done. All the work is based on some similarity metrics. We use average mutual information as the global similarity metric to do classification. The clustering process is top-down splitting and the binary tree is growing with splitting. In natural language, the effect of left neighbours and right neighbours of a word are asymmetric. To utilize this directional information, we induce the left–right binary and the right–left binary tree to represent this property. The probability is also introduced in our algorithm to merge the resulting classes from the left–right and the right–left binary tree. Also, we use the resulting classes to do experiments on a word class-based language model. Some classes' results and the perplexity of a word class-based language model are presented.

Technical achievements of the DARPA speech recognition program

During the first 4 years, several significant milestones have been reached by participants of the DARPA strategic computing speech program in the development of recognition systems, related component technology, and common speech databases and research tools. From the systems' standpoint, speaker‐dependent, ‐adaptive, and ‐independent systems that operate in several times real time with word error rates that range between 5% and 10% have been demonstrated with a 1000‐word vocabulary and a language model perplexity of 60. Real‐time performance of isolated‐word, speaker‐dependent, 100‐word recognition systems that operate under noisy and stressful environments was demonstrated with word accuracy of better than 98%. Meanwhile, advances have been made in component technology including auditory modeling, acoustic phonetic recognition, lexical representation, language modeling, and the implementation of algorithms in general‐ and special‐purpose hardware. The component technology is intended to provide robust performance in real time, and to support future expansion in task complexity. The program also resulted in the development of several large databases, and a set of common research tools. The use of these databases in conjunction with the development and implementation of standardized performance test procedures has permitted consistent documentation of progress. This talk will describe these technical areas in greater detail. [Work supported by DARPA‐ISTO.]