GPT-2 Model Research Articles

A high precision 3-D tropospheric delay model over China using ERA5 data

The zenith tropospheric delay (ZTD) is an important error in Global Navigation Satellite System (GNSS) navigation and positioning. The current most empirical tropospheric delay models are two-dimensional with low spatial and temporal resolution and cannot capture high-frequent accurate ZTD variations in small areas and short periods, particularly in Mainland China with diverse climate changes and large terrain differences. In this paper, a high precision three-dimensional (3-D) tropospheric delay grid model (ZTD_3D) over China with a horizontal resolution of 0.25° × 0.25° and a time resolution of 1 hour was established with the piecewise function describing the height change of ZTD from 2016-2019 fifth-generation European Centre for Medium-Range Weather Forecasts (ECMWF) reanalysis data (ERA5). The performance of the proposed model is verified by using radiosonde data and ZTD products from the Crustal Movement Observation Network of China (CMONOC), as well as the global pressure and temperature models (GPT2, GPT3-1 and GPT3-5). Results show that the RMS of ZTD_3D model is 2.04 cm when compared to radiosonde observations, and the accuracy is 55.1 %, 54.7 % and 63.1 % higher than that of GPT3-1 model, GPT3-5 model and GPT2 model, respectively, while the RMS of ZTD_3D model is 3.66 cm when compared to GPS ZTD from CMONOC, and the accuracy is 1.6 % and 29.7 % higher than that of GPT3-5 mode and GPT2 model, respectively. The application of the ZTD_3D model in GNSS precise point positioning (PPP) has showed accuracy improvement in the vertical direction. The model proposed in this study can provide accurate ZTD information and a reliable tropospheric delay correction model for precise GNSS positioning.

Generative AI Models for the Protein Scaffold Filling Problem.

De novo protein sequencing is an important problem in proteomics, playing a crucial role in understanding protein functions, drug discovery, design and evolutionary studies, etc. Top-down and bottom-up tandem mass spectrometry are popular approaches used in the field of mass spectrometry to analyze and sequence proteins. However, these approaches often produce incomplete protein sequences with gaps, namely scaffolds. The protein scaffold filling problem refers to filling the missing amino acids in the gaps of a scaffold to infer the complete protein sequence. In this article, we tackle the protein scaffold filling problem based on generative AI techniques, such as convolutional denoising autoencoder, transformer, and generative pretrained transformer (GPT) models, to complete the protein sequences and compare our results with recently developed convolutional long short-term memory-based sequence model. We evaluate the model performance both on a real dataset and generated datasets. All proposed models show outstanding prediction accuracy. Notably, the GPT-2 model achieves 100% gap-filling accuracy and 100% full sequence accuracy on the MabCampth protein scaffold, which outperforms the other models.

Post-translational modification prediction via prompt-based fine-tuning of a GPT-2 model

Post-translational modifications (PTMs) are pivotal in modulating protein functions and influencing cellular processes like signaling, localization, and degradation. The complexity of these biological interactions necessitates efficient predictive methodologies. In this work, we introduce PTMGPT2, an interpretable protein language model that utilizes prompt-based fine-tuning to improve its accuracy in precisely predicting PTMs. Drawing inspiration from recent advancements in GPT-based architectures, PTMGPT2 adopts unsupervised learning to identify PTMs. It utilizes a custom prompt to guide the model through the subtle linguistic patterns encoded in amino acid sequences, generating tokens indicative of PTM sites. To provide interpretability, we visualize attention profiles from the model’s final decoder layer to elucidate sequence motifs essential for molecular recognition and analyze the effects of mutations at or near PTM sites to offer deeper insights into protein functionality. Comparative assessments reveal that PTMGPT2 outperforms existing methods across 19 PTM types, underscoring its potential in identifying disease associations and drug targets.

Automated Extraction of Patient-Centered Outcomes After Breast Cancer Treatment: An Open-Source Large Language Model-Based Toolkit.

Patient-centered outcomes (PCOs) are pivotal in cancer treatment, as they directly reflect patients' quality of life. Although multiple studies suggest that factors affecting breast cancer-related morbidity and survival are influenced by treatment side effects and adherence to long-term treatment, such data are generally only available on a smaller scale or from a single center. The primary challenge with collecting these data is that the outcomes are captured as free text in clinical narratives written by clinicians. Given the complexity of PCO documentation in these narratives, computerized methods are necessary to unlock the wealth of information buried in unstructured text notes that often document PCOs. Inspired by the success of large language models (LLMs), we examined the adaptability of three LLMs, GPT-2, BioGPT, and PMC-LLaMA, on PCO tasks across three institutions, Mayo Clinic, Emory University Hospital, and Stanford University. We developed an open-source framework for fine-tuning LLM that can directly extract the five different categories of PCO from the clinic notes. We found that these LLMs without fine-tuning (zero-shot) struggle with challenging PCO extraction tasks, displaying almost random performance, even with some task-specific examples (few-shot learning). The performance of our fine-tuned, task-specific models is notably superior compared with their non-fine-tuned LLM models. Moreover, the fine-tuned GPT-2 model has demonstrated a significantly better performance than the other two larger LLMs. Our discovery indicates that although LLMs serve as effective general-purpose models for tasks across various domains, they require fine-tuning when applied to the clinician domain. Our proposed approach has the potential to lead more efficient, adaptable models for PCO information extraction, reducing reliance on extensive computational resources while still delivering superior performance for specific tasks.

A chatbot based question and answer system for the auxiliary diagnosis of chronic diseases based on large language model

In recent years, artificial intelligence has made remarkable strides, improving various aspects of our daily lives. One notable application is in intelligent chatbots that use deep learning models. These systems have shown tremendous promise in the medical sector, enhancing healthcare quality, treatment efficiency, and cost-effectiveness. However, their role in aiding disease diagnosis, particularly chronic conditions, remains underexplored. Addressing this issue, this study employs large language models from the GPT series, in conjunction with deep learning techniques, to design and develop a diagnostic system targeted at chronic diseases. Specifically, performed transfer learning and fine-tuning on the GPT-2 model, enabling it to assist in accurately diagnosing 24 common chronic diseases. To provide a user-friendly interface and seamless interactive experience, we further developed a dialog-based interface, naming it Chat Ella. This system can make precise predictions for chronic diseases based on the symptoms described by users. Experimental results indicate that our model achieved an accuracy rate of 97.50% on the validation set, and an area under the curve (AUC) value reaching 99.91%. Moreover, conducted user satisfaction tests, which revealed that 68.7% of participants approved of Chat Ella, while 45.3% of participants found the system made daily medical consultations more convenient. It can rapidly and accurately assess a patient’s condition based on the symptoms described and provide timely feedback, making it of significant value in the design of medical auxiliary products for household use.

Content-based Filtering Movie Recommender System Using Semantic Approach with Recurrent Neural Network Classification and SGD

The application of recommendation systems has been applied in various types of platforms, especially applications for watching movies such as Netflix and Disney+. The recommendation system is purposed to make it easier for users, especially in choosing a movie because currently the number of movie productions is increasing every day. This research proposed a CBF movie recommendation system by comparing the performance of several semantic methods to be able to get the best rating prediction results. In order to improve the performance quality to get the best rating prediction results, this research utilized semantic feature methods by comparing the performance of the evaluation results produced by the TF-IDF method and word embedding applications, such as BERT, GPT-2, RoBERTa, and implemented RNN model to classify the results of rating prediction. The data were used to generate the recommendation system by involving 854 data movie and 39 accounts with a total of 34,056 movie reviews on Twitter. This research has succeeded in getting a method that produced rating predictions, namely RoBERTa. In the classification process with the RNN model and SGD optimization, the measurement results with confusion matrix by classifying the RoBERTA rating prediction obtained an evaluation value of 0.6514 loss, 95.59% accuracy, and 0.6514 precision.

Revolutionizing Cybersecurity: The GPT-2 Enhanced Attack Detection and Defense (GEADD) Method for Zero-Day Threats

The escalating sophistication of cyber threats, particularly zero-day attacks, necessitates advanced detection methodologies in cybersecurity. This study introduces the GPT-2 Enhanced Attack Detection and Defense (GEADD) method, an innovative approach that integrates the GPT-2 model with metaheuristic optimization techniques for enhanced detection of zero-day threats. The GEADD method encompasses data preprocessing, Equilibrium Optimization (EO)-based feature selection, and Salp Swarm Algorithm-Based Optimization (SABO) for hyperparameter tuning, culminating in a robust framework capable of identifying and classifying zero-day attacks with high accuracy. Through a comprehensive evaluation using standard datasets, the GEADD method demonstrates superior performance in detecting zero-day threats compared to existing models, highlighting its potential as a significant contribution to the field of cybersecurity. This study not only presents a novel application of deep learning for cyber threat detection but also sets a foundation for future research in AI-driven cybersecurity solutions

From easy to hard: Improving personalized response generation of task-oriented dialogue systems by leveraging capacity in open-domain dialogues

A task-oriented dialogue system (TOD) is an important application of artificial intelligence. In the past few years, works on personalized TODs have attracted increased research attention and have seen much progress. The main challenge of such dialogue systems is finding ways to exploit user profiles under conditions of fixed and monolithic dialogue style. However, most of the existing works overlook the observation that the personalization capability of the dialogue system is fundamental and generic, and they treat all attributes of the user profile equally throughout the dialogue flows, which makes them inadequate for developing a well-performing personalized TOD. In this paper, we propose a two-stage learning framework equipped with GPT2 as a backbone to alleviate the above two problems. In the first learning phase, we finetune the GPT2 model on the personalized open-domain dialogues to preliminarily acquire personalization power. Then, we transfer this power to personalized task-oriented dialogues for the second stage of learning. After that, we enable the proposed model to be capable of the desired personalization capacity. Moreover, we present a dynamic profile fusion mechanism and an auxiliary task that detects which attributes contribute to the current utterance to facilitate the model’s performance. Eventually, we rewrite the attribute descriptions of user profiles in sentences to mitigate the consistency gap between the open-domain and task-oriented dialogues. The experimental results show that the proposed model achieves superior results compared to the state-of-the-art models on two versions of the Personalized bAbI dataset.

H3D-Transformer: A Heterogeneous 3D (H3D) Computing Platform for Transformer Model Acceleration on Edge Devices

Prior hardware accelerator designs primarily focused on single-chip solutions for 10 MB-class computer vision models. The GB-class transformer models for natural language processing (NLP) impose challenges on existing accelerator design due to the massive number of parameters and the diverse matrix multiplication (MatMul) workloads involved. This work proposes a heterogeneous 3D-based accelerator design for transformer models, which adopts an interposer substrate with multiple 3D memory/logic hybrid cubes optimized for accelerating different MatMul workloads. An approximate computing scheme is proposed to take advantage of heterogeneous computing paradigms of mixed-signal compute-in-memory (CIM) and digital tensor processing units (TPU). From the system-level evaluation results, 10 TOPS/W energy efficiency is achieved for the BERT and GPT2 model, which is about 2.6× ∼ 3.1× higher than the baseline with 7 nm TPU and stacked FeFET memory.

Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training.

Artificial neural networks have emerged as computationally plausible models of human language processing. A major criticism of these models is that the amount of training data they receive far exceeds that of humans during language learning. Here, we use two complementary approaches to ask how the models' ability to capture human fMRI responses to sentences is affected by the amount of training data. First, we evaluate GPT-2 models trained on 1 million, 10 million, 100 million, or 1 billion words against an fMRI benchmark. We consider the 100-million-word model to be developmentally plausible in terms of the amount of training data given that this amount is similar to what children are estimated to be exposed to during the first 10 years of life. Second, we test the performance of a GPT-2 model trained on a 9-billion-token dataset to reach state-of-the-art next-word prediction performance on the human benchmark at different stages during training. Across both approaches, we find that (i) the models trained on a developmentally plausible amount of data already achieve near-maximal performance in capturing fMRI responses to sentences. Further, (ii) lower perplexity-a measure of next-word prediction performance-is associated with stronger alignment with human data, suggesting that models that have received enough training to achieve sufficiently high next-word prediction performance also acquire representations of sentences that are predictive of human fMRI responses. In tandem, these findings establish that although some training is necessary for the models' predictive ability, a developmentally realistic amount of training (∼100 million words) may suffice.

Survey of transformers and towards ensemble learning using transformers for natural language processing

The transformer model is a famous natural language processing model proposed by Google in 2017. Now, with the extensive development of deep learning, many natural language processing tasks can be solved by deep learning methods. After the BERT model was proposed, many pre-trained models such as the XLNet model, the RoBERTa model, and the ALBERT model were also proposed in the research community. These models perform very well in various natural language processing tasks. In this paper, we describe and compare these well-known models. In addition, we also apply several types of existing and well-known models which are the BERT model, the XLNet model, the RoBERTa model, the GPT2 model, and the ALBERT model to different existing and well-known natural language processing tasks, and analyze each model based on their performance. There are a few papers that comprehensively compare various transformer models. In our paper, we use six types of well-known tasks, such as sentiment analysis, question answering, text generation, text summarization, name entity recognition, and topic modeling tasks to compare the performance of various transformer models. In addition, using the existing models, we also propose ensemble learning models for the different natural language processing tasks. The results show that our ensemble learning models perform better than a single classifier on specific tasks.Graphical

A Study of the Assistive Nature of Artificial Intelligence Technology for Japanese Translation and Interpretation

Abstract Traditional Japanese translation methods have certain disadvantages, and the introduction of artificial intelligence technology into them can enhance the effect of Japanese interpretation and translation. In this paper, the Japanese language data of Twitter and Facebook are used as the basis to construct a Japanese language interpretation and translation corpus, and the GPT-2 model is constructed on the basis of Transformer for Japanese text translation. To optimize the Seq2Seq model for Japanese speech interpretation, the Attention mechanism is introduced to establish a Japanese speech translation model. A Japanese oral and written corpus was used to analyze the validity of the methods mentioned above. The results show that the class/form ratio in the Japanese oral/translated corpus fluctuates between [0.1231, 0.1448], but the survival rate of borrowed words under the scientific category reaches the highest of 54.14%, and the average number of occurrences of each word is between [4.35, 4.95]. Japanese verbal and translated texts had an average sentence length of 40 hours, and their translation accuracy was approximately 74.16%. The quality of translation can be effectively improved, and cultural exchange between China and Japan can be enhanced by integrating AI technology with Japanese interpretation and translation.

On Methods of Data Standardization of German Social Media Comments

This article is part of a larger project aiming at identifying discursive strategies in social media discourses revolving around the topic of gender diversity, for which roughly 350,000 comments were scraped from the comments sections below YouTube videos relating to the topic in question. This article focuses on different methods of standardizing social media data in order to enhance further processing. More specifically, the data are corrected in terms of casing, spelling, and punctuation. Different tools and models (LanguageTool, T5, seq2seq, GPT-2) were tested. The best outcome was achieved by the German GPT-2 model: It scored highest in all of the applied scores (ROUGE, GLEU, BLEU), making it the best model for the task of Grammatical Error Correction in German social media data.

From text to threats: A language model approach to software vulnerability detection

Abstract In the rapidly-evolving landscape of software development, the detection of vulnerabilities in source code has become of paramount importance. Our study introduces a novel knowledge distillation (KD) technique aimed at enhancing vulnerability detection in software codebases. Using benchmark datasets such as SARD, SeVC, Devign, and D2A, we assess the prowess of the KD method when applied to different classifiers, specifically GPT2, CodeBERT, and LSTM. The empirical results are revealed a marked improvement in the performance of these classifiers upon the implementation of the KD technique, particularly with the GPT-2 model demonstrating the most promising outcomes. This work underscores the potential of integrating transformer-based learning models, like GPT-2, with knowledge distillation for more efficient and accurate vulnerability detection.

Novel Paintings from the Latent Diffusion Model through Transfer Learning

With the development of deep learning, image synthesis has achieved unprecedented achievements in the past few years. Image synthesis models, represented by diffusion models, demonstrated stable and high-fidelity image generation. However, the traditional diffusion model computes in pixel space, which is memory-heavy and computing-heavy. Therefore, to ease the expensive computing and improve the accessibility of diffusion models, we train the diffusion model in latent space. In this paper, we are devoted to creating novel paintings from existing paintings based on powerful diffusion models. Because the cross-attention layer is adopted in the latent diffusion model, we can create novel paintings with conditional text prompts. However, direct training of the diffusion model on the limited dataset is non-trivial. Therefore, inspired by the transfer learning, we train the diffusion model with the pre-trained weights, which eases the training process and enhances the image synthesis results. Additionally, we introduce the GPT-2 model to expand text prompts for detailed image generation. To validate the performance of our model, we train the model on paintings of the specific artist from the dataset WikiArt. To make up for missing image context descriptions of the WikiArt dataset, we adopt a pre-trained language model to generate corresponding context descriptions automatically and clean wrong descriptions manually, and we will make it available to the public. Experimental results demonstrate the capacity and effectiveness of the model.

Image based agorithm for automatic generation of chinese couplets

Chinese couplet is one of the important forms of expression in Chinese and even world literature, with its own unique charm and beauty. In order to meet the needs of users who only need one image to obtain corresponding couplets, realize the function of computer automatically writing couplets with images, and improve the literary expression ability of couplets to images, this paper proposes an image based intelligent generative model of couplets. The model first outputs corresponding descriptions based on image extraction features, and then extracts keywords through an improved hybrid algorithm according to the descriptions. Then, based on the keywords, the Chinese GPT-2 model automatically expands the first line of a couplet, and finally generates the second line of a couplet from the first line of a couplet through the encoding and decoding framework. Through experiments, it has been shown that the generated couplets of the model meet the requirements for image description, and the effectiveness of the model has been confirmed by manual evaluation results.

Case scenario generators for trauma surgery simulation utilizing autoregressive language models

Trauma is the leading cause of death in adults under the age of 45 and the fourth leading cause of death in the United States. Effective delivery of trauma care centers on being well versed in the Advanced Trauma Life Support (ATLS) protocol, which requires high levels of clinical experience. Often this comes from having been exposed to the many permutations of common types of injuries as well as exposed to rarer scenarios, but with potential harm to patients. Case scenarios, which are sequential representations of clinical events, can help trainees receive clinical exposure without harming patients. However authoring case scenarios requires domain expertise, wide experience, and the ability to intelligently respond to inputs, and as such is currently an arduous task. Autoregressive generative models trained on large amounts of clinical data, such as the National Trauma Data Bank (NTDB), pose a possible solution to overcome the cost of authorship while providing broad and accessible clinical experience to trainees. We have developed a Trauma AI model composed of an autoregressive generative model based on the transformer architecture for generating potential case scenario combined with an out-of-domain detection for filtering out less plausible scenarios. The GPT2 model is trained on 1.1 million case scenarios derived from the NTDB data. We demonstrate that Trauma AI is capable of generating realistic case scenarios that encode the ATLS protocol as a latent feature of the sequence of provider interventions, including scenarios that do not have any parallels in the original dataset. We also present an unsupervised means of filtering out unrealistic sequences by identifying out-of-domain sequences, and demonstrate that this improves the realism of the generated case scenarios.

Survey Paper: Automatic Title Generation for Text with RNN and Pre-trained Transformer Language Model

Abstract: Nowadays huge amounts of text data are available due to the evolution of the Internet. Although search engines are used to select text data, it is unfeasible to go through the entire search results of text that are related to search intent. Therefore, text summarization is the only method by which we can reduce text data without the loss of information. A new method for Title or Text generation is the Transformer language model that has been trained i.e. GPT-2, GPTNeo, Chat-GPT, and LSTM Model of RNN by which we can generate catchy titles to take readers’ attention and attract them to read a full article. This paper has discovered recent literature on all the previously mentioned topics which are related to Title or Text generators and examines several methods which propose to make use of various language models. The suggested approaches are also contrasted with one another to highlight their unique advantages, and to suggest constantly better ways.

Text Generator using GPT2 Model

Abstract- GPT-2 is state of art algorithm transfer learning with respect to nlp task we can do it like text summarizationand many more applications. The text generation application programming interface is supported by a large-scale unsupervised language model capable of generating paragraphs of text This transformer-based language model, based on OpenAI's GPT-2 model, accepts a sentence or partial sentence as input and predicts subsequent text. GPT-2 uses input textto set the initial context for further text generation. The length of an input string can range from few words to a maximumsequence length of 1024 tokens. The longer an initial input, the more subject context is provided to a model. Generally, longer inputs produce a more coherent text output. It was specifically trained to guess the next word in sentences. GPT-2 isa massive model that contains a massive amount of compressed knowledge from various parts of the internet. It can be usedto forecast the likelihood of a sentence. The model learns an internal representation of the English language, which it can then use to extract features useful for subsequent tasks. We have used a tkinter which is a GUI to display the generated output. Keywords—Text generation, Graphical User Interface, gpt2 transformer,Open AI,tkinter,

Popular Tag Recommendation by Neural Network in Social Media.

Although "a picture is worth a thousand words," this may not be enough to get your post seen on social media. This study's main objective was to determine the best ways to characterize a photo in terms of viral marketing and public appealing. We have to obtain this dataset for this reason from the social media site such as Instagram. A total of 1.4 million hashtags were used in the 570,000 photos that we crawled. Prior to training the text generation module to produce such popular hashtags, we had to determine the components and features of the photo. We trained a multilabel image classification module using a ResNet neural network model for the first section. In order to create hashtags pertaining to their popularity, we trained a cutting-edge GPT-2 language model for the second portion. This work differs from others in that, and it initially offered a cutting-edge GPT-2 model for hashtag generation using a combination of the multilabel image classification module. The popularity issues and ways to make an Instagram post popular are also highlighted in our essay. Social science and marketing research can both be conducted on this subject. Which content can be considered popular from the perspective of consumers can be researched in the social science setting. As a marketing strategy, end users can help by offering such well-liked hashtags for social media accounts. This essay adds to the body of knowledge by demonstrating the two possible uses of popularity. Compared to the base model, our popular hashtag generating algorithm creates 11% more relevant, acceptable, and trending hashtags, according to the evaluation that was carried out.