Use Of Language Models Research Articles

The Use of Language Models to Support the Development of Cartographic Descriptions of a Building’s Interior

The Use of Language Models to Support the Development of Cartographic Descriptions of a Building’s Interior

Proof-of-concept study of a small language model chatbot for breast cancer decision support – a transparent, source-controlled, explainable and data-secure approach

PurposeLarge language models (LLM) show potential for decision support in breast cancer care. Their use in clinical care is currently prohibited by lack of control over sources used for decision-making, explainability of the decision-making process and health data security issues. Recent development of Small Language Models (SLM) is discussed to address these challenges. This preclinical proof-of-concept study tailors an open-source SLM to the German breast cancer guideline (BC-SLM) to evaluate initial clinical accuracy and technical functionality in a preclinical simulation.MethodsA multidisciplinary tumor board (MTB) is used as the gold-standard to assess the initial clinical accuracy in terms of concordance of the BC-SLM with MTB and comparing it to two publicly available LLM, ChatGPT3.5 and 4. The study includes 20 fictional patient profiles and recommendations for 5 treatment modalities, resulting in 100 binary treatment recommendations (recommended or not recommended). Statistical evaluation includes concordance with MTB in % including Cohen’s Kappa statistic (κ). Technical functionality is assessed qualitatively in terms of local hosting, adherence to the guideline and information retrieval.ResultsThe overall concordance amounts to 86% for BC-SLM (κ = 0.721, p < 0.001), 90% for ChatGPT4 (κ = 0.820, p < 0.001) and 83% for ChatGPT3.5 (κ = 0.661, p < 0.001). Specific concordance for each treatment modality ranges from 65 to 100% for BC-SLM, 85–100% for ChatGPT4, and 55–95% for ChatGPT3.5. The BC-SLM is locally functional, adheres to the standards of the German breast cancer guideline and provides referenced sections for its decision-making.ConclusionThe tailored BC-SLM shows initial clinical accuracy and technical functionality, with concordance to the MTB that is comparable to publicly-available LLMs like ChatGPT4 and 3.5. This serves as a proof-of-concept for adapting a SLM to an oncological disease and its guideline to address prevailing issues with LLM by ensuring decision transparency, explainability, source control, and data security, which represents a necessary step towards clinical validation and safe use of language models in clinical oncology.

Optimization of traditional methods for determining the similarity of project names and purchases using large language models

The subject of the study is the analysis and improvement of methods for determining the relevance of project names to the information content of purchases using large language models. The object of the study is a database containing the names of projects and purchases in the field of electric power industry, collected from open sources. The author examines in detail such aspects of the topic as the use of TF-IDF and cosine similarity metrics for primary data filtering, and also describes in detail the integration and evaluation of the effectiveness of large language models such as GigaChat, GPT-3.5, and GPT-4 in text data matching tasks. Special attention is paid to the methods of clarifying the similarity of names based on reflection introduced into the prompta of large language models, which makes it possible to increase the accuracy of data comparison. The study uses TF-IDF and cosine similarity methods for primary data analysis, as well as large GigaChat, GPT-3.5 and GPT-4 language models for detailed verification of the relevance of project names and purchases, including reflection in model prompta to improve the accuracy of results. The novelty of the research lies in the development of a combined approach to determining the relevance of project names and purchases, combining traditional methods of processing text information (TF-IDF, cosine similarity) with the capabilities of large language models. A special contribution of the author to the research of the topic is the proposed methodology for improving the accuracy of data comparison by clarifying the results of primary selection using GPT-3.5 and GPT-4 models with optimized prompta, including reflection. The main conclusions of the study are confirmation of the prospects of using the developed approach in the tasks of information support for procurement processes and project implementation, as well as the possibility of using the results obtained for the development of text data mining systems in various sectors of the economy. The study showed that the use of language models makes it possible to improve the value of the F2 measure to 0.65, which indicates a significant improvement in the quality of data comparison compared with basic methods.

ChatGPT adoption and anxiety: a cross-country analysis utilising the unified theory of acceptance and use of technology (UTAUT)

ABSTRACT The public release of ChatGPT in November 2022 brought excitement and concerns regarding students’ use of language models in higher education. However, little research has empirically investigated students’ intention to adopt ChatGPT. This study developed a theoretical model based on the Unified Theory of Acceptance and Use of Technology (UTAUT) with an additional construct- anxiety, to investigate university students’ adoption intention of ChatGPT in two higher education contexts– the UK and Nepal. 239 and 226 questionnaires were deemed sufficient for data analysis for Nepal and the UK, respectively. We utilised the structural equation modelling technique to test the hypotheses. Our results reveal that performance expectancy, effort expectancy and social influence significantly impacted the adoption intention of ChatGPT for both countries. However, anxiety's impact varied between Nepal and the UK. Integrating UTAUT with a cross-country comparative approach provides insights into how ChatGPT's reception diverges between different higher education contexts. Our results also have implications for technology companies aiming to expand language models’ availability worldwide.

Optimizing classification of diseases through language model analysis of symptoms

This paper investigated the use of language models and deep learning techniques for automating disease prediction from symptoms. Specifically, we explored the use of two Medical Concept Normalization—Bidirectional Encoder Representations from Transformers (MCN-BERT) models and a Bidirectional Long Short-Term Memory (BiLSTM) model, each optimized with a different hyperparameter optimization method, to predict diseases from symptom descriptions. In this paper, we utilized two distinct dataset called Dataset-1, and Dataset-2. Dataset-1 consists of 1,200 data points, with each point representing a unique combination of disease labels and symptom descriptions. While, Dataset-2 is designed to identify Adverse Drug Reactions (ADRs) from Twitter data, comprising 23,516 rows categorized as ADR (1) or Non-ADR (0) tweets. The results indicate that the MCN-BERT model optimized with AdamP achieved 99.58% accuracy for Dataset-1 and 96.15% accuracy for Dataset-2. The MCN-BERT model optimized with AdamW performed well with 98.33% accuracy for Dataset-1 and 95.15% for Dataset-2, while the BiLSTM model optimized with Hyperopt achieved 97.08% accuracy for Dataset-1 and 94.15% for Dataset-2. Our findings suggest that language models and deep learning techniques have promise for supporting earlier detection and more prompt treatment of diseases, as well as expanding remote diagnostic capabilities. The MCN-BERT and BiLSTM models demonstrated robust performance in accurately predicting diseases from symptoms, indicating the potential for further related research.

Comparison of ChatGPT and Bard for using in hybrid intelligent information systems

The purpose of this paper is to conduct research and comparative analysis of modern large language models, in particular, such as ChatGPT and Google Bard. As part of the research, the analysis of the advantages and disadvantages of advanced artificial intelligence technologies in various fields of application was carried out. Optimal conditions for using these models were identified, and methods for overcoming the identified shortcomings of large language models based on the mivar approach were proposed. Special attention is paid to the areas of application of large language models, such as providing a quick and effective response to user requests, as well as their use in training and staff adaptation tasks. This paper analyzes large language models, taking into account their integration methods, as well as the possibilities of creating personalized systems for automating communications. The research results include an analysis and comparison of the capabilities of LLM and identifying their advantages and disadvantages with a focus on the problem of “hallucinations”. The paper also proposes hypotheses about the potential overcoming of LLM limitations using the mivar approach. The results of experiments with ChatGPT confirm the relevance of creating structured knowledge and automating the process of building mivar data models, as well as indicate the prospects for combining LLM and the mivar approach. This can reduce the likelihood of generating erroneous information, increase the interpretability of results, and ensure more effective use of language models in various scenarios of artificial intelligence use.

Повышение эффективности государственной политики в сфере кинематографа с помощью машинного обучения

A study was conducted on the distribution data of a range of Russian national films from 2004 to September 2023 using machine learning methods, with successful and unsuccessful films and patriotic projects considered separately. The study utilized the ensemble machine learning model HistGradientBoostingClassifier and a sequential fully connected three-layer neural network based on the TensorFlow library, along with basic methods of natural language processing. It was found that patriotic films exhibit distribution characteristics lower than the market average and significantly lag behind successful Russian cinema projects at the box office. The study demonstrated the possibility of accurately predicting film box office receipts, distribution characteristics, as well as selecting project parameters and the composition of its creative team to enhance distribution results and increase the reach of target audiences. The study also illustrated the use of language models, specifically through film annotations, to create effective film content. A conclusion was drawn regarding the need for a set of measures to implement state policy in the field of education, information technology, culture, and art, focusing on the creation of unified media franchises and umbrella brands encompassing films, TV series, computer and online games, music, and other products. The implementation of the research results will enhance the effectiveness of public policy and the return on public and private- public investments in the creative industries sector, positioning cinema and the creative industry as self-sustaining and income-generating sectors of the economy. Leveraging rental data from other markets will facilitate the creation of content with higher demand globally, serving as a soft power and cultural influence tool. The results of the study can aid in project selection and film project development by the Russian Cinema Fund, the Ministry of Culture of the Russian Federation, private Russian investors and film studios, as well as authorities and organizations responsible for distributing Russian film content abroad.

A Survey on the Intersection of Research in the Use of Language Models and Knowledge Graphs

A Survey on the Intersection of Research in the Use of Language Models and Knowledge Graphs

Leveraging Language Models for Inpatient Diagnosis Coding

Medical coding plays an essential role in medical billing, health resource planning, clinical research and quality assessment. Automated coding systems offer promising solutions to streamline the coding process, improve accuracy and reduce the burden on medical coders. To date, there has been limited research focusing on inpatient diagnosis coding using an extensive comprehensive dataset and encompassing the full ICD-10 code sets. In this study, we investigate the use of language models for coding inpatient diagnoses and examine their performance using an institutional dataset comprising 230,645 inpatient admissions and 8677 diagnosis codes spanning over a six-year period. A total of three language models, including two general-purpose models and a domain-specific model, were evaluated and compared. The results show competitive performance among the models, with the domain-specific model achieving the highest micro-averaged F1 score of 0.7821 and the highest mean average precision of 0.8097. Model performance varied by disease and condition, with diagnosis codes with larger sample sizes producing better results. The rarity of certain diseases and conditions posed challenges to accurate coding. The results also indicated the potential difficulties of the model with long clinical documents. Our models demonstrated the ability to capture relevant associations between diagnoses. This study advances the understanding of language models for inpatient diagnosis coding and provides insights into the extent to which the models can be used.

The Utility of Language Models in Cardiology: A Narrative Review of the Benefits and Concerns of ChatGPT-4.

Artificial intelligence (AI) and language models such as ChatGPT-4 (Generative Pretrained Transformer) have made tremendous advances recently and are rapidly transforming the landscape of medicine. Cardiology is among many of the specialties that utilize AI with the intention of improving patient care. Generative AI, with the use of its advanced machine learning algorithms, has the potential to diagnose heart disease and recommend management options suitable for the patient. This may lead to improved patient outcomes not only by recommending the best treatment plan but also by increasing physician efficiency. Language models could assist physicians with administrative tasks, allowing them to spend more time on patient care. However, there are several concerns with the use of AI and language models in the field of medicine. These technologies may not be the most up-to-date with the latest research and could provide outdated information, which may lead to an adverse event. Secondly, AI tools can be expensive, leading to increased healthcare costs and reduced accessibility to the general population. There is also concern about the loss of the human touch and empathy as AI becomes more mainstream. Healthcare professionals would need to be adequately trained to utilize these tools. While AI and language models have many beneficial traits, all healthcare providers need to be involved and aware of generative AI so as to assure its optimal use and mitigate any potential risks and challenges associated with its implementation. In this review, we discuss the various uses of language models in the field of cardiology.

Discriminator-Enhanced Knowledge-Distillation Networks

Query auto-completion (QAC) serves as a critical functionality in contemporary textual search systems by generating real-time query completion suggestions based on a user’s input prefix. Despite the prevalent use of language models (LMs) in QAC candidate generation, LM-based approaches frequently suffer from overcorrection issues during pair-wise loss training and efficiency deficiencies. To address these challenges, this paper presents a novel framework—discriminator-enhanced knowledge distillation (Dis-KD)—for the QAC task. This framework combines three core components: a large-scale pre-trained teacher model, a lightweight student model, and a discriminator for adversarial learning. Specifically, the discriminator aids in discerning generative-level differences between the teacher and the student models. An additional discriminator score loss is amalgamated with the traditional knowledge-distillation loss, resulting in enhanced performance of the student model. Contrary to the stepwise evaluation of each generated word, our approach assesses the entire generation sequence. This method alleviates the prevalent overcorrection issue in the generation process. Consequently, our proposed framework boasts improvements in model accuracy and a reduction in parameter size. Empirical results highlight the superiority of Dis-KD over established baseline methods, with the student model surpassing the teacher model in QAC tasks for sub-word languages.

Assisting ELT Teachers: Designing Activities for the Use of ChatGPT in Teaching and Learning

This research aims to explore the potential of ChatGPT, a language model developed by OpenAI, in enhancing teaching and learning. The study focuses on designing activities that use ChatGPT to engage learners in language and content-based tasks. The research method includes a literature review and a design-based research approach. The literature review examines the current state of the art in the use of language models in education and identifies key issues, challenges, and opportunities. The design-based research approach involves developing, implementing, and evaluating a series of activities that use ChatGPT to facilitate language and content learning. The study aims to contribute to the development of innovative and effective pedagogical practices that leverage the power of language models in education. The results of the study will be of interest to educators, researchers, and developers who are interested in the integration of AI and education.

The future of Pediatric Neurosurgery and ChatGPT: an editor's perspective

The field of pediatric neurosurgery is constantly evolving, and it is difficult to predict the exact future of the field. However, there are some trends and developments that are likely to shape the future of pediatric neurosurgery:&#x0D; -Advancements in technology: Technology is rapidly advancing, and this is likely to have a significant impact on pediatric neurosurgery. For example, robotic-assisted surgery is becoming increasingly common, which can improve precision and reduce the risk of complications.&#x0D; -Gene therapy: Gene therapy is a promising area of research that has the potential to revolutionize the treatment of many neurological disorders. In the future, gene therapy may be used to treat disorders such as brain tumors and epilepsy.&#x0D; -Stem cell therapy: Stem cells have the potential to differentiate into a wide range of cells, including neurons. This makes stem cell therapy a promising area of research for the treatment of neurological disorders.&#x0D; -Minimally invasive procedures: As technology advances, minimally invasive procedures are becoming increasingly common. These procedures typically involve smaller incisions, less scarring, and a faster recovery time.&#x0D; -Precision medicine: Precision medicine is an approach to healthcare that involves tailoring treatment to the specific needs of each patient. In the future, pediatric neurosurgeons may use precision medicine to develop personalized treatment plans for each patient.&#x0D; Overall, the future of pediatric neurosurgery looks promising, with advances in technology, gene therapy, stem cell therapy, minimally invasive procedures, and precision medicine likely to have a significant impact on the field.&#x0D; As the human author of this article, I would like to state that the majority of this article was written by ChatGPT. The use of chatbots and natural language processing (NLP) technology, such as generative pretrained transformer (GPT), has the potential to revolutionize the field of medical writing, albeit with caution.&#x0D; &#x0D; So, What is ChatGPT?&#x0D; ChatGPT is a type of artificial intelligence (AI) model developed by the company OpenAI and has been designed to generate human-like text. GPT is a type of language model, which means that it has been trained on a large dataset of text and is able to generate new text that is similar to the text it was trained on. It has been particularly successful in the field of NLP, which involves the development of computer systems that are able to understand, interpret, and generate human language (1).&#x0D; Currently, it is important to note that language models such as chatGPT are not yet able to fully replace human medical writers, as they do not have the same level of understanding and expertise in the field of medicine. In addition, the use of language models in medical writing raises ethical concerns, such as the potential for errors or biases in the generated text (2,3).&#x0D; There are factors that still need to be discussed at a deeper level such as ethics in scientific writing, medico-legal factors, accountability, accuracy, and transparency.&#x0D; In addition, the use of any AI tools such as ChatGPT should be transparently declared, similar to the general practice of acknowledging writing assistance in an article or study.&#x0D; These large language models can be a useful aid in scientific writing and editing, especially for non-native English speakers, and can even aid the author in adding accurately formatted references. However, authors should limit the use of these tools to topics that they are subject matter experts on to ensure that the information provided is accurate and up to date. No matter what AI tool is used, the authors remain responsible for the scientific integrity of their publications.

EVALUATING BERT AND PARSBERT FOR ANALYZING PERSIAN ADVERTISEMENT DATA

This paper discusses the impact of the Internet on modern trading and the importance of data generated from these transactions for organizations to improve their marketing efforts. The paper uses the example of Divar, an online marketplace for buying and selling products and services in Iran, and presents a competition to predict the percentage of a car sales ad that would be published on the Divar website. Since the dataset provides a rich source of Persian text data, the authors use the Hazm library, a Python library designed for processing Persian text, and two state-of-the-art language models, mBERT and ParsBERT, to analyze it. The paper's primary objective is to compare the performance of mBERT and ParsBERT on the Divar dataset. The authors provide some background on data mining, Persian language, and the two language models, examine the dataset's composition and statistical features, and provide details on their fine-tuning and training configurations for both approaches. They present the results of their analysis and highlight the strengths and weaknesses of the two language models when applied to Persian text data. The paper offers valuable insights into the challenges and opportunities of working with low-resource languages such as Persian and the potential of advanced language models like BERT for analyzing such data. The paper also explains the data mining process, including steps such as data cleaning and normalization techniques. Finally, the paper discusses the types of machine learning problems, such as supervised, unsupervised, and reinforcement learning, and the pattern evaluation techniques, such as confusion matrix. Overall, the paper provides an informative overview of the use of language models and data mining techniques for analyzing text data in low-resource languages, using the example of the Divar dataset.

Augmenting human innovation teams with artificial intelligence: Exploring transformer‐based language models

AbstractThe use of transformer‐based language models in artificial intelligence (AI) has increased adoption in various industries and led to significant productivity advancements in business operations. This article explores how these models can be used to augment human innovation teams in the new product development process, allowing for larger problem and solution spaces to be explored and ultimately leading to higher innovation performance. The article proposes the use of the AI‐augmented double diamond framework to structure the exploration of how these models can assist in new product development (NPD) tasks, such as text summarization, sentiment analysis, and idea generation. It also discusses the limitations of the technology and the potential impact of AI on established practices in NPD. The article establishes a research agenda for exploring the use of language models in this area and the role of humans in hybrid innovation teams. (Note: Following the idea of this article, GPT‐3 alone generated this abstract. Only minor formatting edits were performed by humans.)

МЕТОДИ ПІДВИЩЕННЯ ЯКОСТІ ПЕРЕТВОРЕННЯ МОВИ НА ТЕКСТ В СИСТЕМАХ БІОМЕТРИЧНОЇ АУТЕНТИФІКАЦІЇ

The article discusses the methods and algorithms of speech-to-text conversion, modern open and commercial systems for creating systems, as well as the use of these technologies in the field of cyber security. It is proposed to create a high-quality speech-to-text conversion system. An analysis of the mathematical algorithms used to reduce the error rate, which makes it possible to create unique voice prints and increase protection against forgery, has been carried out. The structure of modern speech-to-text conversion systems is described. By changing datasets, parameters of hidden Markov models, a high-quality dictionary of phonemes, and the use of language models, there is an opportunity to reduce the percentage of errors in language recognition, as well as the use of a system for multilingualism such as "surzhyk". The mathematical methods of assessing the quality of the system of speech to text (WER), as well as various methods of calculation, which is important for their further improvement and optimization, are considered. The structure of modern systems is considered, namely, signal pre-processing, feature extraction, acoustic modeling, speech modeling, decoding, post-processing. For each of the stages, study vectors have been proposed that can reduce the error rate of the system as a whole. Reducing speech recognition errors and the ability to fake a voice is achieved using various methods: deep neural networks, hidden Markov models, Baum-Welch algorithm, N-gram models, models with attention, creation of a high-quality phonemes dictionary, dataset, and fillers. Speech-to-text conversion technology can be used in biometric authentication systems to detect and analyze the unique features of the user's voice. However, modern speech-to-text conversion systems for Ukrainian, Russian, and "surzhyk" need improvement in acoustic and language units. Scientific works, which are devoted to research and optimization of these systems for biometric authentication, do not fully cover these issues. This became the reason for further research in this direction, so this work aims to create a speech recognition system with a minimum error rate.

Language and culture internalization for human-like autotelic AI

Building autonomous agents able to grow open-ended repertoires of skills across their lives is a fundamental goal of artificial intelligence (AI). A promising developmental approach recommends the design of intrinsically motivated agents that learn new skills by generating and pursuing their own goals - autotelic agents. But despite recent progress, existing algorithms still show serious limitations in terms of goal diversity, exploration, generalisation or skill composition. This perspective calls for the immersion of autotelic agents into rich socio-cultural worlds, an immensely important attribute of our environment that shapes human cognition but is mostly omitted in modern AI. Inspired by the seminal work of Vygotsky, we propose Vygotskian autotelic agents - agents able to internalise their interactions with others and turn them into cognitive tools. We focus on language and show how its structure and informational content may support the development of new cognitive functions in artificial agents as it does in humans. We justify the approach by uncovering several examples of new artificial cognitive functions emerging from interactions between language and embodiment in recent works at the intersection of deep reinforcement learning and natural language processing. Looking forward, we highlight future opportunities and challenges for Vygotskian Autotelic AI research, including the use of language models as cultural models supporting artificial cognitive development.

ASRS-CMFS vs. RoBERTa: Comparing Two Pre-Trained Language Models to Predict Anomalies in Aviation Occurrence Reports with a Low Volume of In-Domain Data Available

We consider the problem of solving Natural Language Understanding (NLU) tasks characterized by domain-specific data. An effective approach consists of pre-training Transformer-based language models from scratch using domain-specific data before fine-tuning them on the task at hand. A low domain-specific data volume is problematic in this context, given that the performance of language models relies heavily on the abundance of data during pre-training. To study this problem, we create a benchmark replicating realistic field use of language models to classify aviation occurrences extracted from the Aviation Safety Reporting System (ASRS) corpus. We compare two language models on this new benchmark: ASRS-CMFS, a compact model inspired from RoBERTa, pre-trained from scratch using only little domain-specific data, and the regular RoBERTa model, with no domain-specific pre-training. The RoBERTa model benefits from its size advantage, while the ASRS-CMFS benefits from the pre-training from scratch strategy. We find no compelling statistical evidence that RoBERTa outperforms ASRS-CMFS, but we show that ASRS-CMFS is more compute-efficient than RoBERTa. We suggest that pre-training a compact model from scratch is a good strategy for solving domain-specific NLU tasks using Transformer-based language models in the context of domain-specific data scarcity.

Image captioning for effective use of language models in knowledge-based visual question answering

Integrating outside knowledge for reasoning in visio-linguistic tasks such as visual question answering (VQA) is an open problem. Given that pretrained language models have been shown to include world knowledge, we propose to use a unimodal (text-only) train and inference procedure based on automatic off-the-shelf captioning of images and pretrained language models. More specifically, we verbalize the image contents and allow language models to better leverage their implicit knowledge to solve knowledge-intensive tasks. Focusing on a visual question answering task which requires external knowledge (OK-VQA), our contributions are: (i) a text-only model that outperforms pretrained multimodal (image-text) models of comparable number of parameters; (ii) confirmation that our text-only method is specially effective for tasks requiring external knowledge, as it is less effective in standard a VQA task (VQA 2.0); and (iii) our method attains results in the state-of-the-art when increasing the size of the language model. We also significantly outperform current multimodal systems, even though augmented with external knowledge. Our qualitative analysis on OK-VQA reveals that automatic captions often fail to capture relevant information in the images, which seems to be balanced by the better inference ability of the text-only language models. Our work opens up possibilities to further improve inference in visio-linguistic tasks.

Generate, Annotate, and Learn: NLP with Synthetic Text

Abstract This paper studies the use of language models as a source of synthetic unlabeled text for NLP. We formulate a general framework called “generate, annotate, and learn (GAL)” to take advantage of synthetic text within knowledge distillation, self-training, and few-shot learning applications. To generate high-quality task-specific text, we either fine-tune LMs on inputs from the task of interest, or prompt large LMs with few examples. We use the best available classifier to annotate synthetic text with soft pseudo labels for knowledge distillation and self-training, and use LMs to obtain hard labels for few-shot learning. We train new supervised models on the combination of labeled and pseudo-labeled data, which results in significant gains across several applications. We investigate key components of GAL and present theoretical and empirical arguments against the use of class-conditional LMs to generate synthetic labeled text instead of unlabeled text. GAL achieves new state-of-the-art knowledge distillation results for 6-layer transformers on the GLUE leaderboard.