Language Processing Tasks Research Articles

Point-to-Pixel Prompting for Point Cloud Analysis With Pre-Trained Image Models.

Nowadays, pre-training big models on large-scale datasets has achieved great success and dominated many downstream tasks in natural language processing and 2D vision, while pre-training in 3D vision is still under development. In this paper, we provide a new perspective of transferring the pre-trained knowledge from 2D domain to 3D domain with Point-to-Pixel Prompting in data space and Pixel-to-Point distillation in feature space, exploiting shared knowledge in images and point clouds that display the same visual world. Following the principle of prompting engineering, Point-to-Pixel Prompting transforms point clouds into colorful images with geometry-preserved projection and geometry-aware coloring. Then the pre-trained image models can be directly implemented for point cloud tasks without structural changes or weight modifications. With projection correspondence in feature space, Pixel-to-Point distillation further regards pre-trained image models as the teacher model and distills pre-trained 2D knowledge to student point cloud models, remarkably enhancing inference efficiency and model capacity for point cloud analysis. We conduct extensive experiments in both object classification and scene segmentation under various settings to demonstrate the superiority of our method. In object classification, we reveal the important scale-up trend of Point-to-Pixel Prompting and attain 90.3% accuracy on ScanObjectNN dataset, surpassing previous literature by a large margin. In scene-level semantic segmentation, our method outperforms traditional 3D analysis approaches and shows competitive capacity in dense prediction tasks.

Clinical Prompt Learning With Frozen Language Models.

When the first transformer-based language models were published in the late 2010s, pretraining with general text and then fine-tuning the model on a task-specific dataset often achieved the state-of-the-art performance. However, more recent work suggests that for some tasks, directly prompting the pretrained model matches or surpasses fine-tuning in performance with few or no model parameter updates required. The use of prompts with language models for natural language processing (NLP) tasks is known as prompt learning. We investigated the viability of prompt learning on clinically meaningful decision tasks and directly compared this with more traditional fine-tuning methods. Results show that prompt learning methods were able to match or surpass the performance of traditional fine-tuning with up to 1000 times fewer trainable parameters, less training time, less training data, and lower computation resource requirements. We argue that these characteristics make prompt learning a very desirable alternative to traditional fine-tuning for clinical tasks, where the computational resources of public health providers are limited, and where data can often not be made available or not be used for fine-tuning due to patient privacy concerns. The complementary code to reproduce the experiments presented in this work can be found at https://github.com/NtaylorOX/Public_Clinical_Prompt.

Deep Learning based Part-of-Speech tagging for Assamese using RNN and GRU

Deep Learning (DL) techniques have been widely used in different Natural Language Processing (NLP) tasks. Parts of Speech (PoS) tagging is one where a wide variety of DL techniques have been experimented with across the languages. Here in the present work, Recurrent Neural Network (RNN) and Gated Recurrent Unit (GRU) based Parts of Speech taggers have been trained and modelled for Assamese, an Indo Aryan family language. Universal Parts of Speech (UPoS) tag set of 17 tags were used for the experiment. A dataset of 30000 sequences has been used for the work, which is originally a BIS tag set tagged dataset, and customized through conversion from BIS tagged sequences to UPoS tagged sequences. RNN and GRU based systems have been configured using tensorflow platform and the performance measurement was done through accuracy, precision, recall and F1 scores. The accuracy of the RNN based system has been found to be 93.78%. Precision of 94.75 and recall of 93.28 were recorded for the RNN model. Accuracy of 94.38%, precision of 95.44 and recall of 93.7 were recorded for the GRU model. RNN and GRU models respectively yield F1 scores of 94.01 and 94.56. Although PoS tagging with other tag sets like BIS have been attempted by other researchers, UPoS tagging using DL approaches for Assamese is attempted for the first time. And this baseline work with observed accuracies of 93.78 and 94.38 for RNN and GRU respectively, shall serve as reference models for further works.

TCM-GPT: Efficient pre-training of large language models for domain adaptation in Traditional Chinese Medicine

Pre-training and fine-tuning have emerged as a promising paradigm across various natural language processing (NLP) tasks. The effectiveness of pretrained large language models (LLM) has witnessed further enhancement, holding potential for applications in the field of medicine, particularly in the context of Traditional Chinese Medicine (TCM). However, the application of these general models to specific domains often yields suboptimal results, primarily due to challenges like lack of domain knowledge, unique objectives, and computational efficiency. Furthermore, their effectiveness in specialized domains, such as Traditional Chinese Medicine, requires comprehensive evaluation.To address the above issues, we propose a novel domain specific TCMDA (TCM Domain Adaptation) approach, efficient pre-training with domain-specific corpus. Specifically, we first construct a large TCM-specific corpus, TCM-Corpus-1B, by identifying domain keywords and retrieving from general corpus. Then, our TCMDA leverages the LoRA which freezes the pretrained model’s weights and uses rank decomposition matrices to efficiently train specific dense layers for pre-training and fine-tuning, efficiently aligning the model with TCM-related tasks, namely TCM-GPT-7B. We further conducted extensive experiments on two TCM tasks, including TCM examination and TCM diagnosis. TCM-GPT-7B archived the best performance across both datasets, outperforming other models by relative increments of 17% and 12% in accuracy, respectively. To the best of our knowledge, our study represents the pioneering validation of domain adaptation of a large language model with 7 billion parameters in TCM domain. We will release both TCM-Corpus-1B and TCM-GPT-7B model once accepted to facilitate interdisciplinary development in TCM and NLP, serving as the foundation for further study.

Study of Text Patterns Found on Social Networks of Mental Health Reactions to COVID-19.

SARS-CoV-2 is an infectious disease caused by the coronavirus that was first reported in December 2019 in China and immediately spread around the world causing a pandemic, which has caused countless deaths and cases in global health. Mental health has not gone untouched by this pandemic; due to the lockdown and the vast amounts of information disseminated, the Panamanian population has begun to feel the collateral effects. We propose classifying tweets using a machine learning (ML) and deep learning (DL) approach and pattern search to make recommendations to the emotional and psychological reactions of the Panamanian population. Our study has been carried out with a corpus in spanish extracted from X for the automatic classification of texts, from which we have categorized, through the ML&DL approach, the tweets about Covid-19 in Panama, in order to know if the population has suffered any mental health effects. We can say that the ML models provide competitive results in terms of automatic identification of texts with an accuracy of 90%. X is a social network and an important information channel where you can explore, analyze and organize opinions to make better decisions. Text mining and patron search are a natural language processing (NLP) task that, using ML&DL algorithms, can integrate innovative strategies into information and communication technologies.

Syntactic Category based Assamese Question Pattern Extraction using N-grams

Syntactic categories are important because they play a major role in analysis of sentences. Assamese is a versatile language added with diversified linguistic features. Analysis and manipulation of such a rich language in terms of its vocabulary is really challenging. It requires incorporation of syntactic level annotations like Parts of Speech (PoS) tags into the text for the purpose of efficient and effective processing for different Natural Language Processing (NLP) tasks. Our research mainly focuses on extraction of patterns of syntactic categories in Assamese question sentences. We have also worked on extracting the Bi-grams and Tri-grams of these patterns so that analysis and processing can be done on more basic level. In this paper, we have tried to focus on extraction of PoS tag based Assamese question patterns. We also have tried to generate the Bi-grams and Tri-grams with their frequencies of occurrences throughout the patterns. The work is a noval attempt in modeling question sentence patterns with the embedded PoS patterns.

Automated Spelling Error Detection in Assamese Texts using Deep Learning Approaches

The growing prevalence of regional languages, including Assamese, in user-generated content on social media platforms, has brought the problem of spelling errors into focus. In the realm of natural language processing (NLP) tasks, especially in hate speech and toxic comment detection models, the accuracy of spelling becomes of utmost importance to ensure that misspelled terms do not slip through undetected. This highlights the need for effective approaches to address spelling issues in such language-based applications. This research work suggests a novel approach for identifying word-level spelling errors in Assamese text. The approach employs two deep learning techniques, Long-short-term memory (LSTM) and Bidirectional Long-short-term memory (BiLSTM), to accurately detect misspelled words by analyzing the context of each word inside a sentence. The models are trained using a dataset of 2677 Assamese sentences that include both correctly and incorrectly spelled words. The LSTM and Bi-LSTM models have shown significant improvement over previous spelling detection research in Assamese. LSTM achieved an accuracy of 92.72%, while Bi-LSTM achieved 94.59% accuracy. These results highlight the efficacy of deep learning methods in identifying spelling errors in the Assamese language. By ensuring precise spelling detection and reducing misunderstandings caused by misspelled words, the proposed approach can be implemented to improve the reliability of NLP tasks such as identifying toxic comments and hate speech.

Efficient Loss Landscape Reshaping for Convolutional Neural Networks.

Theoretical and empirical evidence highlights a positive correlation between the flatness of loss landscapes around minima and generalization. However, most current approaches that seek to find flat minima either incur high computational costs or struggle to balance generalization, training stability, and convergence. This work proposes reshaping the loss landscape to induce the optimizer toward flat regions, an approach that has negligible computational costs and does not compromise training stability, convergence, or efficiency. We focus on nonlinear, loss-dependent reshaping functions underpinned by theoretical insights to reshape the loss landscape. To design these functions, we first identify where and how these functions should be applied. With the aid of recently developed tools in stochastic optimization, theoretical analysis shows that steepening the low-loss landscape improves the rate of sharp minimum escape while flattening the high-and ultralow-loss landscapes enhances training stability and optimization performance, respectively. Simulations and experiments reveal that the subtly designed reshaping functions not only induce optimizers to find flat minima and improve generalization performance but also stabilize training, promote optimization, and keep efficiency. Our approach is evaluated on image classification, adversarial robustness, and natural language processing (NLP) tasks and achieves significant improvement in generalization performance with negligible computational cost. We believe that the new perspective introduced in this work will broadly impact the field of deep neural network training. The code is available at https://github.com/LongJin-lab/LLR.

Investigating Natural Language Techniques for Accurate Noun and Verb Extraction

Natural language processing (NLP) has witnessed significant advancements in recent decades. Automatically classifying parts of speech, like nouns and verbs, from textual input has transformed text analysis and language understanding. Using natural language processing techniques, we explore various methods for identifying noun and verb phrases automatically, with an emphasis on high accuracy. Our study explores rule-based, statistical, and Machine Learning (ML) approaches for determining the nouns and verbs from sentences. The effectiveness of these approaches is clearly evident, especially when NLP libraries such as SpaCy and the Natural Language Toolkit (NLTK) are used. As well as demonstrating their potential applications across diverse language processing tasks and industries, we conduct comparative research to showcase their advantages and disadvantages. The performance of these methods is also examined in terms of retrieving subject and action terms. SpaCy achieves an impressive accuracy of 95% in noun and verb extraction, while Part-Of-Speech (POS) technology tagging delivers an even higher accuracy of 96%. The results obtained with these methods illustrate how nouns, verbs, and names can be classified in text successfully.

Unlocking the Potential: A Comprehensive Systematic Review of ChatGPT in Natural Language Processing Tasks

Unlocking the Potential: A Comprehensive Systematic Review of ChatGPT in Natural Language Processing Tasks

Enhancing Embedding Performance through Large Language Model-based Text Enrichment and Rewriting

Embedding models are crucial for various natural language processing tasks but can be limited by factors such as limited vocabulary, lack of context, and grammatical errors. This paper proposes a novel approach to improve embedding performance by leveraging Large Language Models (LLMs) to enrich and rewrite input text before the embedding process. By utilizing ChatGPT 3.5 to provide additional context, correct inaccuracies, and incorporate metadata, the proposed method aims to enhance the utility and accuracy of embedding models. The effectiveness of this approach is evaluated on three datasets: Banking77Classification, TwitterSemEval 2015, and Amazon Counter-factual Classification. The results demonstrate significant improvements over the baseline model on the TwitterSemEval 2015 dataset, with the best-performing prompt achieving an average precision based on cosine similarity score of 85.34 compared to the previous best of 81.52 on the Massive Text Embedding Benchmark (MTEB) Leaderboard. However, performance on the other two datasets i.e. Banking77Classification and Amazon Counter Factual was less impressive. The findings suggest that LLM-based text enrichment has shown promising results to improve embedding performance, particularly in certain domains. Hence, numerous limitations in the process of embedding can be avoided.

Word Level Language Identification in Indonesian-Javanese-English Code-Mixed Text

Code-mixing is a prevalent phenomenon in contemporary communication, particularly in social media interactions. In Indonesia, code-mixing is common, with languages such as Indonesian, Javanese, and English frequently mixed in everyday conversation. Accurate language identification at the word level is important for various downstream natural language processing tasks, such as sentiment analysis and translation. This study investigates machine learning techniques for word-level language identification in Indonesian-Javanese-English code-mixed texts. We compared three categories of machine learning models: traditional machine learning, BLSTM-based architectures, and transformer-based pre-trained models. Our experiments demonstrate that fine-tuning pre-trained models achieves strong performance, with XLM-RoBERTa and IndoBERTweet achieving the best F1 scores of 93.69% and 93.63%, respectively. Our results highlight the effectiveness of fine-tuning pre-trained models for language identification tasks, demonstrating their ability to comprehend sentence contexts and produce accurate label predictions.

CVE Severity Prediction From Vulnerability Description - A Deep Learning Approach

The Common Vulnerabilities and Exposures (CVE) system is a widely used standard for identifying and tracking known vulnerabilities in software systems. The severity of these vulnerabilities must be determined in order to prioritize mitigation efforts. However, assigning severity to a vulnerability is a challenging task that requires careful analysis of its characteristics and potential impact. Considering the vast number of vulnerabilities identified every year, it is vital to automate the severity assignment, thereby reducing manual effort. This paper proposes a novel approach for predicting the severity of vulnerabilities based on their CVE description using GPT-2, a state-of-the-art language model. The CVSS severity values distribution imbalance is addressed using oversampling and contextual data augmentation techniques. This approach leverages the large-scale language modeling capabilities of GPT-2 to automatically extract relevant features from CVE descriptions and predict the severity level of the vulnerability. The model is evaluated on a test data set of 7,765 CVEs and achieves a high accuracy of 84.2% and an F1 score of 0.82 in predicting the severity of the vulnerabilities on the test data. A comparative analysis of this approach was done against state-of-the-art methods, demonstrating the superior performance of the proposed approach. Based on the results, the proposed approach could be considered a valuable tool for quickly and accurately identifying high-severity vulnerabilities, facilitating more efficient and effective vulnerability management practices. Furthermore, this approach could be extended to other natural language processing tasks related to vulnerability analysis and management.

Exploring elliptic geometry in neural networks : The concept of elliptic neural networks (ENN)

The exploration of non-Euclidean geometries in the context of neural network architectures presents a novel avenue for enhancing the processing of complex data structures. This paper introduces the concept of Elliptic Neural Networks (ENN), a framework that integrates the intrinsic properties of elliptic geometry into the fabric of neural computation. Characterized by constant positive curvature and closed geodesic paths, elliptic geometry provides a unique perspective for representing data, especially advantageous for configurations exhibiting cyclic and dense hierarchical interconnections. We begin by delineating the fundamental aspects of elliptic geometry, focusing on its impact on conventional notions of distance and global structural interpretation within data sets. This foundational understanding paves the way for our proposed mathematical schema for ENNs. Within this framework, we articulate the methodologies for embedding data in elliptic spaces and adapt neural network operations by encompassing neuron activation, signal propagation, and learning paradigms to conform to the topological idiosyncrasies of elliptic geometry. In this paper addressing the implementation of ENNs, we discuss the computational challenges and the development of novel algorithms requisite for navigating the elliptic geometric landscape. The prospective applications of ENNs are extensive and diverse, ranging from the enhancement of cyclic pattern recognition in time-series analysis to more sophisticated representations in graph-based data and natural language processing tasks. This theoretical exposition aims to set the groundwork for subsequent empirical research to validate the proposed model and assess its practical performance relative to conventional neural network architectures. By harnessing the distinctive characteristics of elliptic geometry, ENNs mark a significant stride towards enriching the arsenal of machine learning methodologies, potentially leading to more versatile and efficacious computational tools in data analysis and artificial intelligence.

Bidirectional Gated Recurrent Unit with Glove Embedding and Attention Mechanism for Movie Review Classification

During the current Internet era, enormous volume of structured and unstructured textual data is generated and exchanged online that made the text classification more crucial. With the massive amount of user-generated content on social media platforms, businesses and organizations can leverage sentiment analysis to understand customer opinions and attitudes towards the products, services, or brand. Text classification is a natural language processing task in which a machine learning model is trained to categorize the text into predefined classes or categories. Due to the advent of promising Deep Learning techniques, intelligent and accurate text classification system can be developed. In movie review classification, algorithms are used to automatically categorize movie reviews into positive, negative, or sometimes neutral sentiments based on the opinions expressed in the text. The goal is to automate the process of determining whether a review reflects a positive or negative sentiment, helping users quickly understand the overall reception of a movie. This work proposes an automated movie review classification system based on Bidirectional Gated Recurrent Unit encoder with attention module. This method encodes the words using GloVe word embedding technique. Context information is captured in an efficient manner and the encoding is more reliable. Bidirectional GRU layer captures long-term dependencies in sentences, followed by a custom Attention layer that assigns higher weights to key components. The attention mechanism allows the model to selectively focus on important parts of the input sequence, by dynamically determining attention weights based on task relevance. The model was individually trained and evaluated using popular benchmark IMDb dataset. Experimental studies have proved the better performance of proposed movie review classification system that obtained accuracy of 98%.

Binary or Graded, Few-Shot or Zero-Shot: Prompt Design for GPTs in Relevance Evaluation

Evaluating the relevance between a query and a passage is a pivotal task in Information Retrieval (IR). Utilizing such relevance evaluations can assist in ranking as well as in the creation of datasets for training and testing. The recent advancements in Large Language Models (LLMs) like GPT-4 have contributed to performance enhancements across many natural language processing tasks. Specifically, in the IR domain, many studies are being conducted on tasks related to relevance judgment, showing notable improvements. However, the efficacy of LLMs is considerably influenced by the design of the prompt. Despite this significance, there is a lack of research on prompts specifically tailored for relevance evaluation. The proposed prompts for this evaluation can be categorized based on how they distinguish relevance (binary or graded) and their reliance on in-context examples (few-shot or zero-shot). In this study, we experimentally investigate these two dimensions to determine which configurations are most advantageous for relevance evaluation. Our findings, based on the GPT-4 model, demonstrate that graded prompts in a zero-shot format are more effective.

Text Correction for Modern Standard Arabic

Arabic poses a unique challenge for Natural Language Processing tasks due to its morphological complexity, rich vocabulary, and syntactic flexibility. Mistakes written in Arabic are common even among fluent speakers, and slight mistakes can obstruct a word's meaning. Our project investigates Large Language Models (LLMs)’ capabilities for detecting and automatically correcting syntax and semantic errors in Arabic text. Our project includes an overview of Qatar Arabic Language Bank (QALB), a shared task on automatic correction of Arabic text, which focuses on correcting errors in Arabic text produced by native speakers. We used the QALB dataset for training and evaluation, achieving a WER score of 0.2203 and a GLEU of 0.5956.

A morphological word embedding for Arabic language

Word embedding is a technique for representing words as dense real vectors and is crucial for most natural language processing tasks. Common approaches include non-contextual embeddings such as Word2vec and contextual embeddings like BERT, both of which operate on the principle that words appearing in similar contexts have close vector representations. Despite their effectiveness, these methods often fail to capture some morphological information useful in many natural language processing applications. In this study, we propose an original word embedding of Arabic words based on the morphological characteristics of the words. Tests show that integrating this morphology-based word embedding into a topic detection model outperforms models using Word2vec or AraBERT embeddings.

ChatGPT’nin kadınlarda alt üriner sistem semptomları ile ilgili soruları yanıtlama performansının değerlendirilmesi

OBJECTIVE: ChatGPT is a language model developed by OpenAI. As a language model, ChatGPT can be used for various natural language processing tasks, such as answering questions, providing information, engaging in conversations, and more. We aimed to evaluate ChatGPT’s suitability to accurately inform patients and medical students by analyzing the reliability and accuracy of lower urinary tract symptoms (LUTS) information in women. MATERIAL and METHODS: Systematically organized frequently asked questions from urology association websites, hospitals, and social media platforms regarding lower urinary tract symptoms (LUTS) in women, along with questions compiled from European Association of Urology (EAU) recommendations, were analyzed. All questions were systematically submitted to ChatGPT 3.5 and 4.0 versions. Two expert urologists independently evaluated the data and answers to the questions and assigned scores from 1 to 4 to each answer. RESULTS: A hundred and three frequently asked questions and 36 questions compiled from EAU guideline recommendations regarding LUTS in women were included in the study. 83.50% of the frequently asked questions (85.44% for version 4.0) and 80.56% of the questions compiled from EAU recommendations (80.56% for version 4.0) were rated as ‘completely correct’ (Grade 1). The rate of completely incorrect answers to the questions (Grade 4) was 0.97% in the frequently asked questions group (0.97% for version 4.0) and 2.78% in the guide-based questions group (2.78% for version 4.0). The success rate of both versions in answering questions was found to be close to each other. CONCLUSION: ChatGPT provided acceptable responses to the posed questions. Despite its limitations, we believe that utilizing this continuously evolving platform under the guidance of urologists in urology clinics will contribute to the diagnosis and treatment processes for patients and their families. Keywords: artificial intelligence, ChatGPT, education, female LUTS

CurieLM: Enhancing Large Language Models for Nuclear Domain Applications

Large Language Models (LLMs), such as the Mistral model, have exhibited remarkable performance across diverse tasks. However, their efficacy in nuclear applications remains constrained by a lack of domain-specific knowledge and an inability to effectively leverage that knowledge. Nuclear-related tasks, including safety assessments and requirement analyses, pose unique challenges due to the intricate domain expertise and diverse constraints involved. To address these limitations, we introduce CurieLM, an LLM specifically tailored for the nuclear domain. CurieLM builds upon the Mistral model, enhancing its capabilities through domain-specific fine-tuning. Our team of nuclear engineers overcame the initial hurdle of accessing high-quality nuclear data, enabling CurieLM to comprehend and accurately respond to nuclear-specific instructions. This manuscript outlines the development and optimization process of CurieLM, marking a significant step toward enhancing nuclear-related natural language processing tasks. Experimental results demonstrate a 13% performance improvement over base LLMs, underscoring the effectiveness of our approach. Domain-specific LLMs like CurieLM hold a great potential across various applications, and this study sets the stage for further exploration in this emerging field.