Challenges In Natural Language Processing Research Articles

Speech-Based Techniques for Emotion Detection in Natural Arabic Audio Files

Emotion detection is one of the greatest challenges of Natural Language Processing (NLP). Often referred to as emotion recognition, it is the process of identifying a person’s various feelings or emotions such as: happiness, sadness, or anger. Emotions are a strong feeling regarding a human's situation or relation with others. They are the mental states that affect human behavior and interactions. In this paper, we propose an approach for emotion detection in audio files, focusing on a natural Arabic audio dataset and applying several Machine Learning (ML) classifiers: Sequential Minimal Optimization (SMO), Random Forest (RF), K-Nearest Neighbours (KNN), and Simple Logistic (SL). The classification experiments were conducted using sixteen acoustic feature sets. Many acoustic features were explored including Mel Frequency Cepstral Coefficient (MFCC), Mel spectrogram, spectral contrast, Zero Crossing Rate (ZCR), and Intensity. The experimental results show that SMO and SL classifiers achieved the highest overall accuracy 83.82% when using combinations of all acoustic features (MFCC, Mel spectrogram, Spectral contrast, ZCR and intensity). Additionally, The RF and KNN classifiers yielded Competitive results, with accuracies of 81.71% and 77.34%, respectively. These results suggest that combining multiple acoustic features significantly enhances the performance of emotion detection models, especially for complex emotions in natural Arabic audio datasets

Review of Language Structures and NLP Techniques for Chinese, Japanese, and English

The increasing demand for cross-lingual communication in a globally interconnected world has spurred significant advancements in Natural Language Processing (NLP), a branch of artificial intelligence aimed at enabling computers to comprehend, generate, and process human language. Over the past few years, NLP has transitioned from traditional rule-based and statistical approaches to deep learning-based methods, with pre-trained models like BERT and GPT demonstrating remarkable performance across a variety of tasks, including text classification and machine translation. However, the linguistic diversity of the world's languages presents ongoing challenges, as many NLP models are primarily developed for high-resource languages like English, leaving other languages underserved. This paper examines the linguistic features of three representative languagesChinese, Japanese, and Englishand explores NLP methodologies tailored to their distinct grammatical, lexical, and structural properties. Through comparative linguistic analysis, this study discusses the challenges of multilingual NLP and the potential for cross-lingual applications. The findings underscore the importance of adapting NLP models to different language characteristics to enhance the effectiveness and inclusivity of cross-lingual processing.

IndoGovBERT: A Domain-Specific Language Model for Processing Indonesian Government SDG Documents

Achieving the Sustainable Development Goals (SDGs) requires collaboration among various stakeholders, particularly governments and non-state actors (NSAs). This collaboration results in but is also based on a continually growing volume of documents that needs to be analyzed and processed in a systematic way by government officials. Artificial Intelligence and Natural Language Processing (NLP) could, thus, offer valuable support for progressing towards SDG targets, including automating the government budget tagging and classifying NSA requests and initiatives, as well as helping uncover the possibilities for matching these two categories of activities. Many non-English speaking countries, including Indonesia, however, face limited NLP resources, such as, for instance, domain-specific pre-trained language models (PTLMs). This circumstance makes it difficult to automate document processing and improve the efficacy of SDG-related government efforts. The presented study introduces IndoGovBERT, a Bidirectional Encoder Representations from Transformers (BERT)-based PTLM built with domain-specific corpora, leveraging the Indonesian government’s public and internal documents. The model is intended to automate various laborious tasks of SDG document processing by the Indonesian government. Different approaches to PTLM development known from the literature are examined in the context of typical government settings. The most effective, in terms of the resultant model performance, but also most efficient, in terms of the computational resources required, methodology is determined and deployed for the development of the IndoGovBERT model. The developed model is then scrutinized in several text classification and similarity assessment experiments, where it is compared with four Indonesian general-purpose language models, a non-transformer approach of the Multilabel Topic Model (MLTM), as well as with a Multilingual BERT model. Results obtained in all experiments highlight the superior capability of the IndoGovBERT model for Indonesian government SDG document processing. The latter suggests that the proposed PTLM development methodology could be adopted to build high-performance specialized PTLMs for governments around the globe which face SDG document processing and other NLP challenges similar to the ones dealt with in the presented study.

Disambiguating Clinical Abbreviations by One-to-All Classification: Algorithm Development and Validation Study.

Electronic medical records store extensive patient data and serve as a comprehensive repository, including textual medical records like surgical and imaging reports. Their utility in clinical decision support systems is substantial, but the widespread use of ambiguous and unstandardized abbreviations in clinical documents poses challenges for natural language processing in clinical decision support systems. Efficient abbreviation disambiguation methods are needed for effective information extraction. This study aims to enhance the one-to-all (OTA) framework for clinical abbreviation expansion, which uses a single model to predict multiple abbreviation meanings. The objective is to improve OTA by developing context-candidate pairs and optimizing word embeddings in Bidirectional Encoder Representations From Transformers (BERT), evaluating the model's efficacy in expanding clinical abbreviations using real data. Three datasets were used: Medical Subject Headings Word Sense Disambiguation, University of Minnesota, and Chia-Yi Christian Hospital from Ditmanson Medical Foundation Chia-Yi Christian Hospital. Texts containing polysemous abbreviations were preprocessed and formatted for BERT. The study involved fine-tuning pretrained models, ClinicalBERT and BlueBERT, generating dataset pairs for training and testing based on Huang et al's method. BlueBERT achieved macro- and microaccuracies of 95.41% and 95.16%, respectively, on the Medical Subject Headings Word Sense Disambiguation dataset. It improved macroaccuracy by 0.54%-1.53% compared to two baselines, long short-term memory and deepBioWSD with random embedding. On the University of Minnesota dataset, BlueBERT recorded macro- and microaccuracies of 98.40% and 98.22%, respectively. Against the baselines of Word2Vec + support vector machine and BioWordVec + support vector machine, BlueBERT demonstrated a macroaccuracy improvement of 2.61%-4.13%. This research preliminarily validated the effectiveness of the OTA method for abbreviation disambiguation in medical texts, demonstrating the potential to enhance both clinical staff efficiency and research effectiveness.

Scientific landscape on opportunities and challenges of large language models and natural language processing

Scientific landscape on opportunities and challenges of large language models and natural language processing

Fast Hybrid Approach for Thai News Summarization

News summarization presents a significant challenge in Natural Language Processing (NLP). Lengthy news articles not only consume valuable time but also lead to confusion regarding key points. The ideal news summarization should swiftly produce a succinct summary while retaining the essence of the information conveyed by the news writer. While intelligent chatbots like ChatGPT and Gemini offer user-friendly text summarization, their embedded Large Language Model (LLM) cannot be downloaded for private use. Moreover, implementing them in a business process can be expensive, both in terms of pay-per-use costs and response time. The objective of this research is to develop a private Thai news summarization that effectively extracts sentences encapsulating the main idea and abstractly summarizes them. The proposed model consists of two components. The first extracts a contiguous region containing important sentences using the TextRank algorithm, while the second employs the finetuned mBART as an LLM to generate the abstractive summary from the previously extracted sentences. In other words, the proposed model extracts an important news region before passing it to mBART. This approach produces a news summary with key information and a syntactic style akin to the natural Thai language. We evaluate the summarization quality by ROUGE scores and BERTScore (precision, recall, and F1-score). The evaluation metrics Experimental results on the ThaiSum dataset show relatively high ROUGE scores and BERTScore for the proposed model compared to most of the other approaches. Furthermore, it significantly reduces the runtime, keeping it within a reasonable limit.

Extracting Features from Text Flows based on Semantic Similarity for Text Classification: an Approach Inspired by Audio Analysis

Text classification is a mainly investigated challenge in Natural Language Processing (NLP) research. The higher performance of a classification model depends on a representation that can extract valuable information about the texts. Aiming not to lose crucial local text information, a way to represent texts is through flows, sequences of information collected from texts. This paper proposes an approach that combines various techniques to represent texts: the representation by flows, the benefit of the word embeddings text representation associated with lexicon information via semantic similarity distances, and the extraction of features inspired by well-established audio analysis features.In order to perform text classification, this approach splits the text into sentences and calculates a semantic similarity metric to a lexicon on an embedding vector space. The sequence of semantic similarity metrics composes the text flow. Then, the method performs the extraction of twenty-five features inspired by audio analysis (named Audio-Like Features). The features adaptation from audio analysis comes from a similitude between a text flow and a digital signal, in addition to the existing relationship between text, speech, and audio. We evaluated the method in three NLP classification tasks: Fake News Detection in English, Fake News Detection in Portuguese, and Newspaper Columns versus News Classification. The approach efficacy is compared to baselines that embed semantics in text representation: the Paragraph Vector and the BERT. The objective of the experiments was to investigate if the proposed approach could compete with the baselines methods improve their efficacy when associated with them. The experimental evaluation demonstrates that the association between the proposed and the baseline methods can enhance the baseline classification efficacy in all three scenarios. In the Fake News Detection in Portuguese task, our approach surpassed the baselines and obtained the best effectiveness (PR-AUC = 0.98).

Text-to-text generative approach for enhanced complex word identification

This paper presents a novel approach for solving the Complex Word Identification (CWI) task using the text-to-text generative model. The CWI task involves identifying complex words in text, which is a challenging Natural Language Processing task. To our knowledge, it is a first attempt to address CWI problem into text-to-text context. In this work, we propose a new methodology that leverages the power of the Transformer model to evaluate complexity of words in binary and probabilistic settings. We also propose a novel CWI dataset, which consists of 62,200 phrases, both complex and simple. We train and fine-tune our proposed model on our CWI dataset. We also evaluate its performance on separate test sets across three different domains. Our experimental results demonstrate the effectiveness of our proposed approach compared to state-of-the-art methods.

Towards explainable fake news detection and automated content credibility assessment: Polish internet and digital media use-case

Detecting fake news and disinformation is a challenging and complex task. With the proliferation of the transformer architectures, researchers and practitioners have started actively using them to combat this phenomenon. In that regard, different approaches have been proposed by the community. Unfortunately, a significant number of solutions have started to treat misinformation in a similar way as other natural language processing challenges (e.g., sentiment classification). However, the reality is more complex, and the fake news detection problem often require additional context that may be essential to provide a complete assessment. Another pitfall in combating fake news is treating it as a binary classification challenge (e.g., fake vs. legitimate news). Combining such a simplified response with a deep black-box neural system effectively impacts the interpretability and eventual trustworthiness of the detection system. In this paper, the phenomenon of fake news is approached from a multi-factor perspective. Therefore, the proposed innovative approach does not limit news content rating to a one-dimensional binary response. In particular, the authors utilize a vast spectrum of processing techniques, including text embedding, machine translation, and various users’ preferences to provide content assessment. The results are presented using a Polish digital media use-case.

Chimp Optimization Algorithm with Deep Learning-Driven Fine-grained Emotion Recognition in Arabic Corpus

Recently, emotion analysis and classification of tweets have become a crucial area of research. The Arabic language had experienced difficulties with emotion classification on Twitter(X), needing preprocessing more than other languages. Emotion detection is a major challenge in Natural Language Processing (NLP), which allows machines to ascertain the emotions expressed in the text. The task includes recognizing and identifying human feelings such as fear, anger, sadness, and joy. The discovered sentiments and feelings expressed in tweets have gained much recognition in recent years. The Arab region has played a substantial role in international politics and the global economy needs to scrutinize the emotions and sentiments in the Arabic language. Lexicon-based and machine-learning techniques are two common models that address the problems of emotion classification. This study introduces a Chimp Optimization Algorithm with a Deep Learning-Driven Arabic Fine-grained Emotion Recognition (COADL-AFER) technique. The presented COADL-AFER technique mainly aims to detect several emotions in Arabic tweets. In addition to its academic significance, the COADL-AFER technique has practical applications in various fields, including enhancing applications of E-learning, aiding psychologists in recognising terrorist performance, improving product quality, and enhancing customer service. The COADL-AFER technique applies the long short-term memory (LSTM) model for emotion detection. Finally, the hyperparameter selection of the LSTM method can be accomplished by COA. The experimental validation of the COADL-AFER system, a crucial step in our research, is verified utilizing the Arabic tweets dataset. The simulation results stated the betterment of the COADL-AFER technique, further reinforcing the reliability of our research.

Navigating the currents of natural language processing: A comprehensive overview of modern techniques and applications

Natural Language Processing (NLP) has been a trend over the years, and many young researchers are inspired to study this field. In this paper, the author conducts a comprehensive overview of NLP from rule-based systems to machine learning techniques. Besides, the core principles of NLP and the impact of deep learning architectures, such as Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and Transformers, are explored. Additionally, this paper delves into significant NLP tasks, for instance, text classification, sentiment analysis, and machine translation. At the same time, the ethical considerations and challenges in NLP, including issues related to data bias and privacy, are also discussed. As a result, this research concludes that NLP is versatile and valuable, but it also needs to overcome challenges related to language ambiguity, data bias, privacy concerns, and so on. The ethical use of AI is necessary in order to ensure progress in NLP in the future.

Artificial intelligence and management education: A conceptualization of human-machine interaction

The increasing use of Advanced Natural Language Processing (ANLP) models, particularly ChatGPT-4, presents opportunities and challenges to management education and research. These models can enhance the style, creativity, and analytical power of research papers, potentially shifting human scholars' roles from creators to ‘prompters’. If machines can perform educational and research tasks more effectively the role of human educators becomes a salient question in a world in which ANLP models offer clear, coherent, and polished insights, the use of which has potentially paradoxical possibilities. From one perspective, a new type of high-quality scholarship and education characterized by strong human involvement that synergistically leverages ANLP models' analytical capabilities, enabling human scholars to probe complex phenomena and make management research truly meaningful and impactful for broader audiences, is possible. We explore these questions through an ‘ideal type’ conceptualization of the possible relations between AI and management education and research.

STVANet: A spatio-temporal visual attention framework with large kernel attention mechanism for citywide traffic dynamics prediction

Enhancing the efficiency and safety of the Intelligent Transportation System requires effective modeling and prediction of citywide traffic dynamics. Most studies employ convolutional neural networks (CNNs) with a 3D convolutional structure or spatio-temporal models with self-attention mechanisms to capture the spatio-temporal information of traffic distribution. Although 3D CNNs excel at capturing local contextual information, they are computationally complex due to the large number of parameters and cannot capture long-range dependence. By contrast, although self-attention mechanisms originally designed to address challenges in natural language processing can capture long-range dependence, their application to 2D image structures requires breaking down the inherent 2D context into a 1D sequence, increasing the computational complexity and neglecting the adaptability between local contextual information and channels. Accordingly, we propose a spatio-temporal visual attention neural network (STVANet), a novel spatio-temporal visual attention 2D CNN, which integrates a unique visual attention module with a large kernel attention (LKA) mechanism, a squeeze-and-excitation (SE) mechanism and a feedforward component to capture long-range dependence and channel information in urban traffic data while preserving the 2D image structure. LKA-based spatio-temporal attention networks extract spatial and temporal features from weekly, daily, and recent hourly periods, and aggregate them with weighted consideration of external features to make predictions. Evaluation of real-world datasets demonstrates STVANet’s superiority over baseline models, showcasing its potential in citywide traffic prediction.

Comparative Analysis of Deep Learning Models for Part of Speech Tagging in the Malay Language

Despite the widespread use of Malay, under-resourced languages like Malay face challenges in Natural Language Processing (NLP), particularly in Part-of-Speech (POS) tagging. The scarcity of annotated corpora poses a primary obstacle to POS tagging in Malay. This study aims to enhance the effectiveness and reliability of POS tagging models explicitly tailored for under-resourced languages within the field of NLP, focusing on Malay. Existing models, which rely on Conditional Random Fields and Hidden Markov Models, exhibit limitations, underscoring the need for more robust approaches. The research conducts a comparative analysis of various deep-learning models with different encoders for POS tagging in Malay sentences. The experimental analysis demonstrates that the Bidirectional Long Short-Term Memory (Bi-LSTM) model, leveraging a pre-trained Bidirectional Encoder Representations from Transformers (BERT) embedding model, achieves exceptional accuracy, precision, recall, and F1 scores in predicting tags. Notably, the BERT + Bi-LSTM model, boasting an accuracy of 98.82%, outperforms other models, showcasing superior performance across all evaluated metrics. Additionally, this combined model effectively handles known and unknown words, yielding highly accurate POS tagging results for Malay sentences. Doi: 10.28991/HIJ-2024-05-02-04 Full Text: PDF

Comprehensive analysis of natural language processing

Natural Language Processing (NLP) is a fascinating field of study that teaches computers to understand and use human language. This means that computers can read, write, and even translate text just like humans. NLP has many practical uses, such as categorizing text, identifying the tone of language, recognizing names in text, translating languages, and answering questions. NLP has come a long way since it was first developed. In the past, it relied on strict rules to understand language, but now it uses advanced techniques like machine learning and deep learning to understand text. However, there are still some challenges in NLP, such as understanding the meaning of words in context and considering cultural differences. Despite these challenges, NLP is being used in many different areas, from healthcare and finance to education and customer service. NLP is transforming the way humans interact with computers and is making it easier to extract important information from large amounts of text.

Language Threshold for Multilingual Sentiment Analysis System

Code-mixed sentences are very common in social media platforms especially in countries such as Malaysia that have more than 2 speaking languages. Although multilingual Bidirectional Encoder Representations from Transformers (mBERT) has the capability of understanding multilingualism, the sentence embeddings obtained from mBERT can be very complex for a code-mixed sentence. This is a challenge in Natural Language processing when processing informal social media text due to its complexity, especially in mixed languages like Malay-English where there is an insufficient amount of training datasets available. Thus, this paper proposes a language threshold to translate the affected words or sentence into a single language sentence and relabel the language of the sentence. The result shows an increase of 8% in accuracy when translating affected words in a sentence at the 60% language threshold using SEC PCA-200.

Large language models for biomedicine: foundations, opportunities, challenges, and best practices.

Generative large language models (LLMs) are a subset of transformers-based neural network architecture models. LLMs have successfully leveraged a combination of an increased number of parameters, improvements in computational efficiency, and large pre-training datasets to perform a wide spectrum of natural language processing (NLP) tasks. Using a few examples (few-shot) or no examples (zero-shot) for prompt-tuning has enabled LLMs to achieve state-of-the-art performance in a broad range of NLP applications. This article by the American Medical Informatics Association (AMIA) NLP Working Group characterizes the opportunities, challenges, and best practices for our community to leverage and advance the integration of LLMs in downstream NLP applications effectively. This can be accomplished through a variety of approaches, including augmented prompting, instruction prompt tuning, and reinforcement learning from human feedback (RLHF). Our focus is on making LLMs accessible to the broader biomedical informatics community, including clinicians and researchers who may be unfamiliar with NLP. Additionally, NLP practitioners may gain insight from the described best practices. We focus on 3 broad categories of NLP tasks, namely natural language understanding, natural language inferencing, and natural language generation. We review the emerging trends in prompt tuning, instruction fine-tuning, and evaluation metrics used for LLMs while drawing attention to several issues that impact biomedical NLP applications, including falsehoods in generated text (confabulation/hallucinations), toxicity, and dataset contamination leading to overfitting. We also review potential approaches to address some of these current challenges in LLMs, such as chain of thought prompting, and the phenomena of emergent capabilities observed in LLMs that can be leveraged to address complex NLP challenge in biomedical applications.

Adapting transformer-based language models for heart disease detection and risk factors extraction

Efficiently treating cardiac patients before the onset of a heart attack relies on the precise prediction of heart disease. Identifying and detecting the risk factors for heart disease such as diabetes mellitus, Coronary Artery Disease (CAD), hyperlipidemia, hypertension, smoking, familial CAD history, obesity, and medications is critical for developing effective preventative and management measures. Although Electronic Health Records (EHRs) have emerged as valuable resources for identifying these risk factors, their unstructured format poses challenges for cardiologists in retrieving relevant information. This research proposed employing transfer learning techniques to automatically extract heart disease risk factors from EHRs. Leveraging transfer learning, a deep learning technique has demonstrated a significant performance in various clinical natural language processing (NLP) applications, particularly in heart disease risk prediction. This study explored the application of transformer-based language models, specifically utilizing pre-trained architectures like BERT (Bidirectional Encoder Representations from Transformers), RoBERTa, BioClinicalBERT, XLNet, and BioBERT for heart disease detection and extraction of related risk factors from clinical notes, using the i2b2 dataset. These transformer models are pre-trained on an extensive corpus of medical literature and clinical records to gain a deep understanding of contextualized language representations. Adapted models are then fine-tuned using annotated datasets specific to heart disease, such as the i2b2 dataset, enabling them to learn patterns and relationships within the domain. These models have demonstrated superior performance in extracting semantic information from EHRs, automating high-performance heart disease risk factor identification, and performing downstream NLP tasks within the clinical domain. This study proposed fine-tuned five widely used transformer-based models, namely BERT, RoBERTa, BioClinicalBERT, XLNet, and BioBERT, using the 2014 i2b2 clinical NLP challenge dataset. The fine-tuned models surpass conventional approaches in predicting the presence of heart disease risk factors with impressive accuracy. The RoBERTa model has achieved the highest performance, with micro F1-scores of 94.27%, while the BERT, BioClinicalBERT, XLNet, and BioBERT models have provided competitive performances with micro F1-scores of 93.73%, 94.03%, 93.97%, and 93.99%, respectively. Finally, a simple ensemble of the five transformer-based models has been proposed, which outperformed the most existing methods in heart disease risk fan, achieving a micro F1-Score of 94.26%. This study demonstrated the efficacy of transfer learning using transformer-based models in enhancing risk prediction and facilitating early intervention for heart disease prevention.

Waste Pollution Classification in Indonesian Language using DistilBERT

In Indonesia, waste pollution poses pressing environmental and health challenges, making accurate classification vital for targeted mitigation efforts. DistilBERT emerges as a streamlined counterpart to the acclaimed BERT architecture, designed to mirror BERT's advanced linguistic comprehension but with reduced computational demands. By leveraging the essence of transfer learning, DistilBERT benefits from a wealth of information obtained from extensive textual datasets, positioning it as an ideal choice for scenarios marked by limited data accessibility. In our research, we adopted DistilBERT to address the niche challenge of classifying waste types using a constrained dataset derived from Twitter conversations in Indonesian language—a medium notorious for its concise and often ambiguous content. Notwithstanding the dataset's restricted scope and the noise inherent to Twitter, DistilBERT demonstrated an astounding efficacy, registering a precision rate of 98%. This outcome accentuates DistilBERT's capability to navigate and discern complex textual nuances even in data-restricted environments and further highlights the significance of transfer learning in contemporary natural language processing challenges, especially in contexts as critical as Indonesia's waste management efforts

Challenges of Natural Language Processing from a Linguistic Perspective

As the foundation of artificial intelligence, natural language processing is a research field with great development prospects, closely related to linguistics. Linguistic research has realized the initial natural language processing, and the development of natural language processing has raised higher requirements for linguistic research. Although natural language processing technology has achieved some results, there is still a distance from the most natural human-machine interaction. Language is the expression form of human thinking, and the fundamental purpose of artificial intelligence is to simulate human thinking with computers. To achieve this goal, it is necessary to start with simulating human natural language. This paper mainly analyzes the current challenges of natural language processing from a linguistic perspective, aiming to further improve the level of natural language processing.