Bidirectional Encoder Representations Research Articles

Fine-Tuned Bidirectional Encoder Representations From Transformers Versus ChatGPT for Text-Based Outpatient Department Recommendation: Comparative Study.

Patients often struggle with determining which outpatient specialist to consult based on their symptoms. Natural language processing models in health care offer the potential to assist patients in making these decisions before visiting a hospital. This study aimed to evaluate the performance of ChatGPT in recommending medical specialties for medical questions. We used a dataset of 31,482 medical questions, each answered by doctors and labeled with the appropriate medical specialty from the health consultation board of NAVER (NAVER Corp), a major Korean portal. This dataset includes 27 distinct medical specialty labels. We compared the performance of the fine-tuned Korean Medical bidirectional encoder representations from transformers (KM-BERT) and ChatGPT models by analyzing their ability to accurately recommend medical specialties. We categorized responses from ChatGPT into those matching the 27 predefined specialties and those that did not. Both models were evaluated using performance metrics of accuracy, precision, recall, and F1-score. ChatGPT demonstrated an answer avoidance rate of 6.2% but provided accurate medical specialty recommendations with explanations that elucidated the underlying pathophysiology of the patient's symptoms. It achieved an accuracy of 0.939, precision of 0.219, recall of 0.168, and an F1-score of 0.134. In contrast, the KM-BERT model, fine-tuned for the same task, outperformed ChatGPT with an accuracy of 0.977, precision of 0.570, recall of 0.652, and an F1-score of 0.587. Although ChatGPT did not surpass the fine-tuned KM-BERT model in recommending the correct medical specialties, it showcased notable advantages as a conversational artificial intelligence model. By providing detailed, contextually appropriate explanations, ChatGPT has the potential to significantly enhance patient comprehension of medical information, thereby improving the medical referral process.

Can the sentiment of the official media predict the return volatility of the Chinese crude oil futures?

This study investigates whether the sentiment of Chinese official media towards crude oil influences price volatility of the Chinese crude oil futures (SC). By leveraging textual analysis through Bidirectional Encoder Representations from Transformers (BERT), we quantify the sentiment of oil-related articles published by the primary official media in China. Our main finding, building on both in-sample and out-of-sample analyses, robustly reveals that this sentiment significantly forecasts the one-day-ahead intraday return volatility of SC. Moreover, we extend the analysis to different time horizons (i.e., one-week and one-month-ahead) and find the prominent forecasting power of the official media sentiment as well. We also find that the official media sentiment fails to forecast the price volatility of WTI oil futures, implying that the official media sentiment contains some unique Chinese information. Overall, our study contributes to the existing literature on predicting the return volatility of the Chinese crude oil futures, and offers fresh insights into an essential yet underexplored sentiment, i.e., official media sentiment.

GPT-Driven Radiology Report Generation with Fine-Tuned Llama 3.

The integration of deep learning into radiology has the potential to enhance diagnostic processes, yet its acceptance in clinical practice remains limited due to various challenges. This study aimed to develop and evaluate a fine-tuned large language model (LLM), based on Llama 3-8B, to automate the generation of accurate and concise conclusions in magnetic resonance imaging (MRI) and computed tomography (CT) radiology reports, thereby assisting radiologists and improving reporting efficiency. A dataset comprising 15,000 radiology reports was collected from the University of Medicine and Pharmacy of Craiova's Imaging Center, covering a diverse range of MRI and CT examinations made by four experienced radiologists. The Llama 3-8B model was fine-tuned using transfer-learning techniques, incorporating parameter quantization to 4-bit precision and low-rank adaptation (LoRA) with a rank of 16 to optimize computational efficiency on consumer-grade GPUs. The model was trained over five epochs using an NVIDIA RTX 3090 GPU, with intermediary checkpoints saved for monitoring. Performance was evaluated quantitatively using Bidirectional Encoder Representations from Transformers Score (BERTScore), Recall-Oriented Understudy for Gisting Evaluation (ROUGE), Bilingual Evaluation Understudy (BLEU), and Metric for Evaluation of Translation with Explicit Ordering (METEOR) metrics on a held-out test set. Additionally, a qualitative assessment was conducted, involving 13 independent radiologists who participated in a Turing-like test and provided ratings for the AI-generated conclusions. The fine-tuned model demonstrated strong quantitative performance, achieving a BERTScore F1 of 0.8054, a ROUGE-1 F1 of 0.4998, a ROUGE-L F1 of 0.4628, and a METEOR score of 0.4282. In the human evaluation, the artificial intelligence (AI)-generated conclusions were preferred over human-written ones in approximately 21.8% of cases, indicating that the model's outputs were competitive with those of experienced radiologists. The average rating of the AI-generated conclusions was 3.65 out of 5, reflecting a generally favorable assessment. Notably, the model maintained its consistency across various types of reports and demonstrated the ability to generalize to unseen data. The fine-tuned Llama 3-8B model effectively generates accurate and coherent conclusions for MRI and CT radiology reports. By automating the conclusion-writing process, this approach can assist radiologists in reducing their workload and enhancing report consistency, potentially addressing some barriers to the adoption of deep learning in clinical practice. The positive evaluations from independent radiologists underscore the model's potential utility. While the model demonstrated strong performance, limitations such as dataset bias, limited sample diversity, a lack of clinical judgment, and the need for large computational resources require further refinement and real-world validation. Future work should explore the integration of such models into clinical workflows, address ethical and legal considerations, and extend this approach to generate complete radiology reports.

Design of agricultural question answering information extraction method based on improved BILSTM algorithm

With the rapid growth of the agricultural information and the need for data analysis, how to accurately extract useful information from massive data has become an urgent first step in agricultural data mining and application. In this study, an agricultural question-answering information extraction method based on the BE-BILSTM (Improved Bidirectional Long Short-Term Memory) algorithm is designed. Firstly, it uses Python’s Scrapy crawler framework to obtain the information of soil types, crop diseases and pests, and agricultural trade information, and remove abnormal values. Secondly, the information extraction converts the semi-structured data by using entity extraction methods. Thirdly, the BERT (Bidirectional Encoder Representations from Transformers) algorithm is introduced to improve the performance of the BILSTM algorithm. After comparing with the BERT-CRF (Conditional Random Field) and BILSTM algorithm, the result shows that the BE-BILSTM algorithm has better information extraction performance than the other two algorithms. This study improves the accuracy of the agricultural information recommendation system from the perspective of information extraction. Compared with other work that is done from the perspective of recommendation algorithm optimization, it is more innovative; it helps to understand the semantics and contextual relationships in agricultural question and answer, which improves the accuracy of agricultural information recommendation systems. By gaining a deeper understanding of farmers’ needs and interests, the system can better recommend relevant and practical information.

Sequential lexicon enhanced bidirectional encoder representations from transformers: Chinese named entity recognition using sequential lexicon enhanced BERT

Lexicon Enhanced Bidirectional Encoder Representations from Transformers (LEBERT) has achieved great success in Chinese Named Entity Recognition (NER). LEBERT performs lexical enhancement with a Lexicon Adapter layer, which facilitates deep lexicon knowledge fusion at the lower layers of BERT. However, this method is likely to introduce noise words and does not consider the possible conflicts between words when fusing lexicon information. To address this issue, we advocate for a novel lexical enhancement method, Sequential Lexicon Enhanced BERT (SLEBERT) for the Chinese NER, which builds sequential lexicon to reduce noise words and resolve the problem of lexical conflict. Compared with LEBERT, it leverages the position encoding of sequential lexicon and adaptive attention mechanism of sequential lexicon to enhance the lexicon feature. Experiments on the four available datasets identified that SLEBERT outperforms other lexical enhancement models in performance and efficiency.

Evaluating Medical Entity Recognition in Health Care: Entity Model Quantitative Study.

Named entity recognition (NER) models are essential for extracting structured information from unstructured medical texts by identifying entities such as diseases, treatments, and conditions, enhancing clinical decision-making and research. Innovations in machine learning, particularly those involving Bidirectional Encoder Representations From Transformers (BERT)-based deep learning and large language models, have significantly advanced NER capabilities. However, their performance varies across medical datasets due to the complexity and diversity of medical terminology. Previous studies have often focused on overall performance, neglecting specific challenges in medical contexts and the impact of macrofactors like lexical composition on prediction accuracy. These gaps hinder the development of optimized NER models for medical applications. This study aims to meticulously evaluate the performance of various NER models in the context of medical text analysis, focusing on how complex medical terminology affects entity recognition accuracy. Additionally, we explored the influence of macrofactors on model performance, seeking to provide insights for refining NER models and enhancing their reliability for medical applications. This study comprehensively evaluated 7 NER models-hidden Markov models, conditional random fields, BERT for Biomedical Text Mining, Big Transformer Models for Efficient Long-Sequence Attention, Decoding-enhanced BERT with Disentangled Attention, Robustly Optimized BERT Pretraining Approach, and Gemma-across 3 medical datasets: Revised Joint Workshop on Natural Language Processing in Biomedicine and its Applications (JNLPBA), BioCreative V CDR, and Anatomical Entity Mention (AnatEM). The evaluation focused on prediction accuracy, resource use (eg, central processing unit and graphics processing unit use), and the impact of fine-tuning hyperparameters. The macrofactors affecting model performance were also screened using the multilevel factor elimination algorithm. The fine-tuned BERT for Biomedical Text Mining, with balanced resource use, generally achieved the highest prediction accuracy across the Revised JNLPBA and AnatEM datasets, with microaverage (AVG_MICRO) scores of 0.932 and 0.8494, respectively, highlighting its superior proficiency in identifying medical entities. Gemma, fine-tuned using the low-rank adaptation technique, achieved the highest accuracy on the BioCreative V CDR dataset with an AVG_MICRO score of 0.9962 but exhibited variability across the other datasets (AVG_MICRO scores of 0.9088 on the Revised JNLPBA and 0.8029 on AnatEM), indicating a need for further optimization. In addition, our analysis revealed that 2 macrofactors, entity phrase length and the number of entity words in each entity phrase, significantly influenced model performance. This study highlights the essential role of NER models in medical informatics, emphasizing the imperative for model optimization via precise data targeting and fine-tuning. The insights from this study will notably improve clinical decision-making and facilitate the creation of more sophisticated and effective medical NER models.

Zero-Shot Learning for Accurate Project Duration Prediction in Crowdsourcing Software Development

Crowdsourcing Software Development (CSD) platforms, i.e., TopCoder, function as intermediaries connecting clients with developers. Despite employing systematic methodologies, these platforms frequently encounter high task abandonment rates, with approximately 19% of projects failing to meet satisfactory outcomes. Although existing research has focused on task scheduling, developer recommendations, and reward mechanisms, there has been insufficient attention to the support of platform moderators, or copilots, who are essential to project success. A critical responsibility of copilots is estimating project duration; however, manual predictions often lead to inconsistencies and delays. This paper introduces an innovative machine learning approach designed to automate the prediction of project duration on CSD platforms. Utilizing historical data from TopCoder, the proposed method extracts pertinent project attributes and preprocesses textual data through Natural Language Processing (NLP). Bidirectional Encoder Representations from Transformers (BERT) are employed to convert textual information into vectors, which are then analyzed using various machine learning algorithms. Zero-shot learning algorithms exhibit superior performance, with an average accuracy of 92.76%, precision of 92.76%, recall of 99.33%, and an f-measure of 95.93%. The implementation of the proposed automated duration prediction model is crucial for enhancing the success rate of crowdsourcing projects, optimizing resource allocation, managing budgets effectively, and improving stakeholder satisfaction.

Domain knowledge-powered attention for air traffic management hazardous events classification

The automated classification of hazardous events in Air Traffic Management (ATM) involves concise texts with specialized terminology. The problem is currently addressed by machine learning text classification methods; however, some methods overlook domain knowledge, while others require it excessively. This research aims to identify a process for integrating low-cost, text-based domain knowledge into classification to address the issue of domain shift. To achieve this, the Wide and Deep Bidirectional Encoder Representations from Transformers (WD-BERT) model is proposed, featuring a unique knowledge-powered attention mechanism. WD-BERT's Wide Attention module assesses the compatibility between input text and domain knowledge to determine class probabilities, while its Deep Attention module extracts contextual features guided by domain knowledge. Additionally, a text mining method is employed to extract domain knowledge from ATM regulation texts. The model is trained and evaluated using a dataset derived from the China ATM Hazard Source Database. It achieves a multi-label classification accuracy of 80.24% and an F1-micro score of 0.88, outperforming comparative models with state-of-the-art performance. Breakdown analyses of the model indicate that the carefully designed attention mechanism endows WD-BERT with the ability to integrate domain knowledge, allowing it to excel in contexts involving complex sentences, multiple terms, and domain shifts. The innovation of this study lies in the proposed domain knowledge-powered attention mechanism, which allows simply organized domain knowledge to effectively guide ATM hazardous events classification, alleviating the impact of domain shift. This method can also be applied to other vertical domains without knowledge graphs.

Investigating the agenda of global warming on Twitter: A machine learning approach

This study investigates the agenda of global warming issue based on the two levels of agenda-setting framework. Using the Bidirectional Encoder Representations from Transformers (BERT)-based models, it identifies nine attributes of global warming issue and estimates the average tone of tweets in each month of 2021. The findings suggest the attributes are interconnected and partisan divides existed to a larger extent in certain attributes than in others. The nine attributes identified, their inherent interconnections as well as different tones associated with these attributes have the implications for policy-makers to formulate and evaluate policies regarding this multifaceted issue.

Applications of BERT in sentimental analysis

This research study emphasizes sentiment analysis and examines Natural Language Processing (NLP) by Bidirectional Encoder Representations from Transformers (BERT). BERT's bidirectional Transformer architecture pre-trained utilizes Next Sentence Prediction (NSP) and Masked Language Modeling (MLM) and has achieved a lot in terms of AI transformation. This paper provides a description of the BERT design, pre-training methods, and fine-tuning for sentiment analysis tasks. The study goes ahead and compares BERT's performance with other deep learning models, machine learning algorithms, and traditional rule-based techniques, highlighting the latter's limited ability to handle linguistic nuances and context. Additionally, studies proving the consistency and accuracy of BERT's sentiment analysis are examined, along with the challenges of handling irony, sarcasm, and domain-specific data. Ethical and privacy concerns that sentiment analysis inherently raises and makes recommendations for further research are also examined in the study, which also shows how integrating sentiment analysis with other domains can lead to multidisciplinary breakthroughs that can offer more comprehensive insights and applications.

Evaluating the Performance of Five Classifiers for Twitter Sentiment Analysis Using Bag of Words

Abstract: Twitter is a social medium for millions of people to express their opinions on different subjects. Natural Language Processing is widely used for sentiment analysis. This Research paper compares five machine learning classifier performances using a bag of words technique. A Kaggle Dataset is used. The Naïve Bayes classifier gave better predictions having an accuracy achieved to 91.59 percent for the four type sentiments from the text, Negative, Positive, Neutral, and Irrelevant. The classifiers used in this research are Logistic Regression, XGBoost, Naïve Bayes, Random Forest, and Support Vector Machine. The bag of words with having n-grams to four words and retaining all the stop words gave better accuracy than having n-grams to one word and removing all stop words. For further improvement in results Transformer based models like Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformers (GPT) must be taken into account. Deep learning models are more likely to give better accuracy than traditional machine learning models, and they use word embeddings rather than a bag of words technique to understand semantics better.

Joint intent detection and slot filling with syntactic and semantic features using multichannel CNN-BiLSTM

Understanding spoken language is crucial for conversational agents, with intent detection and slot filling being the primary tasks in natural language understanding (NLU). Enhancing the NLU tasks can lead to an accurate and efficient virtual assistant thereby reducing the need for human intervention and expanding their applicability in other domains. Traditionally, these tasks have been addressed individually, but recent studies have highlighted their interconnection, suggesting better results when solved together. Recent advances in natural language processing have shown that pretrained word embeddings can enhance text representation and improve the generalization capabilities of models. However, the challenge of poor generalization in joint learning models for intent detection and slot filling remains due to limited annotated datasets. Additionally, traditional models face difficulties in capturing both the semantic and syntactic nuances of language, which are vital for accurate intent detection and slot filling. This study proposes a hybridized text representation method using a multichannel convolutional neural network with three embedding channels: non-contextual embeddings for semantic information, part-of-speech (POS) tag embeddings for syntactic features, and contextual embeddings for deeper contextual understanding. Specifically, we utilized word2vec for non-contextual embeddings, one-hot vectors for POS tags, and bidirectional encoder representations from transformers (BERT) for contextual embeddings. These embeddings are processed through a convolutional layer and a shared bidirectional long short-term memory (BiLSTM) network, followed by two softmax functions for intent detection and slot filling. Experiments on the air travel information system (ATIS) and SNIPS datasets demonstrated that our model significantly outperformed the baseline models, achieving an intent accuracy of 97.90% and slot filling F1-score of 98.86% on the ATIS dataset, and an intent accuracy of 98.88% and slot filling F1-score of 97.07% on the SNIPS dataset. These results highlight the effectiveness of our proposed approach in advancing dialogue systems, and paving the way for more accurate and efficient natural language understanding in real-world applications.

Revealing the impact of social circumstances on the selection of cancer therapy through natural language processing of social work notes.

We aimed to investigate the impact of social circumstances on cancer therapy selection using natural language processing to derive insights from social worker documentation. We developed and employed a Bidirectional Encoder Representations from Transformers (BERT) based approach, using a hierarchical multi-step BERT model (BERT-MS), to predict the prescription of targeted cancer therapy to patients based solely on documentation by clinical social workers. Our corpus included free-text clinical social work notes, combined with medication prescription information, for all patients treated for breast cancer at UCSF between 2012 and 2021. We conducted a feature importance analysis to identify the specific social circumstances that impact cancer therapy regimen. Using only social work notes, we consistently predicted the administration of targeted therapies, suggesting systematic differences in treatment selection exist due to non-clinical factors. The findings were confirmed by several language models, with GatorTron achieving the best performance with an area under the receiver operating characteristic curve (AUROC) of 0.721 and a Macro F1 score of 0.616. The UCSF BERT-MS model, capable of leveraging multiple pieces of notes, surpassed the UCSF-BERT model in both AUROC and Macro-F1. Our feature importance analysis identified several clinically intuitive social determinants of health that potentially contribute to disparities in treatment. Leveraging social work notes can be instrumental in identifying disparities in clinical decision-making. Hypotheses generated in an automated way could be used to guide patient-specific quality improvement interventions. Further validation with diverse clinical outcomes and prospective studies is essential. Our findings indicate that significant disparities exist among breast cancer patients receiving different types of therapies based on social determinants of health. Social work reports play a crucial role in understanding these disparities in clinical decision-making.

Use of SNOMED CT in Large Language Models: Scoping Review.

Large language models (LLMs) have substantially advanced natural language processing (NLP) capabilities but often struggle with knowledge-driven tasks in specialized domains such as biomedicine. Integrating biomedical knowledge sources such as SNOMED CT into LLMs may enhance their performance on biomedical tasks. However, the methodologies and effectiveness of incorporating SNOMED CT into LLMs have not been systematically reviewed. This scoping review aims to examine how SNOMED CT is integrated into LLMs, focusing on (1) the types and components of LLMs being integrated with SNOMED CT, (2) which contents of SNOMED CT are being integrated, and (3) whether this integration improves LLM performance on NLP tasks. Following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines, we searched ACM Digital Library, ACL Anthology, IEEE Xplore, PubMed, and Embase for relevant studies published from 2018 to 2023. Studies were included if they incorporated SNOMED CT into LLM pipelines for natural language understanding or generation tasks. Data on LLM types, SNOMED CT integration methods, end tasks, and performance metrics were extracted and synthesized. The review included 37 studies. Bidirectional Encoder Representations from Transformers and its biomedical variants were the most commonly used LLMs. Three main approaches for integrating SNOMED CT were identified: (1) incorporating SNOMED CT into LLM inputs (28/37, 76%), primarily using concept descriptions to expand training corpora; (2) integrating SNOMED CT into additional fusion modules (5/37, 14%); and (3) using SNOMED CT as an external knowledge retriever during inference (5/37, 14%). The most frequent end task was medical concept normalization (15/37, 41%), followed by entity extraction or typing and classification. While most studies (17/19, 89%) reported performance improvements after SNOMED CT integration, only a small fraction (19/37, 51%) provided direct comparisons. The reported gains varied widely across different metrics and tasks, ranging from 0.87% to 131.66%. However, some studies showed either no improvement or a decline in certain performance metrics. This review demonstrates diverse approaches for integrating SNOMED CT into LLMs, with a focus on using concept descriptions to enhance biomedical language understanding and generation. While the results suggest potential benefits of SNOMED CT integration, the lack of standardized evaluation methods and comprehensive performance reporting hinders definitive conclusions about its effectiveness. Future research should prioritize consistent reporting of performance comparisons and explore more sophisticated methods for incorporating SNOMED CT's relational structure into LLMs. In addition, the biomedical NLP community should develop standardized evaluation frameworks to better assess the impact of ontology integration on LLM performance.

Public Sentiment Analysis of the Israel-Palestine Conflict on Social Media Using BERT

General background Israel-Palestine conflict has drawn significant global attention, particularly in how it is perceived and discussed on social media platforms. Specific background understanding public sentiment surrounding such geopolitical issues is crucial for media monitoring, diplomatic efforts, and reputation management. Knowledge gap previous sentiment analysis studies often lack the ability to accurately handle multilingual and context-rich datasets, especially in analyzing neutral sentiments, which are commonly overlooked. This study aims to apply the Bidirectional Encoder Representations from Transformers (BERT) model to analyze public sentiment towards the Israel-Palestine conflict on the X platform, focusing on Indonesian users. Results Using BERT, the model achieved 93% accuracy, with a precision of 0.95, recall of 0.93, and F1-score of 0.94. The model performed well in predicting positive and negative sentiments but showed room for improvement in handling neutral sentiment. Novelty this study introduces the implementation of the BERT Transformer model for the multilingual and context-sensitive sentiment analysis of tweets, specifically addressing a high-stakes geopolitical conflict. Implications the findings demonstrate the potential for using advanced natural language processing techniques like BERT for monitoring public opinion, brand management, and detecting societal tensions on social media, offering valuable insights for stakeholders involved in conflict resolution and diplomatic strategies. Highlights: Achieved 93% accuracy in sentiment analysis using BERT on X platform. Identified strengths in predicting positive/negative sentiments, with challenges in neutral sentiment. Demonstrated BERT’s effectiveness in handling complex geopolitical social media data. Keywords: Sentiment Analysis, BERT Transformer, Israel-Palestine Conflict, Social Media, NLP

Hybrid Lexicon and Transformer-Based Sentiment Analysis of Student Feedback for Faculty Evaluation: A Speech-to-Text Approach

The conventional methods for assessing the performance of educators in academic settings have often been hindered by outdated feedback systems, leading to assessments that can be either skewed or in- complete. This study unveils an innovative, dual-approach methodology that fuses lexicon-driven sentiment analysis with cutting-edge Transformer architectures, specifically focusing on BERT (Bidirectional Encoder Representations from Transformers). This approach provides a more sophisticated, real-time assessment of educators based on student evaluations. One of the standout features of this study is the integration of Speech- to-Text technologies, which allows for the immediate transformation of verbal feedback into text that can be analyzed. The approach makes use of a specialized Educational Sentiment Lexicon for preliminary sentiment evaluation, which subsequently refines the performance of a pre-existing BERT model. This dual-model is proficient in scrutinizing both text-based and verbal feedback—the latter being converted into text through advanced Speech-to-Text techniques. Our results indicate that this approach significantly outperforms existing lexicon-based and machine learning methods in terms of accuracy and comprehensiveness. By providing a real-time, versatile sentiment analysis tool tailored for educational settings, this research marks a paradigm shift in the scope and quality of faculty evaluations, thereby contributing substantially to the field of educational technology and sentiment analysis.

CADefender: Detection of unknown malicious AutoLISP computer-aided design files using designated feature extraction and machine learning methods

Computer-aided design (CAD) files are used to create digital designs for various structures – from the smallest chips in the high-tech industry to large-scale buildings and bridges in the civil engineering space. We found that most exploits and malicious payloads are deployed through Auto List Processing (AutoLISP) source code (LSP) or Fast Load AutoLISP (FAS) files, which are non-executable files (NEFs) containing scripts in the AutoLISP language that are native to AutoCAD; While antivirus software is capable of detecting many malicious CAD files, the potential to improve protection by using a dedicated machine learning (ML) based detection solution remains, especially against unknown and sophisticated CAD malware. In this study, we are the first to propose designated feature extraction methods and a robust framework aimed at the detection of known and unknown AutoLISP malware using ML algorithms. To accomplish this, we examined the structure, functionality, and ecosystems of AutoLISP files and collected the largest known representative collection of LSP files consisting of 6418 malicious and benign files (labeled and verified). We then explored the use of two novel static-analysis-based feature extraction methods (knowledge-based and structural) designated for LSP files to extract a discriminative set of informative features, which can subsequently be used by ML models to detect malicious LSP files. These two feature extraction methods serve as the basis of the proposed detection framework, whose performance we comprehensively compare to both widely used antiviruses and baseline ML models based on existing feature extraction methods, including MinHash, Bidirectional Encoder Representations from Transformers (BERT), and n-gram. Our results highlight our methods' contributions to the detection of unknown AutoLISP malware and demonstrate their ability to outperform existing methods. The best performance in the task of unknown malicious LSP file detection was obtained by the Artificial Neural Networks (ANN) model trained on 100 knowledge-based features, which obtained a true positive rate (TPR) of 99.49% with a false positive rate (FPR) of 0.57%. Our framework's role in explainability is also highlighted, as we also present the prominent features that contribute most to the model's detection capabilities; this information can be used for explainability purposes. We conclude by evaluating the proposed framework's ability to detect a malicious file from an unknown AutoLISP malware family and by evaluating our framework on an additional independent test set that originated from another source, scenarios that are often faced by malware detection solutions.

Text mining approach for feature extraction and cartilage disease grade classification using knee MRI radiology reports

MRI radiology reporting processes can be improved by exploiting structured and semantically labelled data that can be fed to artificial intelligence (AI) tools. AI-based tools assisting radiology reporting can help to automatically individuate cartilage grading in textual magnetic resonance imaging (MRI) reports, thus supporting clinicians' decisions regarding medical imaging utilisation, diagnosis and treatment. In this study, we extracted information (clinical findings, observations, anatomical regions, etc.) and classified knee cartilage degradation from medical reports utilising transfer-learning techniques applied to the Bidirectional Encoder Representations from Transformers (BERT) model and its variants, pre-trained on an Italian-language corpus. To realise this objective, we used a dataset of 750 MRI knee reports written by three radiologists who contributed to a manual annotation process to perform text classification (TC) and named entity recognition (NER) tasks. The dataset was obtained from an internal database of the IRCCS SYNLAB SDN. Seventy percent of the dataset was used for training, 10% was used for validation and 20% was used for testing. The best-performing configurations for NER and TC tasks were based on the pre-trained BERT model. The macro F1-scores obtained with the NER and TC models are 0.89 and 0.81, respectively. The accuracies calculated on the test set for both tasks are 0.96 and 0.99, respectively.

Technological Advancements in Menstrual Health: The Role of Generative Pre-Trained Transformer and Bees Algorithm

This paper introduces the concept of a smart menstrual cup, incorporating advanced technology to address challenges associated with traditional menstrual health management methods. The proposed method, Hybrid Bidirectional Encoder Generative Transformer based Bees Search (Hybrid BEGT-BS), aims to enhance the longevity and reliability of smart menstrual cups. The Generative Pre-trained Transformer (GPT) is employed to extract key information and determine emotional tones related to menstruation, while the Bidirectional Encoder Representations from Transformer (BERT) classifies menstrual-related data such as cycle tracking, hygiene, and symptoms. Further different validation metrics such as F1-score, specificity, Area Under the Curve-Receiver Operating Characteristic (AUC-ROC), Mean Squared Error (MSE), recall, energy consumption, accuracy, and precision are employed to assess the method's effectiveness. From the comparative results it demonstrates the superior performance of the Hybrid BEGT-BS method in enhancing the performance of smart menstrual cups.

A COMPREHENSIVE STUDY OF MACHINE LEARNING APPROACHES FOR CUSTOMER SENTIMENT ANALYSIS IN BANKING SECTOR

This study explores the application of sentiment analysis in the banking sector, focusing on customer feedback to enhance service quality and customer experiences. We collected a comprehensive dataset of approximately 100,000 entries from diverse sources, including customer satisfaction surveys, social media platforms, and direct feedback. A robust preprocessing pipeline was employed to address challenges associated with unstructured data, informal language, and mixed sentiments. We evaluated several machine learning and natural language processing models, including Logistic Regression, Naive Bayes, Support Vector Machine (SVM), Random Forest, Long Short-Term Memory (LSTM), and BERT (Bidirectional Encoder Representations from Transformers), using metrics such as accuracy, precision, recall, F1 score, AUC-ROC, and training time. The results revealed that advanced models, particularly BERT, achieved superior performance with an accuracy of 88% and an F1 score of 0.86, demonstrating an exceptional ability to capture nuanced sentiments. This study underscores the importance of employing sophisticated sentiment analysis techniques in banking to derive actionable insights from customer feedback. The findings suggest that leveraging advanced models can significantly improve service quality and customer satisfaction, while also presenting avenues for future research into real-time sentiment analysis and its integration with customer relationship management systems.