Performance Of Classifier Research Articles

AugmenToxic: Leveraging Reinforcement Learning to Optimize LLM Instruction Fine-Tuning for Data Augmentation to Enhance Toxicity Detection

Addressing the challenge of toxic language in online discussions is crucial for the development of effective toxicity detection models. This pioneering work focuses on addressing imbalanced datasets in toxicity detection by introducing a novel approach to augment toxic language data. We create a balanced dataset by instructing fine-tuning of Large Language Models (LLMs) using Reinforcement Learning with Human Feedback (RLHF). Recognizing the challenges in collecting sufficient toxic samples from social media platforms for building a balanced dataset, our methodology involves sentence-level text data augmentation through paraphrasing existing samples using optimized generative LLMs. Leveraging generative LLM, we utilize the Proximal Policy Optimizer (PPO) as the RL algorithm to fine-tune the model further and align it with human feedback. In other words, we start by fine-tuning a LLM using an instruction dataset, specifically tailored for the task of paraphrasing while maintaining semantic consistency. Next, we apply PPO and a reward function, to further fine-tune (optimize) the instruction-tuned LLM. This RL process guides the model in generating toxic responses. We utilize the Google Perspective API as a toxicity evaluator to assess generated responses and assign rewards/penalties accordingly. This approach guides LLMs through PPO and the reward function, transforming minority class samples into augmented versions. The primary goal of our methodology is to create a balanced and diverse dataset to enhance the accuracy and performance of classifiers in identifying instances from the minority class. Utilizing two publicly available toxic datasets, we compared various techniques with our proposed method for generating toxic samples, demonstrating that our approach outperforms all others in producing a higher number of toxic samples. Starting with an initial 16,225 toxic prompts, our method successfully generated 122,951 toxic samples with a toxicity score exceeding 30%. Subsequently, we developed various classifiers using the generated balanced datasets and applied a cost-sensitive learning approach to the original imbalanced dataset. The findings highlight the superior performance of classifiers trained on data generated using our proposed method. These results highlight the importance of employing RL and a data-agnostic model as a reward mechanism for augmenting toxic data, thereby enhancing the robustness of toxicity detection models.

Sentiment classification for insider threat identification using metaheuristic optimized machine learning classifiers

This study examines the formidable and complex challenge of insider threats to organizational security, addressing risks such as ransomware incidents, data breaches, and extortion attempts. The research involves six experiments utilizing email, HTTP, and file content data. To combat insider threats, emerging Natural Language Processing techniques are employed in conjunction with powerful Machine Learning classifiers, specifically XGBoost and AdaBoost. The focus is on recognizing the sentiment and context of malicious actions, which are considered less prone to change compared to commonly tracked metrics like location and time of access. To enhance detection, a term frequency-inverse document frequency-based approach is introduced, providing a more robust, adaptable, and maintainable method. Moreover, the study acknowledges the significant impact of hyperparameter selection on classifier performance and employs various contemporary optimizers, including a modified version of the red fox optimization algorithm. The proposed approach undergoes testing in three simulated scenarios using a public dataset, showcasing commendable outcomes.

Optimisation of artefact detection in photoplethysmography heart rate data: influence of different classifiers in machine learning models

Abstract Background Heart rate (HR) tracking by wrist-worn devices using photoplethysmography (PPG) could assist in continuously following up physical activity. However, the accuracy can be impacted by (motion) artefacts. Machine learning models could help to recognise artefacts in PPG-based HR data. The choice of classifier in these machine learning models is a determing factor for task performance of the model. Purpose This study evaluates and determines the optimal classifier for a new machine learning-based approach to enhance the reliability of artefact detection in PPG-based HR data. Methods A total of 62 participants (27 cardiac rehabilitation patients, 35 healthy athletes) wore both a test device and a reference device measuring HR continuously for 24 hours. A training dataset was prepared, assigning two independent labels (i.e. anomaly and activity) to each HR episode based on the reference device data. Fitbit data were processed using our in-house designed artefact removal procedure, which involves the application of two classification models: one for anomaly detection and another for activity detection. Four distinct classifiers were employed for both models: Balanced Bagging, Balanced Bagging with Random Forest, Balanced Random Forest, and Logistic Regression. Each classifier was evaluated using area under the receiver operating characteristic curve (ROC-AUC), accuracy, sensitivity and specificity. Results Of the 1,647,328 HR data points collected, 103,095 (6.26%) were identified as artefacts. Figure 1 and Figure 2 summarise the performance of the distinct classifiers for the anomaly model and the activity model, respectively. Balanced Bagging and Balanced Bagging with Random Forest consistently demonstrate the highest AUC values and accuracies across both anomaly and activity detection models (anomaly detection: AUC = 0.95, accuracy = 89-85%; activity detection: AUC = 0.98, accuracy = 95%). Comparing these two, Balanced Bagging with Random Forest emerges as the preferred option, given the highest sensitivity in both anomaly detection (93%>86%) and activity detection models (99%>96%). In contrast, Balanced Random Forest and Logistic Regression exhibit inferior performance. In the anomaly detection model, Balanced Random Forest exhibits a lower sensitivity of 75%, while Logistic Regression performs even worse with a sensitivity of 25%. Similarly, in the activity detection model, both Balanced Random Forest and Logistic Regression demonstrate diminished performance. Conclusions Balanced Bagging with Random Forest emerges as the optimal classifier to detect anomalies and activities in continuous PPG-based HR data, thus contributing to the optimisation of our in-house designed procedure for removing artefacts. This processing aims to provide a reliable and automatic way for continuous HR monitoring, which can help monitor and guide physical activities.

A Strategy for Predicting the Performance of Supervised and Unsupervised Tabular Data Classifiers

AbstractMachine Learning algorithms that perform classification are increasingly been adopted in Information and Communication Technology (ICT) systems and infrastructures due to their capability to profile their expected behavior and detect anomalies due to ongoing errors or intrusions. Deploying a classifier for a given system requires conducting comparison and sensitivity analyses that are time-consuming, require domain expertise, and may even not achieve satisfactory classification performance, resulting in a waste of money and time for practitioners and stakeholders. This paper predicts the expected performance of classifiers without needing to select, craft, exercise, or compare them, requiring minimal expertise and machinery. Should classification performance be predicted worse than expectations, the users could focus on improving data quality and monitoring systems instead of wasting time in exercising classifiers, saving key time and money. The prediction strategy uses scores of feature rankers, which are processed by regressors to predict metrics such as Matthews Correlation Coefficient (MCC) and Area Under ROC-Curve (AUC) for quantifying classification performance. We validate our prediction strategy through a massive experimental analysis using up to 12 feature rankers that process features from 23 public datasets, creating additional variants in the process and exercising supervised and unsupervised classifiers. Our findings show that it is possible to predict the value of performance metrics for supervised or unsupervised classifiers with a mean average error (MAE) of residuals lower than 0.1 for many classification tasks. The predictors are publicly available in a Python library whose usage is straightforward and does not require domain-specific skill or expertise.

Automatic Negation Detection for Semantic Analysis in Arabic Hotel Reviews Through Lexical and Structural Features: A Supervised Classification

One significant challenge in sentiment analysis is the presence of negation, which reverses the meanings of sentences, transformingpositive statements into negative ones and impacting the sentiment conveyed in the text. This issue is particularly pronounced in Arabic, a language known for its complex morphology. Detecting negation is crucial for enhancing sentiment analysis performance and various natural language processing applications. This paper presents an approach for automatically detecting negation in user-generated Arabic hotel reviews through lexical and structural features. It comprises several stages: data collection, text pre-processing, feature extraction, supervised learning classification, and evaluation. The study employed multiple supervised classification techniques, including naïve Bayes, random forest, logistic regression, support vector machines, and deep learning, to analyse lexical and structural features extracted from the dataset. The results of the experiments yielded promising outcomes, demonstrating the feasibility of the approach for practical applications. The classifiers exhibited highly comparable performance in identifying negation, with only marginal deviations in their performance metrics. Notably, the deep learning classifier consistently emerged as the top performer, achieving an exceptionally high overall accuracy rate of 99.24 percent, surpassing established benchmarks in Arabic text processing and underscoring its potential for practical applications. These findings hold significant implications for advancing Arabic text processing, particularly in sentiment analysis and related NLP tasks. The high accuracy of 99.24 percent achieved by the deep learning classifier highlights its robustness in accurately detecting negation, a critical challenge in sentiment analysis. This classifier performance demonstrates the potential to be integrated into real-world applications, such as automated review systems and opinion mining tools, where accurate sentiment interpretation is essential.

Enhancing Activity Recognition After Stroke: Generative Adversarial Networks for Kinematic Data Augmentation

The generalizability of machine learning (ML) models for wearable monitoring in stroke rehabilitation is often constrained by the limited scale and heterogeneity of available data. Data augmentation addresses this challenge by adding computationally derived data to real data to enrich the variability represented in the training set. Traditional augmentation methods, such as rotation, permutation, and time-warping, have shown some benefits in improving classifier performance, but often fail to produce realistic training examples. This study employs Conditional Generative Adversarial Networks (cGANs) to create synthetic kinematic data from a publicly available dataset, closely mimicking the experimentally measured reaching movements of stroke survivors. This approach not only captures the complex temporal dynamics and common movement patterns after stroke, but also significantly enhances the training dataset. By training deep learning models on both synthetic and experimental data, we enhanced task classification accuracy: models incorporating synthetic data attained an overall accuracy of 80.0%, significantly higher than the 66.1% seen in models trained solely with real data. These improvements allow for more precise task classification, offering clinicians the potential to monitor patient progress more accurately and tailor rehabilitation interventions more effectively.

Integrating standard epilepsy protocol, ASL-perfusion, MP2RAGE/EDGE and the MELD-FCD classifier in the detection of subtle epileptogenic lesions: a 3 Tesla MRI pilot study.

Malformations of cortical development (MCDs) in children with focal epilepsy pose significant diagnostic challenges, and a precise radiological diagnosis is crucial for surgical planning. New MRI sequences and the use of artificial intelligence (AI) algorithms are considered very promising in this regard, yet studies evaluating the relative contribution of each diagnostic technique are lacking. The study was conducted using a dedicated "EPI-MCD MR protocol" with a 3 Tesla MRI scanner in patients with focal epilepsy and previously negative MRI. MRI sequences evaluated included 3D FLAIR, 3D T1 MPRAGE, T2 Turbo Spin Echo (TSE), 3D T1 MP2RAGE, and Arterial Spin Labelling (ASL). Two paediatric neuroradiologists scored each sequence for localisation and extension of the lesion. The MELD-FCD AI classifier's performance in identifying pathological findings was also assessed. We only included patients where a diagnosis of MCD was subsequently confirmed on histology and/or sEEG. The 3D FLAIR sequence showed the highest yield in detecting epileptogenic lesions, with 3D T1 MPRAGE, T2 TSE, and 3D T1 MP2RAGE sequences showing moderate to low yield. ASL was the least useful. The MELD-FCD classifier achieved a 69.2% true positive rate. In one case, MELD identified a subtle area of cortical dysplasia overlooked by the neuroradiologists, changing the management of the patient. The 3D FLAIR sequence is the most effective in the MRI-based diagnosis of subtle epileptogenic lesions, outperforming other sequences in localisation and extension. This pilot study emphasizes the importance of careful assessment of the value of additional sequences.

Conceptualizing bias in EHR data: A case study in performance disparities by demographic subgroups for a pediatric obesity incidence classifier.

Electronic Health Records (EHRs) are increasingly used to develop machine learning models in predictive medicine. There has been limited research on utilizing machine learning methods to predict childhood obesity and related disparities in classifier performance among vulnerable patient subpopulations. In this work, classification models are developed to recognize pediatric obesity using temporal condition patterns obtained from patient EHR data in a U.S. study population. We trained four machine learning algorithms (Logistic Regression, Random Forest, Gradient Boosted Trees, and Neural Networks) to classify cases and controls as obesity positive or negative, and optimized hyperparameter settings through a bootstrapping methodology. To assess the classifiers for bias, we studied model performance by population subgroups then used permutation analysis to identify the most predictive features for each model and the demographic characteristics of patients with these features. Mean AUC-ROC values were consistent across classifiers, ranging from 0.72-0.80. Some evidence of bias was identified, although this was through the models performing better for minority subgroups (African Americans and patients enrolled in Medicaid). Permutation analysis revealed that patients from vulnerable population subgroups were over-represented among patients with the most predictive diagnostic patterns. We hypothesize that our models performed better on under-represented groups because the features more strongly associated with obesity were more commonly observed among minority patients. These findings highlight the complex ways that bias may arise in machine learning models and can be incorporated into future research to develop a thorough analytical approach to identify and mitigate bias that may arise from features and within EHR datasets when developing more equitable models.

G4 & the balanced metric family – a novel approach to solving binary classification problems in medical device validation & verification studies

BackgroundIn medical device validation and verification studies, the area under the receiver operating characteristic curve (AUROC) is often used as a primary endpoint despite multiple reports showing its limitations. Hence, researchers are encouraged to consider alternative metrics as primary endpoints. A new metric called G4 is presented, which is the geometric mean of sensitivity, specificity, the positive predictive value, and the negative predictive value. G4 is part of a balanced metric family which includes the Unified Performance Measure (also known as P4) and the Matthews’ Correlation Coefficient (MCC). The purpose of this manuscript is to unveil the benefits of using G4 together with the balanced metric family when analyzing the overall performance of binary classifiers.ResultsSimulated datasets encompassing different prevalence rates of the minority class were analyzed under a multi-reader-multi-case study design. In addition, data from an independently published study that tested the performance of a unique ultrasound artificial intelligence algorithm in the context of breast cancer detection was also considered. Within each dataset, AUROC was reported alongside the balanced metric family for comparison. When the dataset prevalence and bias of the minority class approached 50%, all three balanced metrics provided equivalent interpretations of an AI’s performance. As the prevalence rate increased / decreased and the data became more imbalanced, AUROC tended to overvalue / undervalue the true classifier performance, while the balanced metric family was resistant to such imbalance. Under certain circumstances where data imbalance was strong (minority-class prevalence < 10%), MCC was preferred for standalone assessments while P4 provided a stronger effect size when evaluating between-groups analyses. G4 acted as a middle ground for maximizing both standalone assessments and between-groups analyses.ConclusionsUse of AUROC as the primary endpoint in binary classification problems provides misleading results as the dataset becomes more imbalanced. This is explicitly noticed when incorporating AUROC in medical device validation and verification studies. G4, P4, and MCC do not share this limitation and paint a more complete picture of a medical device’s performance in a clinical setting. Therefore, researchers are encouraged to explore the balanced metric family when evaluating binary classification problems.

Exploring the role of project status information in effective code smell detection

Repairing code smells detected in the code or design of the system is one of the activities contributing to increasing the software quality. In this study, we investigate the impact of non-numerical information of software, such as project status information combined with machine learning techniques, on improving code smell detection. For this purpose, we constructed a dataset consisting of 22 systems with various project statuses, 12,040 classes, and 18 features that included 1935 large classes. A set of experiments was conducted with ten different machine learning techniques by dividing the dataset into training, validation, and testing sets to detect the large class code smell. Feature selection and data balancing techniques have been applied. The classifier’s performance was evaluated using six indicators: precision, recall, F-measure, MCC, ROC area, and Kappa tests. The preliminary experimental results reveal that feature selection and data balancing have poor influence on the accuracy of machine learning classifiers. Moreover, they vary their behavior when utilized in sets with different values for the selected project status information of their classes. The average value of classifiers performance when fed with status information is better than without. The Random Forest achieved the best behavior according to all performance indicators (100%) with status information, while AdaBoostM1 and SMO achieved the worst in most of them (> 86%). According to the findings of this study, providing machine learning techniques with project status information about the classes to be analyzed can improve the results of large class detection.

Assessing the influence of latency variability on EEG classifiers - a case study of face repetition priming

AbstractData-driven strategies have been widely used to distinguish experimental effects on single-trial EEG signals. However, how latency variability, such as within-condition jitter or latency shifts between conditions, affects the performance of EEG classifiers has not been well investigated. Without explicitly considering and disentangling such attributes of single trials, neural network-based classifiers have limitations in measuring their contributions. Inspired by domain knowledge of subcomponent latency and amplitude from traditional cognitive neuroscience, this study applies a stepwise latency correction method on single trials to control for their contributions to classifier behavior. As a case study demonstrating the value of this method, we measure repetition priming effects of faces, which induce large reaction time differences, latency shifts, and amplitude effects in averaged event-related potentials. The results show that within-condition jitter negatively impacts classifier performance, but between-condition latency shifts improve accuracy, whereas genuine amplitude differences have no significant influence. While demonstrated in the case of priming effects, this methodology can be generalized to experiments involving many kinds of time-varying signals to account for the contributions of latency variability to classifier performance.

Semi-supervised comparative learning compensation method for chemical gas sensor drift.

The gradual and unpredictable variation in chemo-sensory signal responses when exposed to the same analyte under identical conditions, commonly referred to as sensor drift, has long been recognized as one of the most serious challenges faced by chemical sensors. The traditional drift compensation method is both labor-intensive and expensive, as it requires frequent collection and labeling of gas samples for recalibration. Introducing a small number of meaningful drift calibration samples can be an attractive strategy to reduce the computational load and improve the performance of the updated classifier. However, under the influence of drift, new challenges arise due to the difference in the distribution of source and target domain data. This paper proposes a novel algorithm framework called semi-supervised contrastive learning drift compensation (SSCLDC). The framework automatically extracts high-level abstract features based on a multilayer perceptron to better represent the structure of the source data. In addition, to address the issue of data distribution differences caused by drift between the source and target domains. We add a small number of reference sample pairs into the training for semi-supervised learning. Combining a contrastive loss function that can represent the matching degree of paired samples effectively overcomes the problem of sensor drift. The Kennard-Stone sequential algorithm is used to select the representative reference sample from the set of candidate reference samples. Experiments conducted on a widely used long-term chemical gas sensor drift dataset demonstrate that the proposed method outperforms several classic drift compensation techniques, highlighting its effectiveness and practical applicability.

AI-DPAPT: a machine learning framework for predicting PROTAC activity.

Proteolysis Targeting Chimeras are part of targeted protein degradation (TPD) techniques, which are significant for pharmacological and therapy development. Small-molecule interaction with the targeted protein is a complicated endeavor and a challenge to predict the proteins accurately. This study used machine learning algorithms and molecular fingerprinting techniques to build an AI-powered PROTAC Activity Prediction Tool that could predict PROTAC activity by examining chemical structures. The chemical structures of a diverse set of PROTAC drugs and their corresponding activities are selected as a dataset for training the tool. The processes used in this study included data preparation, feature extraction, and model training. Further, evaluation was done for the performance of the various classifiers, such as AdaBoost, Support Vector Machine, Random Forest, Gradient Boosting, and Multi-Layer Perceptron. The findings show that the methods selected here depict accurate PROTAC activities. All the models in this study showed an ROC curve better than 0.9, while the random forest on the test set of the AI-DPAPT had an area under the curve score of 0.97, thus showing accurate results. Furthermore, the study revealed significant insights into the molecular features that can influence the functions of the PROTAC. These findings can potentially increase the understanding of the structure-activity correlations involved in the TPD. Overall, the investigation contributes to computational drug development by introducing this platform powered by artificial intelligence that predicts the function of PROTAC. In addition, it sped up the processes of identifying and improving previously unknown medications. The AI-DPAPT platform can be accessed online using a web server at https://ai-protac.streamlit.app/ .

Large-scale structural covariance networks changes relate to executive function deficit in betel quid-dependent chewers.

Previous studies demonstrate deficits in executive function for betel quid-dependent (BQD) patients. Large-scale structural covariance network (SCN) based on gray matter (GM) morphometry may be able to explore the neural mechanism of executive dysfunction in BQD individuals. This study aims to identify spatial covariance patterns of GM volume and to investigate their association with executive dysfunction in BQD individuals. Sixty-four BQD individuals and 48 sex- and age-matched healthy controls (HCs) underwent T1-weighted structural MRI examination and executive function assessments, including the Backward Digit Span (BDS) test and Stroop Color and Word (SCW) test. Seventy SCNs based on GM volume covariance patterns were defined using independent component analysis. An SCN-based classifier was constructed to differentiate between BQD and HC individuals. Receiver operating characteristic (ROC) curves were applied to evaluate the performance of the SCN-based classifier. Linear regression analyses were performed to investigate the association between SCN network indices and executive function indices. Six SCNs had higher classifications for differentiating between BQD and HC individuals. The area under the ROC curve of the SCN-based classifier was 0.812 in the training set and 0.771 in the testing set. Furthermore, linear regression analyses demonstrated that the network indices in the thalamus were associated with BDS scores adjusted for age, sex, and education. Large-scale SCNs could provide potential imaging markers for differentiating BQD and HC groups. The loss of network index in the thalamus was associated with working memory, indicating that SCNs could reveal executive dysfunction in BQD individuals.

SWL-LSE: A Dataset of Health-Related Signs in Spanish Sign Language with an ISLR Baseline Method

Progress in automatic sign language recognition and translation has been hindered by the scarcity of datasets available for the training of machine learning algorithms, a challenge that is even more acute for languages with smaller signing communities, such as Spanish. In this paper, we introduce a dataset of 300 isolated signs in Spanish Sign Language, collected online via a web application with contributions from 124 participants, resulting in a total of 8000 instances. This dataset, which is openly available, includes keypoints extracted using MediaPipe Holistic. The goal of this paper is to describe the construction and characteristics of the dataset and to provide a baseline classification method using a spatial–temporal graph convolutional network (ST-GCN) model, encouraging the scientific community to improve upon it. The experimental section offers a comparative analysis of the method’s performance on the new dataset, as well as on two other well-known datasets. The dataset, code, and web app used for data collection are freely available, and the web app can also be used to test classifier performance on-line in real-time.

Improving acoustic species identification using data augmentation within a deep learning framework

Convolutional neural networks (CNNs) are effective tools for acoustic classification tasks such as species identification. Large datasets of labelled recordings are required to develop CNN classifiers which can be difficult to obtain, particularly if species are rare or vocalise infrequently. Additionally, data often requires manual labelling which can be time consuming requiring expert analysis. Artificially generating data using augmentation can address these challenges, however the impact of data augmentation on CNN performance is poorly understood and often omitted in bioacoustic studies. Here, we empirically test the impact of CNN architecture and 20 data augmentation methods on classifier performance. We use acoustic identification of 18 small mammal species as a case study of a species group that can be effectively surveyed by acoustic monitoring, but recordings for training data are scarce and difficult to collect. Networks that achieved the highest accuracy across all sample sizes was a 10-layer CNN (96.43 %) and a pre-trained ResNet50 model (96.37 %). Overall, all augmentation effects improved ResNet50 model performance and 17 effects improved Conv10 performance, increasing relative change in accuracy (RCA) by 0.021–0.641. Three augmentation effects negatively impacted Conv10 RCA by −0.042 to −0.182. We also show that adding augmented data when the number of original samples is low has the greatest positive impact on accuracy and this effect was larger with ResNet50 models. Our work demonstrates that using data augmentation where few original samples are available can considerably improve model performance and highlights the potential of augmentation in developing acoustic classifiers for species where data are limited or difficult to obtain.

Leveraging Classifier Performance Using Heuristic Optimization for Detecting Cardiovascular Disease from PPG Signals.

Photoplethysmography (PPG) signals, which measure blood volume changes through light absorption, are increasingly used for non-invasive cardiovascular disease (CVD) detection. Analyzing PPG signals can help identify irregular heart patterns and other indicators of CVD. This research involves a total of 41 subjects sourced from the CapnoBase database, consisting of 21 normal subjects and 20 CVD cases. In the initial stage, heuristic optimization algorithms, such as ABC-PSO, the Cuckoo Search algorithm (CSA), and the Dragonfly algorithm (DFA), were applied to reduce the dimension of the PPG data. Next, these Dimensionally Reduced (DR) PPG data are then fed into various classifiers such as Linear Regression (LR), Linear Regression with Bayesian Linear Discriminant Classifier (LR-BLDC), K-Nearest Neighbors (KNN), PCA-Firefly, Linear Discriminant Analysis (LDA), Kernel LDA (KLDA), Probabilistic LDA (ProbLDA), SVM-Linear, SVM-Polynomial, and SVM-RBF, to identify CVD. Classifier performance is evaluated using Accuracy, Kappa, MCC, F1 Score, Good Detection Rate (GDR), Error rate, and Jaccard Index (JI). The SVM-RBF classifier for ABC PSO dimensionality reduced values outperforms other classifiers, achieving the highest accuracy of 95.12% along with the minimum error rate of 4.88%. In addition to that, it provides an MCC and kappa value of 0.90, a GDR and F1 score of 95%, and a Jaccard Index of 90.48%. This study demonstrated that heuristic-based optimization and machine learning classification of PPG signals are highly effective for the non-invasive detection of cardiovascular disease.

Enhancing Intent Classifier Training with Large Language Model-generated Data

ABSTRACT Intent classification is essential in Natural Language Processing, serving applications like virtual assistants and customer service by categorizing user inputs into predefined classes. Despite its importance, the effectiveness of intent classifiers is often constrained by the scarcity of labeled data, as acquiring substantial, annotated datasets is costly and impractical. Data augmentation addresses this by expanding datasets with modified or synthetic examples, a common practice in computer vision but more complex in NLP due to the discrete nature of language. Traditional NLP data augmentation methods have been explored but exhibit limitations. This paper investigates the use of Large Language Models for generating labeled data to enhance intent classification. We explore whether LLM-generated data can effectively augment training sets, comparing its impact on intent classifier performance against traditional augmentation methods. Our study reveals that LLMs can generate diverse and realistic data, potentially improving classifier accuracy in low-data scenarios, thereby providing valuable insights into leveraging generative AI for NLP tasks in real-world applications.

CGAN Facilitated Data Augmentation of Voice and Speech Parameters for Detecting Parkinson’s Disease in the Prodromal Phase

Parkinson’s disease is a multi-faceted disease affecting the brain. The enormity of its recent rise is quite alarming. This calls for intense research to diagnose early to hasten the progress of diagnosis. Voice distortion is considered an early precursor for Parkinson’s disease. Though several studies in Machine Learning using voice parameters have provided useful information, none of them have been successful in evolving an efficient and generalized model to detect it. Deep Learning techniques were applied to improve the performance of the model but its major limitation was the size of the dataset. Hence, a need arose to extend the dataset using an appropriate data augmentation method. At this juncture, the conditional generative adversarial network (CGAN) proved to be a useful technique because of its innate feature for generating synthetic data from input noise. The RNN-LSTM classifier could achieve a training accuracy of 87.32%, testing accuracy of 86.3%, training precision of 87.92 %, and testing precision of 89.94%. The results of the experimental study are compared with other state-of-the-art methods. This technique succeeded in reducing the problem of over-fitting and could elevate the performance of the RNN-LSTM classifier in the prediction of Parkinson’s disease.

Habitat-based CT radiomics enhances the ability to predict spread through air spaces in stage T1 invasive lung adenocarcinoma.

Spread through air spaces (STAS) represents a novel invasive pattern in lung adenocarcinoma (LUAD) and is a risk factor for poor prognosis in stage T1 LUAD. This study aims to develop and validate a CT habitat imaging analysis model for predicting STAS in stage T1 invasive LUAD. We retrospectively analyzed 217 patients with preoperative stage T1 invasive LUAD (115 STAS-positive and 102 STAS-negative cases, including 151 in the train set and 66 in the test set). Semi-automatic segmentation was performed on the regions of interest (ROIs) in all CT images, with an automatic 3mm expansion around the tumor, considering the intratumoral and peritumoral 3mm area. This area was divided into three sub-regions via K-means clustering, and 1197 radiomic features were extracted from each sub-region and the overall combined region. After dimension reduction through the Mann-Whitney U test, Pearson correlation analysis, and least absolute shrinkage and selection operator(LASSO), the best features for each sub-region and overall were selected. Models were then built using the selected radiomic features through the Adaptive Boosting (AdaBoost) and Multilayer Perceptron (MLP) classifiers. Four different models were established based on different sub-regions and the overall features. The performance of these models was evaluated through receiver operating characteristic curves (AUC) under the DeLong test, calibration curves via the Hosmer-Lemeshow test, and decision curve analysis to assess the performance of these features. In this study, we evaluated the predictive performance of AdaBoost and MLP classifiers on rad feature models across various subregions and the overall dataset. In the test set, the AdaBoost classifier achieved a maximum AUC of 0.871 in Habitat 3, whereas the MLP classifier demonstrated slightly superior performance with an AUC of 0.879. Both classifiers exhibited high efficiency in habitat 3, with the MLP algorithm showing enhanced model performance. CT habitat imaging analysis for the preoperative prediction of STAS in stage T1 invasive LUAD shows satisfactory diagnostic performance, with the habitat3 model exhibiting the highest efficacy, reflecting tumor heterogeneity.