Best-performing Method Research Articles

Segmentation-based truncated-SVD for effective feature extraction in hyperspectral image classification

ABSTRACT Remote sensing hyperspectral images (HSIs) are rich sources of information about land cover captured across hundreds of narrow, contiguous spectral wavelength bands. However, using the entire original HSI for practical applications can lead to suboptimal classification accuracy. To address this, band reduction techniques, categorized as feature extraction and feature selection methods, are employed to enhance classification results. One commonly used feature extraction approach for HSIs is Principal Component Analysis (PCA). However, PCA may fall short of capturing the local and specific characteristics present in the HSI data. In this paper, we introduce two novel feature extraction methods: Segmented Truncated Singular Value Decomposition (STSVD) and Spectrally Segmented Truncated Singular Value Decomposition (SSTSVD) to improve classification performance. Segmentation is carried out based on highly correlated bands’ segments and spectral bands’ segments within the HSI data. Our study evaluates and compares these newly proposed methods against classical feature extraction methods, including PCA, Incremental PCA, Sparse-PCA, Kernel PCA, Segmented-PCA (SPCA), and Truncated Singular Value Decomposition (TSVD). We perform this analysis on three distinct HSI datasets, namely the Indian Pines HSI, the Pavia University HSI, and the Kennedy Space Center HSI, using per-pixel Support Vector Machine (SVM) and Random Forest (RF) classification. The experimental results demonstrate the superiority of our proposed methods for all three datasets. The best-performing feature extraction methods when classification is performed using an SVM classifier are STSVD3 (89.03%), SSTSVD2 (95.55%), and STSVD3 (97.74%) for the Indian Pines, Pavia University, and Kennedy Space Center datasets, respectively. Similarly, for the RF classifier, the best-performing feature extraction methods are SSTSVD4 (88.98%), SSTSVD3 (96.04%), and SSTSVD4 (96.09%) for Indian Pines, Pavia University, and Kennedy Space Center datasets, respectively.

Comparison of whole-brain task-modulated functional connectivity methods for fMRI task connectomics

Higher brain functions require flexible integration of information across widely distributed brain regions depending on the task context. Resting-state functional magnetic resonance imaging (fMRI) has provided substantial insight into large-scale intrinsic brain network organisation, yet the principles of rapid context-dependent reconfiguration of that intrinsic network organisation are much less understood. A major challenge for task connectome mapping is the absence of a gold standard for deriving whole-brain task-modulated functional connectivity matrices. Here, we perform biophysically realistic simulations to control the ground-truth task-modulated functional connectivity over a wide range of experimental settings. We reveal the best-performing methods for different types of task designs and their fundamental limitations. Importantly, we demonstrate that rapid (100 ms) modulations of oscillatory neuronal synchronisation can be recovered from sluggish haemodynamic fluctuations even at typically low fMRI temporal resolution (2 s). Finally, we provide practical recommendations on task design and statistical analysis to foster task connectome mapping.

Comparing Different Methods for the Diagnosis of Liver Steatosis: What Are the Best Diagnostic Tools?

Due to the ongoing organ shortage, marginal grafts with steatosis are more frequently used in liver transplantation, leading to higher occurrences of graft dysfunction. A histological analysis is the gold standard for the quantification of liver steatosis (LS), but has its drawbacks: it is an invasive method that varies from one pathologist to another and is not available in every hospital at the time of organ procurement. This study aimed to compare non-invasive diagnostic tools to a histological analysis for the quantification of liver steatosis. Male C57BL6J mice were fed with a methioninecholine-deficient (MCD) diet for 14 days or 28 days to induce LS, and were compared to a control group of animals fed with a normal diet. The following non-invasive techniques were performed and compared to the histological quantification of liver steatosis: magnetic resonance spectroscopy (MRS), CARS microscopy, 99mTc MIBI SPECT imaging, and a new near-infrared spectrometer (NIR-SG1). After 28 days on the MCD diet, an evaluation of LS showed ≥30% macrovesicular steatosis. High correlations were found between the NIR-SG1 and the blinded pathologist analysis (R2 = 0.945) (p = 0.001), and between the CARS microscopy (R2 = 0.801 (p < 0.001); MRS, R2 = 0.898 (p < 0.001)) and the blinded pathologist analysis. The ROC curve analysis showed that the area under the curve (AUC) was 1 for both the NIR-SG1 and MRS (p = 0.021 and p < 0.001, respectively), while the AUC = 0.910 for the Oil Red O stain (p < 0.001) and the AUC = 0.865 for the CARS microscopy (p < 0.001). The AUC for the 99mTc MIBI SPECT was 0.640 (p = 0.013), and this was a less discriminating technique for LS quantification. The best-performing non-invasive methods for LS quantification are MRS, CARS microscopy, and the NIR-SG1. The NIR-SG1 is particularly appropriate for clinical practice and needs to be validated by clinical studies on liver grafts.

Comparison of whole-brain task-modulated functional connectivity methods for fMRI task connectomics.

Higher brain functions require flexible integration of information across widely distributed brain regions depending on the task context. Resting-state functional magnetic resonance imaging (fMRI) has provided substantial insight into large-scale intrinsic brain network organisation, yet the principles of rapid context-dependent reconfiguration of that intrinsic network organisation are much less understood. A major challenge for task connectome mapping is the absence of a gold standard for deriving whole-brain task-modulated functional connectivity matrices. Here, we perform biophysically realistic simulations to control the ground-truth task-modulated functional connectivity over a wide range of experimental settings. We reveal the best-performing methods for different types of task designs and their fundamental limitations. Importantly, we demonstrate that rapid (100 ms) modulations of oscillatory neuronal synchronisation can be recovered from sluggish haemodynamic fluctuations even at typically low fMRI temporal resolution (2 s). Finally, we provide practical recommendations on task design and statistical analysis to foster task connectome mapping.

Optimizing and benchmarking polygenic risk scores with GWAS summary statistics

BackgroundPolygenic risk score (PRS) is a major research topic in human genetics. However, a significant gap exists between PRS methodology and applications in practice due to often unavailable individual-level data for various PRS tasks including model fine-tuning, benchmarking, and ensemble learning.ResultsWe introduce an innovative statistical framework to optimize and benchmark PRS models using summary statistics of genome-wide association studies. This framework builds upon our previous work and can fine-tune virtually all existing PRS models while accounting for linkage disequilibrium. In addition, we provide an ensemble learning strategy named PUMAS-ensemble to combine multiple PRS models into an ensemble score without requiring external data for model fitting. Through extensive simulations and analysis of many complex traits in the UK Biobank, we demonstrate that this approach closely approximates gold-standard analytical strategies based on external validation, and substantially outperforms state-of-the-art PRS methods.ConclusionsOur method is a powerful and general modeling technique that can continue to combine the best-performing PRS methods out there through ensemble learning and could become an integral component for all future PRS applications.

Interpretable Spatial–Temporal Graph Convolutional Network for System Log Anomaly Detection

To ensure seamless information flow and operational integrity, computer systems need effectively to manage their system logs, but the expansion in their scale and complexity makes it hard to detect anomalies. Current methodologies exhibit deficiencies, including inefficiencies in handling abnormal sequences, lack of interpretability, and limited consideration of both temporal and spatial information. To improve, this paper develops a semi-supervised graph neural network model termed the Interpretable Spatial–Temporal Graph Convolutional Network (IST-GCN). By integrating temporal and event similarity perspectives, the IST-GCN harnesses directed and undirected graphs to capture the temporal and spatial aspects of system log events. Hence, the IST-GCN offers temporal and spatial interpretability. Further, a lightweight feature regularization technique is developed to enhance interpretability in both time and space domains, and thus facilitates anomaly detection efficiently. Comprehensive testing verifies that the IST-GCN approach surpasses nearly all state-of-the-art methods across five public log anomaly datasets. On average, IST-GCN improves Average Precision (AP) by approximately 3% and ROC AUC (RC) by about 4% compared to the best-performing baseline methods, underscoring its effectiveness and robustness.

Keywords-enhanced Contrastive Learning Model for travel recommendation

Travel recommendation aims to infer travel intentions of users by analyzing their historical behaviors on Online Travel Agencies (OTAs). However, crucial keywords in clicked travel product titles, such as destination and itinerary duration, indicating tourists’ intentions, are often overlooked. Additionally, most previous studies only consider stable long-term user interests or temporary short-term user preferences, making the recommendation performance unreliable. To mitigate these constraints, this paper proposes a novel Keywords-enhanced Contrastive Learning Model (KCLM). KCLM simultaneously implements personalized travel recommendation and keywords generation tasks, integrating long-term and short-term user preferences within both tasks. Furthermore, we design two kinds of contrastive learning tasks for better user and travel product representation learning. The preference contrastive learning aims to bridge the gap between long-term and short-term user preferences. The multi-view contrastive learning focuses on modeling the coarse-grained commonality between clicked products and their keywords. Extensive experiments are conducted on two tourism datasets and a large-scale e-commerce dataset. The experimental results demonstrate that KCLM achieves substantial gains in both metrics compared to the best-performing baseline methods. Specifically, HR@20 improved by 5.79%–14.13%, MRR@20 improved by 6.57%–18.50%. Furthermore, to have an intuitive understanding of the keyword generation by the KCLM model, we provide a case study for several randomized examples.

Reliable estimation of tree branch lengths using deep neural networks.

A phylogenetic tree represents hypothesized evolutionary history for a set of taxa. Besides the branching patterns (i.e., tree topology), phylogenies contain information about the evolutionary distances (i.e. branch lengths) between all taxa in the tree, which include extant taxa (external nodes) and their last common ancestors (internal nodes). During phylogenetic tree inference, the branch lengths are typically co-estimated along with other phylogenetic parameters during tree topology space exploration. There are well-known regions of the branch length parameter space where accurate estimation of phylogenetic trees is especially difficult. Several novel studies have recently demonstrated that machine learning approaches have the potential to help solve phylogenetic problems with greater accuracy and computational efficiency. In this study, as a proof of concept, we sought to explore the possibility of machine learning models to predict branch lengths. To that end, we designed several deep learning frameworks to estimate branch lengths on fixed tree topologies from multiple sequence alignments or its representations. Our results show that deep learning methods can exhibit superior performance in some difficult regions of branch length parameter space. For example, in contrast to maximum likelihood inference, which is typically used for estimating branch lengths, deep learning methods are more efficient and accurate. In general, we find that our neural networks achieve similar accuracy to a Bayesian approach and are the best-performing methods when inferring long branches that are associated with distantly related taxa. Together, our findings represent a next step toward accurate, fast, and reliable phylogenetic inference with machine learning approaches.

A comparison of scRNA-seq annotation methods based on experimentally labeled immune cell subtype dataset.

Cell-type annotation is a critical step in single-cell data analysis. With the development of numerous cell annotation methods, it is necessary to evaluate these methods to help researchers use them effectively. Reference datasets are essential for evaluation, but currently, the cell labels of reference datasets mainly come from computational methods, which may have computational biases and may not reflect the actual cell-type outcomes. This study first constructed an experimentally labeled immune cell-subtype single-cell dataset of the same batch and systematically evaluated 18 cell annotation methods. We assessed those methods under five scenarios, including intra-dataset validation, immune cell-subtype validation, unsupervised clustering, inter-dataset annotation, and unknown cell-type prediction. Accuracy and ARI were evaluation metrics. The results showed that SVM, scBERT, and scDeepSort were the best-performing supervised methods. Seurat was the best-performing unsupervised clustering method, but it couldn't fully fit the actual cell-type distribution. Our results indicated that experimentally labeled immune cell-subtype datasets revealed the deficiencies of unsupervised clustering methods and provided new dataset support for supervised methods.

Graph-Based Traffic Forecasting with the Dynamics of Road Symmetry and Capacity Performance

Symmetry in traffic patterns is a fundamental aspect of intelligent transportation systems, aiming to precisely predict traffic flow in real time despite the complex interplay of spatial and temporal factors. This paper presents a novel method of traffic forecasting that incorporates parameters related to road symmetry into a Graph Convolution Network model. Our model is crafted to dynamically adjust to real-time changes in road conditions, including the presence of symmetric and asymmetric road layouts, which substantially influence traffic flow. We have developed a GCN model that not only accounts for standard traffic flow metrics but also integrates a matrix representing road symmetry. The model undergoes training and validation on the METR-LA dataset, showcasing a significant enhancement in prediction accuracy. In the comparative analysis of state-of-the-art methods, our model demonstrated a significant enhancement in performance, achieving 30.68% improvement in Mean Squared Error (MSE) and a 24.28% improvement in Mean Absolute Error (MAE) over the best-performing method. The implications of our research are profound for urban planners, traffic management systems, and navigation service providers, as it offers a more dependable tool for forecasting traffic conditions, aiding in road design, and refining route planning strategies based on the symmetry of road networks.

Terrestrial laser scanning and low magnetic field digitization yield similar architectural coarse root traits for 32-year-old Pinus ponderosa trees

BackgroundUnderstanding how trees develop their root systems is crucial for the comprehension of how wildland and urban forest ecosystems plastically respond to disturbances such as harvest, fire, and climate change. The interplay between the endogenously determined root traits and the response to environmental stimuli results in tree adaptations to biotic and abiotic factors, influencing stability, carbon allocation, and nutrient uptake. Combining the three-dimensional structure of the root system, with root morphological trait information promotes a robust understanding of root function and adaptation plasticity. Low Magnetic Field Digitization coupled with AMAPmod (botAnique et Modelisation de l’Architecture des Plantes) software has been the best-performing method for describing root system architecture and providing reliable measurements of coarse root traits, but the pace and scale of data collection remain difficult. Instrumentation and applications related to Terrestrial Laser Scanning (TLS) have advanced appreciably, and when coupled with Quantitative Structure Models (QSM), have shown some potential toward robust measurements of tree root systems. Here we compare, we believe for the first time, these two methodologies by analyzing the root system of 32-year-old Pinus ponderosa trees.ResultsIn general, at the total root system level and by root-order class, both methods yielded comparable values for the root traits volume, length, and number. QSM for each root trait was highly sensitive to the root size (i.e., input parameter PatchDiam) and models were optimized when discrete PatchDiam ranges were specified for each trait. When examining roots in the four cardinal direction sectors, we observed differences between methodologies for length and number depending on root order but not volume.ConclusionsWe believe that TLS and QSM could facilitate rapid data collection, perhaps in situ, while providing quantitative accuracy, especially at the total root system level. If more detailed measures of root system architecture are desired, a TLS method would benefit from additional scans at differing perspectives, avoiding gravitational displacement to the extent possible, while subsampling roots by hand to calibrate and validate QSM models. Despite some unresolved logistical challenges, our results suggest that future use of TLS may hold promise for quantifying tree root system architecture in a rapid, replicable manner.

Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data.

Despite significant results achieved by Contrastive Language-Image Pretraining (CLIP) in zero-shot image recognition, limited effort has been made exploring its potential for zero-shot video recognition. This paper presents Open-VCLIP++, a simple yet effective framework that adapts CLIP to a strong zero-shot video classifier, capable of identifying novel actions and events during testing. Open-VCLIP++ minimally modifies CLIP to capture spatial-temporal relationships in videos, thereby creating a specialized video classifier while striving for generalization. We formally demonstrate that training Open-VCLIP++ is tantamount to continual learning with zero historical data. To address this problem, we introduce Interpolated Weight Optimization, a technique that leverages the advantages of weight interpolation during both training and testing. Furthermore, we build upon large language models to produce fine-grained video descriptions. These detailed descriptions are further aligned with video features, facilitating a better transfer of CLIP to the video domain. Our approach is evaluated on three widely used action recognition datasets, following a variety of zero-shot evaluation protocols. The results demonstrate that our method surpasses existing state-of-the-art techniques by significant margins. Specifically, we achieve zero-shot accuracy scores of 88.1%, 58.7%, and 81.2% on UCF, HMDB, and Kinetics-600 datasets respectively, outpacing the best-performing alternative methods by 8.5%, 8.2%, and 12.3%. We also evaluate our approach on the MSR-VTT video-text retrieval dataset, where it delivers competitive video-to-text and text-to-video retrieval performance, while utilizing substantially less fine-tuning data compared to other methods.

A lightweight complex-domain acoustic feature extraction method for rotating machinery fault detection

Abstract In the field of fault diagnosis for factory machinery systems, the development of deep learning methods has been hindered by the challenge of acquiring fault data, highlighting the need to extract noise robust features from limited labeled data. In this paper, a light and efficient complex-domain acoustic feature extraction method (CPFCN) is proposed for fault diagnosis in rotating machinery, which consists of a principal frequency filter (PFF) and stacked convolution network (SCN). The PFF filters out non-principal frequency noise to focus on the predominant frequency. The SCN is designed to effectively extract the amplitude and phase features, which can fully leverage the complex-domain information within the acoustic data. The experimental results show that the proposed CPFCN have 33% increasing in accuracy while 87% reduction in training time and 41% reduction in feature extraction time. Additionally, the proposed framework has improved the accuracy by 59% on the dataset with noise compared to the best-performing method in the experimental study, achieving stronger noise robustness in the case of limited samples.

Evaluation of polygenic scoring methods in five biobanks shows larger variation between biobanks than methods and finds benefits of ensemble learning

Methods of estimating polygenic scores (PGSs) from genome-wide association studies are increasingly utilized. However, independent method evaluation is lacking, and method comparisons are often limited. Here, we evaluate polygenic scores derived via seven methods in five biobank studies (totaling about 1.2 million participants) across 16 diseases and quantitative traits, building on a reference-standardized framework. We conducted meta-analyses to quantify the effects of method choice, hyperparameter tuning, method ensembling, and the target biobank on PGS performance. We found that no single method consistently outperformed all others. PGS effect sizes were more variable between biobanks than between methods within biobanks when methods were well tuned. Differences between methods were largest for the two investigated autoimmune diseases, seropositive rheumatoid arthritis and type 1 diabetes. For most methods, cross-validation was more reliable for tuning hyperparameters than automatic tuning (without the use of target data). For a given target phenotype, elastic net models combining PGS across methods (ensemble PGS) tuned in the UK Biobank provided consistent, high, and cross-biobank transferable performance, increasing PGS effect sizes (β coefficients) by a median of 5.0% relative to LDpred2 and MegaPRS (the two best-performing single methods when tuned with cross-validation). Our interactively browsable online-results and open-source workflow prspipe provide a rich resource and reference for the analysis of polygenic scoring methods across biobanks.

Explainable text-tabular models for predicting mortality risk in companion animals

As interest in using machine learning models to support clinical decision-making increases, explainability is an unequivocal priority for clinicians, researchers and regulators to comprehend and trust their results. With many clinical datasets containing a range of modalities, from the free-text of clinician notes to structured tabular data entries, there is a need for frameworks capable of providing comprehensive explanation values across diverse modalities. Here, we present a multimodal masking framework to extend the reach of SHapley Additive exPlanations (SHAP) to text and tabular datasets to identify risk factors for companion animal mortality in first-opinion veterinary electronic health records (EHRs) from across the United Kingdom. The framework is designed to treat each modality consistently, ensuring uniform and consistent treatment of features and thereby fostering predictability in unimodal and multimodal contexts. We present five multimodality approaches, with the best-performing method utilising PetBERT, a language model pre-trained on a veterinary dataset. Utilising our framework, we shed light for the first time on the reasons each model makes its decision and identify the inclination of PetBERT towards a more pronounced engagement with free-text narratives compared to BERT-base’s predominant emphasis on tabular data. The investigation also explores the important features on a more granular level, identifying distinct words and phrases that substantially influenced an animal’s life status prediction. PetBERT showcased a heightened ability to grasp phrases associated with veterinary clinical nomenclature, signalling the productivity of additional pre-training of language models.

Classification of white blood cells (leucocytes) from blood smear imagery using machine and deep learning models: A global scoping review.

Machine learning (ML) and deep learning (DL) models are being increasingly employed for medical imagery analyses, with both approaches used to enhance the accuracy of classification/prediction in the diagnoses of various cancers, tumors and bloodborne diseases. To date however, no review of these techniques and their application(s) within the domain of white blood cell (WBC) classification in blood smear images has been undertaken, representing a notable knowledge gap with respect to model selection and comparison. Accordingly, the current study sought to comprehensively identify, explore and contrast ML and DL methods for classifying WBCs. Following development and implementation of a formalized review protocol, a cohort of 136 primary studies published between January 2006 and May 2023 were identified from the global literature, with the most widely used techniques and best-performing WBC classification methods subsequently ascertained. Studies derived from 26 countries, with highest numbers from high-income countries including the United States (n = 32) and The Netherlands (n = 26). While WBC classification was originally rooted in conventional ML, there has been a notable shift toward the use of DL, and particularly convolutional neural networks (CNN), with 54.4% of identified studies (n = 74) including the use of CNNs, and particularly in concurrence with larger datasets and bespoke features e.g., parallel data pre-processing, feature selection, and extraction. While some conventional ML models achieved up to 99% accuracy, accuracy was shown to decrease in concurrence with decreasing dataset size. Deep learning models exhibited improved performance for more extensive datasets and exhibited higher levels of accuracy in concurrence with increasingly large datasets. Availability of appropriate datasets remains a primary challenge, potentially resolvable using data augmentation techniques. Moreover, medical training of computer science researchers is recommended to improve current understanding of leucocyte structure and subsequent selection of appropriate classification models. Likewise, it is critical that future health professionals be made aware of the power, efficacy, precision and applicability of computer science, soft computing and artificial intelligence contributions to medicine, and particularly in areas like medical imaging.

SR-RDFAN-LOG: Arbitrary-scale logging image super-resolution reconstruction based on residual dense feature aggregation

In the process of oil and gas exploration and development, ultrasonic logging imaging technology visually displays the lithology, structure, fractures, pores, and other characteristics of the wellbore. This technology has been widely applied in the exploration and development of oil and gas resources. However, the low circumferential sampling rate negatively affects the imaging quality and resolution during drilling logging. To address the issues of low resolution and blurriness in ultrasonic logging images caused by these factors, as well as the limitation of existing super-resolution reconstruction algorithms that can only reconstruct fixed scales, this paper presents an Image Super-Resolution Reconstruction Network called Residual Dense Feature Aggregation (SR-RDFAN-LOG), which is specifically designed to improve ultrasonic logging image resolution. The proposed network consists of two stages, including encoding and decoding. In the encoding stage, a local feature extraction module is initially designed to integrate contextual features. Subsequently, a residual dense module is introduced for extracting deep features. Finally, a spatial attention module is constructed to focus on high-frequency details. In the decoding stage, a multilayer perceptron is used to decode the feature maps and achieve super-resolution reconstruction at arbitrary scales. The method is tested on an ultrasonic logging images test set with in-distribution scale factors of × 2 and × 4, and out-of-distribution scale factors of × 8 and × 16. When the scaling factors are × 2 and × 4, compared to the best-performing method, the peak signal-to-noise ratio (PSNR) increased by 0.4892 dB and 0.3849 dB, respectively, the structural similarity (SSIM) values increased by 0.0002 and 0.0012, respectively, and the learned perceptual image patch similarity (LPIPS) decreased by 0.0001 and 0.0008, respectively. Compared to the bicubic interpolation method, the LPIPS decreased by 0.0001 and 0.0008 at scale factors of × 8 and × 16, PSNR increased by 1.9614 dB and 2.1002 dB, respectively, and SSIM values increased by 0.0476 and 0.033, respectively. The LPIPS values decreased by 0.2233 and 0.1070, respectively. The experimental results demonstrate that our method outperforms other mainstream methods, achieves super-resolution reconstruction at arbitrary scale, and provides support for subsequent fracture-cave evaluation and oil and gas exploration.

Bridging the gap: Active learning for efficient domain adaptation in object detection

In practical object detection computer vision applications, training commonly incorporates multiple data sources. Domain adaptation enhances the models’ capacity to generalize across different source and target domain data. Active learning, which serves as a bridge between unsupervised and supervised domain adaptation, is an area that has been underexplored for object detection. Our methods are specifically designed to tackle the challenges of selecting target domain images for annotation, complementing the existing source domain images. First, this paper formally describes active learning in domain adaptation for object detection. We propose three new selection strategies for active learning in domain adaptation for object detection. We utilize multiple machine learning models when selecting target domain data for annotation. In particular, our best-performing selection method utilizes a domain discriminator to predict the target domain likelihood of each sample. Additionally, we introduce an evaluation framework that evaluates per-box and per-image labeling efforts in the unique object detection application. Through extensive experiments we could show that our best-performing method could improve the active learning object detection performance over random selection by about 2.4% while requiring about 13% less annotated bounding boxes on the target domain.

Evaluating text classification: A benchmark study

This paper presents an impartial and extensive benchmark for text classification involving five different text classification tasks, 20 datasets, 11 different model architectures, and 42,800 algorithm runs. The five text classification tasks are fake news classification, topic detection, emotion detection, polarity detection, and sarcasm detection. While in practice, especially in Natural Language Processing (NLP), research tends to focus on the most sophisticated models, we hypothesize that this is not always necessary. Therefore, our main objective is to investigate whether the largest state-of-the-art (SOTA) models are always preferred, or in what cases simple methods can compete with complex models, i.e. for which dataset specifications and classification tasks. We assess the performance of different methods with varying complexity, ranging from simple statistical and machine learning methods to pretrained transformers like robustly optimized BERT (Bidirectional Encoder Representations from Transformers) pretraining approach (RoBERTa). This comprehensive benchmark is lacking in existing literature, with research mainly comparing similar types of methods. Furthermore, with increasing awareness of the ecological impacts of extensive computational resource usage, this comparison is both critical and timely.We find that overall, bidirectional long short-term memory (LSTM) networks are ranked as the best-performing method albeit not statistically significantly better than logistic regression and RoBERTa. Overall, we cannot conclude that simple methods perform worse although this depends mainly on the classification task. Concretely, we find that for fake news classification and topic detection, simple techniques are the best-ranked models and consequently, it is not necessary to train complicated neural network architectures for these classification tasks. Moreover, we also find a negative correlation between F1 performance and complexity for the smallest datasets (with dataset size less than 10,000). Finally, the different models’ results are analyzed in depth to explain the model decisions, which is an increasing requirement in the field of text classification.

Experimental study on short-text clustering using transformer-based semantic similarity measure

Sentence clustering plays a central role in various text-processing activities and has received extensive attention for measuring semantic similarity between compared sentences. However, relatively little focus has been placed on evaluating clustering performance using available similarity measures that adopt low-dimensional continuous representations. Such representations are crucial in domains like sentence clustering, where traditional word co-occurrence representations often achieve poor results when clustering semantically similar sentences that share no common words. This article presents a new implementation that incorporates a sentence similarity measure based on the notion of embedding representation for evaluating the performance of three types of text clustering methods: partitional clustering, hierarchical clustering, and fuzzy clustering, on standard textual datasets. This measure derives its semantic information from pre-training models designed to simulate human knowledge about words in natural language. The article also compares the performance of the used similarity measure by training it on two state-of-the-art pre-training models to investigate which yields better results. We argue that the superior performance of the selected clustering methods stems from their more effective use of the semantic information offered by this embedding-based similarity measure. Furthermore, we use hierarchical clustering, the best-performing method, for a text summarization task and report the results. The implementation in this article demonstrates that incorporating the sentence embedding measure leads to significantly improved performance in both text clustering and text summarization tasks.