Best-performing Method Research Articles

An Innovative Neighbor Attention Mechanism Based on Coordinates for the Recognition of Facial Expressions.

For solving the facial expression recognition (FER) problem, we introduce a novel feature extractor called the coordinate-based neighborhood attention mechanism (CNAM), which uses the coordinate attention (CA) method to capture the directional relationships in separate horizontal and vertical directions, the input features from a preprocessing unit, and then passes this to two residual blocks, one consisting of the neighborhood attention (NA) mechanism, which captures the local interaction of features within the neighborhood of a feature vector, while the other one contains a channel attention implemented by a multilayer perceptron (MLP). We apply the feature extractor, the CNAM module, to four FER benchmark datasets, namely, RAF-DB, AffectNet(7cls), AffectNet(8cls), and CK+, and through qualitative and quantitative analysis techniques, we conclude that the insertion of the CNAM module could decrease the intra-cluster distances and increase the inter-cluster distances among the high-dimensional feature vectors. The CNAM compares well with other state-of-the-art (SOTA) methods, being the best-performing method for the AffectNet(7cls) and CK+ datasets, while for the RAF-DB and AffectNet(8cls) datasets, its performance is among the top-performing SOTA methods.

Enhancing Cover Management Factor Classification Through Imbalanced Data Resolution

This study addresses the persistent challenge of class imbalance in land use and land cover (LULC) classification within the Shihmen Reservoir watershed in Taiwan, where LULC is used to map the Cover Management factor (C-factor). The dominance of forests in the LULC categories leads to an imbalanced dataset, resulting in poor prediction performance for minority classes when using machine learning techniques. To overcome this limitation, we applied the Synthetic Minority Over-sampling Technique (SMOTE) and the 90-model SMOTE-variants package in Python to balance the dataset. Due to the multi-class nature of the data and memory constraints, 42 models were successfully used to create a balanced dataset, which was then integrated with a Random Forest algorithm for C-factor classification. The results show a marked improvement in model accuracy across most SMOTE variants, with the Selected Synthetic Minority Over-sampling Technique (Selected_SMOTE) emerging as the best-performing method, achieving an overall accuracy of 0.9524 and a sensitivity of 0.6892. Importantly, the previously observed issue of poor minority class prediction was resolved using the balanced dataset. This study provides a robust solution to the class imbalance issue in C-factor classification, demonstrating the effectiveness of SMOTE variants and the Random Forest algorithm in improving model performance and addressing imbalanced class distributions. The success of Selected_SMOTE underscores the potential of balanced datasets in enhancing machine learning outcomes, particularly in datasets dominated by a majority class. Additionally, by addressing imbalance in LULC classification, this research contributes to Sustainable Development Goal 15, which focuses on the protection, restoration, and sustainable use of terrestrial ecosystems.

Feature-fused residual network for time series classification

In various fields such as healthcare and transportation, accurately classifying time series data can provide important support for decision-making. To further improve the accuracy of time series classification, we propose a Feature-fused Residual Network based on Multi-scale Signed Recurrence Plot (MSRP-FFRN). This method transforms one-dimensional time series into two-dimensional images, representing the temporal correlation of time series in a two-dimensional space and revealing hidden details within the data. To enhance these details further, we extract multi-scale features by setting receptive fields of different sizes and using adaptive network depths, which improves image quality. To evaluate the performance of this method, we conducted experiments on 43 UCR datasets and compared it with nine state-of-the-art baseline methods. The experimental results show that MSRP-FFRN ranks first on critical difference diagram, achieving the highest accuracy on 18 datasets with an average accuracy of 89.9%, making it the best-performing method overall. Additionally, the effectiveness of this method is further validated through metrics such as Precision, Recall, and F1 score. Results from ablation experiments also highlight the efficacy of the improvements made by MSRP-FFRN.

FairDPFL-SCS: Fair Dynamic Personalized Federated Learning with strategic client selection for improved accuracy and fairness

Personalized federated learning (PFL) addresses the significant challenge of non-independent and identically distributed (non-IID) data across clients in federated learning (FL). Our proposed framework, “FairDPFL-SCS: Fair Dynamic Personalized Federated Learning with Strategic Client Selection”, marks a notable advancement in this domain. By integrating dynamic learning rate adjustments and a strategic client selection mechanism, our approach effectively mitigates the challenges posed by non-IID data while enhancing model personalization, fairness, and efficiency. We evaluated FairDPFL-SCS using standard datasets, including MNIST, FashionMNIST, and SVHN, employing architectures like VGG and CNN. Our model achieved impressive results, attaining 99.04% accuracy on MNIST, 89.19% on FashionMNIST, and 90.9% on SVHN. These results represent a substantial improvement over existing methods, including a highest increase of 16.74% in accuracy on SVHN when compared to the best-performing benchmark methods. In particular, our method also demonstrated lower fairness variance, presenting the importance of fairness in model personalization, a frequently overlooked aspect in FL research. Through extensive experiments, we validate the superior performance of FairDPFL-SCS compared to benchmark PFL approaches, highlighting significant improvements over state-of-the-art methods. This work represents a promising step forward in the field of federated learning, offering a comprehensive solution to the challenges presented by non-IID data while prioritizing fairness and efficiency in model personalization.

Segmentation-based truncated-SVD for effective feature extraction in hyperspectral image classification

ABSTRACT Remote sensing hyperspectral images (HSIs) are rich sources of information about land cover captured across hundreds of narrow, contiguous spectral wavelength bands. However, using the entire original HSI for practical applications can lead to suboptimal classification accuracy. To address this, band reduction techniques, categorized as feature extraction and feature selection methods, are employed to enhance classification results. One commonly used feature extraction approach for HSIs is Principal Component Analysis (PCA). However, PCA may fall short of capturing the local and specific characteristics present in the HSI data. In this paper, we introduce two novel feature extraction methods: Segmented Truncated Singular Value Decomposition (STSVD) and Spectrally Segmented Truncated Singular Value Decomposition (SSTSVD) to improve classification performance. Segmentation is carried out based on highly correlated bands’ segments and spectral bands’ segments within the HSI data. Our study evaluates and compares these newly proposed methods against classical feature extraction methods, including PCA, Incremental PCA, Sparse-PCA, Kernel PCA, Segmented-PCA (SPCA), and Truncated Singular Value Decomposition (TSVD). We perform this analysis on three distinct HSI datasets, namely the Indian Pines HSI, the Pavia University HSI, and the Kennedy Space Center HSI, using per-pixel Support Vector Machine (SVM) and Random Forest (RF) classification. The experimental results demonstrate the superiority of our proposed methods for all three datasets. The best-performing feature extraction methods when classification is performed using an SVM classifier are STSVD3 (89.03%), SSTSVD2 (95.55%), and STSVD3 (97.74%) for the Indian Pines, Pavia University, and Kennedy Space Center datasets, respectively. Similarly, for the RF classifier, the best-performing feature extraction methods are SSTSVD4 (88.98%), SSTSVD3 (96.04%), and SSTSVD4 (96.09%) for Indian Pines, Pavia University, and Kennedy Space Center datasets, respectively.

Comparison of whole-brain task-modulated functional connectivity methods for fMRI task connectomics

Higher brain functions require flexible integration of information across widely distributed brain regions depending on the task context. Resting-state functional magnetic resonance imaging (fMRI) has provided substantial insight into large-scale intrinsic brain network organisation, yet the principles of rapid context-dependent reconfiguration of that intrinsic network organisation are much less understood. A major challenge for task connectome mapping is the absence of a gold standard for deriving whole-brain task-modulated functional connectivity matrices. Here, we perform biophysically realistic simulations to control the ground-truth task-modulated functional connectivity over a wide range of experimental settings. We reveal the best-performing methods for different types of task designs and their fundamental limitations. Importantly, we demonstrate that rapid (100 ms) modulations of oscillatory neuronal synchronisation can be recovered from sluggish haemodynamic fluctuations even at typically low fMRI temporal resolution (2 s). Finally, we provide practical recommendations on task design and statistical analysis to foster task connectome mapping.

Comparing Different Methods for the Diagnosis of Liver Steatosis: What Are the Best Diagnostic Tools?

Due to the ongoing organ shortage, marginal grafts with steatosis are more frequently used in liver transplantation, leading to higher occurrences of graft dysfunction. A histological analysis is the gold standard for the quantification of liver steatosis (LS), but has its drawbacks: it is an invasive method that varies from one pathologist to another and is not available in every hospital at the time of organ procurement. This study aimed to compare non-invasive diagnostic tools to a histological analysis for the quantification of liver steatosis. Male C57BL6J mice were fed with a methioninecholine-deficient (MCD) diet for 14 days or 28 days to induce LS, and were compared to a control group of animals fed with a normal diet. The following non-invasive techniques were performed and compared to the histological quantification of liver steatosis: magnetic resonance spectroscopy (MRS), CARS microscopy, 99mTc MIBI SPECT imaging, and a new near-infrared spectrometer (NIR-SG1). After 28 days on the MCD diet, an evaluation of LS showed ≥30% macrovesicular steatosis. High correlations were found between the NIR-SG1 and the blinded pathologist analysis (R2 = 0.945) (p = 0.001), and between the CARS microscopy (R2 = 0.801 (p < 0.001); MRS, R2 = 0.898 (p < 0.001)) and the blinded pathologist analysis. The ROC curve analysis showed that the area under the curve (AUC) was 1 for both the NIR-SG1 and MRS (p = 0.021 and p < 0.001, respectively), while the AUC = 0.910 for the Oil Red O stain (p < 0.001) and the AUC = 0.865 for the CARS microscopy (p < 0.001). The AUC for the 99mTc MIBI SPECT was 0.640 (p = 0.013), and this was a less discriminating technique for LS quantification. The best-performing non-invasive methods for LS quantification are MRS, CARS microscopy, and the NIR-SG1. The NIR-SG1 is particularly appropriate for clinical practice and needs to be validated by clinical studies on liver grafts.

Comparison of whole-brain task-modulated functional connectivity methods for fMRI task connectomics.

Higher brain functions require flexible integration of information across widely distributed brain regions depending on the task context. Resting-state functional magnetic resonance imaging (fMRI) has provided substantial insight into large-scale intrinsic brain network organisation, yet the principles of rapid context-dependent reconfiguration of that intrinsic network organisation are much less understood. A major challenge for task connectome mapping is the absence of a gold standard for deriving whole-brain task-modulated functional connectivity matrices. Here, we perform biophysically realistic simulations to control the ground-truth task-modulated functional connectivity over a wide range of experimental settings. We reveal the best-performing methods for different types of task designs and their fundamental limitations. Importantly, we demonstrate that rapid (100 ms) modulations of oscillatory neuronal synchronisation can be recovered from sluggish haemodynamic fluctuations even at typically low fMRI temporal resolution (2 s). Finally, we provide practical recommendations on task design and statistical analysis to foster task connectome mapping.

Optimizing and benchmarking polygenic risk scores with GWAS summary statistics

BackgroundPolygenic risk score (PRS) is a major research topic in human genetics. However, a significant gap exists between PRS methodology and applications in practice due to often unavailable individual-level data for various PRS tasks including model fine-tuning, benchmarking, and ensemble learning.ResultsWe introduce an innovative statistical framework to optimize and benchmark PRS models using summary statistics of genome-wide association studies. This framework builds upon our previous work and can fine-tune virtually all existing PRS models while accounting for linkage disequilibrium. In addition, we provide an ensemble learning strategy named PUMAS-ensemble to combine multiple PRS models into an ensemble score without requiring external data for model fitting. Through extensive simulations and analysis of many complex traits in the UK Biobank, we demonstrate that this approach closely approximates gold-standard analytical strategies based on external validation, and substantially outperforms state-of-the-art PRS methods.ConclusionsOur method is a powerful and general modeling technique that can continue to combine the best-performing PRS methods out there through ensemble learning and could become an integral component for all future PRS applications.

Interpretable Spatial–Temporal Graph Convolutional Network for System Log Anomaly Detection

To ensure seamless information flow and operational integrity, computer systems need effectively to manage their system logs, but the expansion in their scale and complexity makes it hard to detect anomalies. Current methodologies exhibit deficiencies, including inefficiencies in handling abnormal sequences, lack of interpretability, and limited consideration of both temporal and spatial information. To improve, this paper develops a semi-supervised graph neural network model termed the Interpretable Spatial–Temporal Graph Convolutional Network (IST-GCN). By integrating temporal and event similarity perspectives, the IST-GCN harnesses directed and undirected graphs to capture the temporal and spatial aspects of system log events. Hence, the IST-GCN offers temporal and spatial interpretability. Further, a lightweight feature regularization technique is developed to enhance interpretability in both time and space domains, and thus facilitates anomaly detection efficiently. Comprehensive testing verifies that the IST-GCN approach surpasses nearly all state-of-the-art methods across five public log anomaly datasets. On average, IST-GCN improves Average Precision (AP) by approximately 3% and ROC AUC (RC) by about 4% compared to the best-performing baseline methods, underscoring its effectiveness and robustness.

Keywords-enhanced Contrastive Learning Model for travel recommendation

Travel recommendation aims to infer travel intentions of users by analyzing their historical behaviors on Online Travel Agencies (OTAs). However, crucial keywords in clicked travel product titles, such as destination and itinerary duration, indicating tourists’ intentions, are often overlooked. Additionally, most previous studies only consider stable long-term user interests or temporary short-term user preferences, making the recommendation performance unreliable. To mitigate these constraints, this paper proposes a novel Keywords-enhanced Contrastive Learning Model (KCLM). KCLM simultaneously implements personalized travel recommendation and keywords generation tasks, integrating long-term and short-term user preferences within both tasks. Furthermore, we design two kinds of contrastive learning tasks for better user and travel product representation learning. The preference contrastive learning aims to bridge the gap between long-term and short-term user preferences. The multi-view contrastive learning focuses on modeling the coarse-grained commonality between clicked products and their keywords. Extensive experiments are conducted on two tourism datasets and a large-scale e-commerce dataset. The experimental results demonstrate that KCLM achieves substantial gains in both metrics compared to the best-performing baseline methods. Specifically, HR@20 improved by 5.79%–14.13%, MRR@20 improved by 6.57%–18.50%. Furthermore, to have an intuitive understanding of the keyword generation by the KCLM model, we provide a case study for several randomized examples.

Empirical and physical modelling of soil erosion in agricultural hillslopes

Abstract Soil erosion is a complex and highly heterogeneous process with a wide range of environmental and economic impacts. Its estimation is particularly challenging and modelling is typically used for erosion estimation over large areas. The aim of this study was to compare the two leading empirical and physical erosion estimation models, i.e. the Revised Universal Soil Loss Equation (RUSLE) and the Water Erosion Prediction Project (WEPP). The models were calibrated and validated using data collected from field experiments conducted in agricultural lands of Mexico. The simulated rainfall experiments involved measuring erosion from field plots subjected to four tillage systems (No crop, Conventional tillage, Conventional tillage + residues, and Handspike) under two antecedent soil moisture conditions (dry and wet). Different calibration approaches based on the factors K and C for RUSLE, and interrill erodibility and hydraulic conductivity in WEPP were tested. The best-performing methods in RUSLE involved measuring the K factor and adopting the recommended C factor by the National Forestry Commission of Mexico. In WEPP, the best results were obtained when interrill erodibility was estimated from experimental measurements. Overall, RUSLE outperformed WEPP in most of the treatments except for CT under WAMC.

Reliable estimation of tree branch lengths using deep neural networks.

A phylogenetic tree represents hypothesized evolutionary history for a set of taxa. Besides the branching patterns (i.e., tree topology), phylogenies contain information about the evolutionary distances (i.e. branch lengths) between all taxa in the tree, which include extant taxa (external nodes) and their last common ancestors (internal nodes). During phylogenetic tree inference, the branch lengths are typically co-estimated along with other phylogenetic parameters during tree topology space exploration. There are well-known regions of the branch length parameter space where accurate estimation of phylogenetic trees is especially difficult. Several novel studies have recently demonstrated that machine learning approaches have the potential to help solve phylogenetic problems with greater accuracy and computational efficiency. In this study, as a proof of concept, we sought to explore the possibility of machine learning models to predict branch lengths. To that end, we designed several deep learning frameworks to estimate branch lengths on fixed tree topologies from multiple sequence alignments or its representations. Our results show that deep learning methods can exhibit superior performance in some difficult regions of branch length parameter space. For example, in contrast to maximum likelihood inference, which is typically used for estimating branch lengths, deep learning methods are more efficient and accurate. In general, we find that our neural networks achieve similar accuracy to a Bayesian approach and are the best-performing methods when inferring long branches that are associated with distantly related taxa. Together, our findings represent a next step toward accurate, fast, and reliable phylogenetic inference with machine learning approaches.

A comparison of scRNA-seq annotation methods based on experimentally labeled immune cell subtype dataset.

Cell-type annotation is a critical step in single-cell data analysis. With the development of numerous cell annotation methods, it is necessary to evaluate these methods to help researchers use them effectively. Reference datasets are essential for evaluation, but currently, the cell labels of reference datasets mainly come from computational methods, which may have computational biases and may not reflect the actual cell-type outcomes. This study first constructed an experimentally labeled immune cell-subtype single-cell dataset of the same batch and systematically evaluated 18 cell annotation methods. We assessed those methods under five scenarios, including intra-dataset validation, immune cell-subtype validation, unsupervised clustering, inter-dataset annotation, and unknown cell-type prediction. Accuracy and ARI were evaluation metrics. The results showed that SVM, scBERT, and scDeepSort were the best-performing supervised methods. Seurat was the best-performing unsupervised clustering method, but it couldn't fully fit the actual cell-type distribution. Our results indicated that experimentally labeled immune cell-subtype datasets revealed the deficiencies of unsupervised clustering methods and provided new dataset support for supervised methods.

Graph-Based Traffic Forecasting with the Dynamics of Road Symmetry and Capacity Performance

Symmetry in traffic patterns is a fundamental aspect of intelligent transportation systems, aiming to precisely predict traffic flow in real time despite the complex interplay of spatial and temporal factors. This paper presents a novel method of traffic forecasting that incorporates parameters related to road symmetry into a Graph Convolution Network model. Our model is crafted to dynamically adjust to real-time changes in road conditions, including the presence of symmetric and asymmetric road layouts, which substantially influence traffic flow. We have developed a GCN model that not only accounts for standard traffic flow metrics but also integrates a matrix representing road symmetry. The model undergoes training and validation on the METR-LA dataset, showcasing a significant enhancement in prediction accuracy. In the comparative analysis of state-of-the-art methods, our model demonstrated a significant enhancement in performance, achieving 30.68% improvement in Mean Squared Error (MSE) and a 24.28% improvement in Mean Absolute Error (MAE) over the best-performing method. The implications of our research are profound for urban planners, traffic management systems, and navigation service providers, as it offers a more dependable tool for forecasting traffic conditions, aiding in road design, and refining route planning strategies based on the symmetry of road networks.

Terrestrial laser scanning and low magnetic field digitization yield similar architectural coarse root traits for 32-year-old Pinus ponderosa trees

BackgroundUnderstanding how trees develop their root systems is crucial for the comprehension of how wildland and urban forest ecosystems plastically respond to disturbances such as harvest, fire, and climate change. The interplay between the endogenously determined root traits and the response to environmental stimuli results in tree adaptations to biotic and abiotic factors, influencing stability, carbon allocation, and nutrient uptake. Combining the three-dimensional structure of the root system, with root morphological trait information promotes a robust understanding of root function and adaptation plasticity. Low Magnetic Field Digitization coupled with AMAPmod (botAnique et Modelisation de l’Architecture des Plantes) software has been the best-performing method for describing root system architecture and providing reliable measurements of coarse root traits, but the pace and scale of data collection remain difficult. Instrumentation and applications related to Terrestrial Laser Scanning (TLS) have advanced appreciably, and when coupled with Quantitative Structure Models (QSM), have shown some potential toward robust measurements of tree root systems. Here we compare, we believe for the first time, these two methodologies by analyzing the root system of 32-year-old Pinus ponderosa trees.ResultsIn general, at the total root system level and by root-order class, both methods yielded comparable values for the root traits volume, length, and number. QSM for each root trait was highly sensitive to the root size (i.e., input parameter PatchDiam) and models were optimized when discrete PatchDiam ranges were specified for each trait. When examining roots in the four cardinal direction sectors, we observed differences between methodologies for length and number depending on root order but not volume.ConclusionsWe believe that TLS and QSM could facilitate rapid data collection, perhaps in situ, while providing quantitative accuracy, especially at the total root system level. If more detailed measures of root system architecture are desired, a TLS method would benefit from additional scans at differing perspectives, avoiding gravitational displacement to the extent possible, while subsampling roots by hand to calibrate and validate QSM models. Despite some unresolved logistical challenges, our results suggest that future use of TLS may hold promise for quantifying tree root system architecture in a rapid, replicable manner.

Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data.

Despite significant results achieved by Contrastive Language-Image Pretraining (CLIP) in zero-shot image recognition, limited effort has been made exploring its potential for zero-shot video recognition. This paper presents Open-VCLIP++, a simple yet effective framework that adapts CLIP to a strong zero-shot video classifier, capable of identifying novel actions and events during testing. Open-VCLIP++ minimally modifies CLIP to capture spatial-temporal relationships in videos, thereby creating a specialized video classifier while striving for generalization. We formally demonstrate that training Open-VCLIP++ is tantamount to continual learning with zero historical data. To address this problem, we introduce Interpolated Weight Optimization, a technique that leverages the advantages of weight interpolation during both training and testing. Furthermore, we build upon large language models to produce fine-grained video descriptions. These detailed descriptions are further aligned with video features, facilitating a better transfer of CLIP to the video domain. Our approach is evaluated on three widely used action recognition datasets, following a variety of zero-shot evaluation protocols. The results demonstrate that our method surpasses existing state-of-the-art techniques by significant margins. Specifically, we achieve zero-shot accuracy scores of 88.1%, 58.7%, and 81.2% on UCF, HMDB, and Kinetics-600 datasets respectively, outpacing the best-performing alternative methods by 8.5%, 8.2%, and 12.3%. We also evaluate our approach on the MSR-VTT video-text retrieval dataset, where it delivers competitive video-to-text and text-to-video retrieval performance, while utilizing substantially less fine-tuning data compared to other methods.

A lightweight complex-domain acoustic feature extraction method for rotating machinery fault detection

Abstract In the field of fault diagnosis for factory machinery systems, the development of deep learning methods has been hindered by the challenge of acquiring fault data, highlighting the need to extract noise robust features from limited labeled data. In this paper, a light and efficient complex-domain acoustic feature extraction method (CPFCN) is proposed for fault diagnosis in rotating machinery, which consists of a principal frequency filter (PFF) and stacked convolution network (SCN). The PFF filters out non-principal frequency noise to focus on the predominant frequency. The SCN is designed to effectively extract the amplitude and phase features, which can fully leverage the complex-domain information within the acoustic data. The experimental results show that the proposed CPFCN have 33% increasing in accuracy while 87% reduction in training time and 41% reduction in feature extraction time. Additionally, the proposed framework has improved the accuracy by 59% on the dataset with noise compared to the best-performing method in the experimental study, achieving stronger noise robustness in the case of limited samples.

Evaluation of polygenic scoring methods in five biobanks shows larger variation between biobanks than methods and finds benefits of ensemble learning

Methods of estimating polygenic scores (PGSs) from genome-wide association studies are increasingly utilized. However, independent method evaluation is lacking, and method comparisons are often limited. Here, we evaluate polygenic scores derived via seven methods in five biobank studies (totaling about 1.2 million participants) across 16 diseases and quantitative traits, building on a reference-standardized framework. We conducted meta-analyses to quantify the effects of method choice, hyperparameter tuning, method ensembling, and the target biobank on PGS performance. We found that no single method consistently outperformed all others. PGS effect sizes were more variable between biobanks than between methods within biobanks when methods were well tuned. Differences between methods were largest for the two investigated autoimmune diseases, seropositive rheumatoid arthritis and type 1 diabetes. For most methods, cross-validation was more reliable for tuning hyperparameters than automatic tuning (without the use of target data). For a given target phenotype, elastic net models combining PGS across methods (ensemble PGS) tuned in the UK Biobank provided consistent, high, and cross-biobank transferable performance, increasing PGS effect sizes (β coefficients) by a median of 5.0% relative to LDpred2 and MegaPRS (the two best-performing single methods when tuned with cross-validation). Our interactively browsable online-results and open-source workflow prspipe provide a rich resource and reference for the analysis of polygenic scoring methods across biobanks.

Explainable text-tabular models for predicting mortality risk in companion animals

As interest in using machine learning models to support clinical decision-making increases, explainability is an unequivocal priority for clinicians, researchers and regulators to comprehend and trust their results. With many clinical datasets containing a range of modalities, from the free-text of clinician notes to structured tabular data entries, there is a need for frameworks capable of providing comprehensive explanation values across diverse modalities. Here, we present a multimodal masking framework to extend the reach of SHapley Additive exPlanations (SHAP) to text and tabular datasets to identify risk factors for companion animal mortality in first-opinion veterinary electronic health records (EHRs) from across the United Kingdom. The framework is designed to treat each modality consistently, ensuring uniform and consistent treatment of features and thereby fostering predictability in unimodal and multimodal contexts. We present five multimodality approaches, with the best-performing method utilising PetBERT, a language model pre-trained on a veterinary dataset. Utilising our framework, we shed light for the first time on the reasons each model makes its decision and identify the inclination of PetBERT towards a more pronounced engagement with free-text narratives compared to BERT-base’s predominant emphasis on tabular data. The investigation also explores the important features on a more granular level, identifying distinct words and phrases that substantially influenced an animal’s life status prediction. PetBERT showcased a heightened ability to grasp phrases associated with veterinary clinical nomenclature, signalling the productivity of additional pre-training of language models.