Layer-wise Relevance Propagation Research Articles

Deep neural network heatmaps capture Alzheimer’s disease patterns reported in a large meta-analysis of neuroimaging studies

Deep neural networks currently provide the most advanced and accurate machine learning models to distinguish between structural MRI scans of subjects with Alzheimer’s disease and healthy controls. Unfortunately, the subtle brain alterations captured by these models are difficult to interpret because of the complexity of these multi-layer and non-linear models. Several heatmap methods have been proposed to address this issue and analyze the imaging patterns extracted from the deep neural networks, but no quantitative comparison between these methods has been carried out so far. In this work, we explore these questions by deriving heatmaps from Convolutional Neural Networks (CNN) trained using T1 MRI scans of the ADNI data set and by comparing these heatmaps with brain maps corresponding to Support Vector Machine (SVM) activation patterns. Three prominent heatmap methods are studied: Layer-wise Relevance Propagation (LRP), Integrated Gradients (IG), and Guided Grad-CAM (GGC). Contrary to prior studies where the quality of heatmaps was visually or qualitatively assessed, we obtained precise quantitative measures by computing overlap with a ground-truth map from a large meta-analysis that combined 77 voxel-based morphometry (VBM) studies independently from ADNI. Our results indicate that all three heatmap methods were able to capture brain regions covering the meta-analysis map and achieved better results than SVM activation patterns. Among them, IG produced the heatmaps with the best overlap with the independent meta-analysis.

Mutual Information-Driven Subject-Invariant and Class-Relevant Deep Representation Learning in BCI.

In recent years, deep learning-based feature representation methods have shown a promising impact on electroencephalography (EEG)-based brain-computer interface (BCI). Nonetheless, owing to high intra- and inter-subject variabilities, many studies on decoding EEG were designed in a subject-specific manner by using calibration samples, with no concern of its practical use, hampered by time-consuming steps and a large data requirement. To this end, recent studies adopted a transfer learning strategy, especially domain adaptation techniques. Among those, we have witnessed the potential of adversarial learning-based transfer learning in BCIs. In the meantime, it is known that adversarial learning-based domain adaptation methods are prone to negative transfer that disrupts learning generalized feature representations, applicable to diverse domains, for example, subjects or sessions in BCIs. In this article, we propose a novel framework that learns class-relevant and subject-invariant feature representations in an information-theoretic manner, without using adversarial learning. To be specific, we devise two operational components in a deep network that explicitly estimate mutual information between feature representations: 1) to decompose features in an intermediate layer into class-relevant and class-irrelevant ones and 2) to enrich class-discriminative feature representation. On two large EEG datasets, we validated the effectiveness of our proposed framework by comparing with several comparative methods in performance. Furthermore, we conducted rigorous analyses by performing an ablation study in regard to the components in our network, explaining our model's decision on input EEG signals via layer-wise relevance propagation, and visualizing the distribution of learned features via t-SNE.

Relevance-based channel selection in motor imagery brain–computer interface

Objective. Channel selection in the electroencephalogram (EEG)-based brain–computer interface (BCI) has been extensively studied for over two decades, with the goal being to select optimal subject-specific channels that can enhance the overall decoding efficacy of the BCI. With the emergence of deep learning (DL)-based BCI models, there arises a need for fresh perspectives and novel techniques to conduct channel selection. In this regard, subject-independent channel selection is relevant, since DL models trained using cross-subject data offer superior performance, and the impact of inherent inter-subject variability of EEG characteristics on subject-independent DL training is not yet fully understood. Approach. Here, we propose a novel methodology for implementing subject-independent channel selection in DL-based motor imagery (MI)-BCI, using layer-wise relevance propagation (LRP) and neural network pruning. Experiments were conducted using Deep ConvNet and 62-channel MI data from the Korea University EEG dataset. Main Results. Using our proposed methodology, we achieved a 61% reduction in the number of channels without any significant drop (p = 0.09) in subject-independent classification accuracy, due to the selection of highly relevant channels by LRP. LRP relevance-based channel selections provide significantly better accuracies compared to conventional weight-based selections while using less than 40% of the total number of channels, with differences in accuracies ranging from 5.96% to 1.72%. The performance of the adapted sparse-LRP model using only 16% of the total number of channels is similar to that of the adapted baseline model (p = 0.13). Furthermore, the accuracy of the adapted sparse-LRP model using only 35% of the total number of channels exceeded that of the adapted baseline model by 0.53% (p = 0.81). Analyses of channels chosen by LRP confirm the neurophysiological plausibility of selection, and emphasize the influence of motor, parietal, and occipital channels in MI-EEG classification. Significance. The proposed method addresses a traditional issue in EEG-BCI decoding, while being relevant and applicable to the latest developments in the field of BCI. We believe that our work brings forth an interesting and important application of model interpretability as a problem-solving technique.

Single-cell gene regulatory network prediction by explainable AI.

The molecular heterogeneity of cancer cells contributes to the often partial response to targeted therapies and relapse of disease due to the escape of resistant cell populations. While single-cell sequencing has started to improve our understanding of this heterogeneity, it offers a mostly descriptive view on cellular types and states. To obtain more functional insights, we propose scGeneRAI, an explainable deep learning approach that uses layer-wise relevance propagation (LRP) to infer gene regulatory networks from static single-cell RNA sequencing data for individual cells. We benchmark our method with synthetic data and apply it to single-cell RNA sequencing data of a cohort of human lung cancers. From the predicted single-cell networks our approach reveals characteristic network patterns for tumor cells and normal epithelial cells and identifies subnetworks that are observed only in (subgroups of) tumor cells of certain patients. While current state-of-the-art methods are limited by their ability to only predict average networks for cell populations, our approach facilitates the reconstruction of networks down to the level of single cells which can be utilized to characterize the heterogeneity of gene regulation within and across tumors.

WSODet: A Weakly Supervised Oriented Detector for Aerial Object Detection

In contrast to natural objects, aerial targets are usually non-axis aligned with arbitrary orientations. However, mainstream Weakly Supervised Object Detection (WSOD) methods can only predict horizontal bounding boxes (HBB) from existing proposals generated by offline algorithms. To predict oriented bounding boxes (OBB) for aerial targets while testing images end-to-end without proposals, WSODet is designed leveraging on Layer-wise Relevance Propagation (LRP) and Point Set Representation (RepPoints). To be specific, based on the mainstream WSOD framework, Layer-wise Relevance Propagation on Multiple Instance Learning branch (MIL-LRP) is conducted to decrease the uncertainty and ambiguity of feature map. Then, a pseudo oriented label generation algorithm is designed to obtain OBB pseudo labels, which serve as supervision to train an Oriented RepPoint Net under the guidance of improved oriented loss function. During the test, input images are sent to oriented RepPoint branch to obtain OBB predictions without proposals. Extensive experiments on DIOR, NWPU VHR-10.v2 and HRSC2016 datasets demonstrate the effectiveness of our method to predict precise oriented aerial objects, achieving 22.2%, 46.5% and 43.3% mAP, respectively. Moreover, training jointly with oriented RepPoint branch boosts the results of original WSOD framework compared with existing WSOD methods even if there is no specific design for original structure.

Channel Contribution In Deep Learning Based Automatic Sleep Scoring - How Many Channels Do We Need?

Machine learning based sleep scoring methods aim to automate the process of annotating polysomnograms with sleep stages. Although sleep signals of multiple modalities and channels should contain more information according to sleep guidelines, most multi-channel multi-modal models in the literature showed only a little performance improvement compared to single-channel EEG models and sometimes even failed to outperform them. In this paper, we investigate whether the high performance of single-channel EEG models can be attributed to specific model features in their deep learning architectures and to which extent multi-channel multi-modal models take the information from different channels of modalities into account. First, we transfer the model features from single-channel EEG models, such as combinations of small and large filters in CNNs, to multi-channel multi-modal models and measure their impacts. Second, we employ two explainability methods, the layer-wise relevance propagation as post-hoc and the embedded channel attention network as intrinsic interpretability methods, to measure the contribution of different channels on predictive performance. We find that i) single-channel model features can improve the performance of multi-channel multi-modal models and ii) multi-channel multi-modal models focus on one important channel per modality and use the remaining channels to complement the information of the focused channels. Our results suggest that more advanced methods for aggregating channel information using complementary information from other channels may improve sleep scoring performance for multi-channel multi-modal models.

An Interpretable Deep Learning Approach for Biomarker Detection in LC-MS Proteomics Data.

Analyzing mass spectrometry-based proteomics data with deep learning (DL) approaches poses several challenges due to the high dimensionality, low sample size, and high level of noise. Additionally, DL-based workflows are often hindered to be integrated into medical settings due to the lack of interpretable explanation. We present DLearnMS, a DL biomarker detection framework, to address these challenges on proteomics instances of liquid chromatography-mass spectrometry (LC-MS) - a well-established tool for quantifying complex protein mixtures. Our DLearnMS framework learns the clinical state of LC-MS data instances using convolutional neural networks. Based on the trained neural networks, we show how biomarkers can be identified using layer-wise relevance propagation. This enables detecting discriminating regions of the data and the design of more robust networks. One of the main advantages over other established methods is that no explicit preprocessing step is needed in our DLearnMS framework. Our evaluation shows that DLearnMS outperforms conventional LC-MS biomarker detection approaches in identifying fewer false positive peaks while maintaining a comparable amount of true positives peaks. Code availability: The code is available from the following GIT repository: https://github.com/SaharIravani/DlearnMS.

Application of explainable artificial intelligence in medical health: A systematic review of interpretability methods

This paper investigates the applications of explainable AI (XAI) in healthcare, which aims to provide transparency, fairness, accuracy, generality, and comprehensibility to the results obtained from AI and ML algorithms in decision-making systems. The black box nature of AI and ML systems has remained a challenge in healthcare, and interpretable AI and ML techniques can potentially address this issue. Here we critically review previous studies related to the interpretability of ML and AI methods in medical systems. Descriptions of various types of XAI methods such as layer-wise relevance propagation (LRP), Uniform Manifold Approximation and Projection (UMAP), Local Interpretable Model-agnostic Explanations (LIME), SHapley Additive exPlanations (SHAP), ANCHOR, contextual importance and utility (CIU), Training calibration-based explainers (TraCE), Gradient-weighted Class Activation Mapping (Grad-CAM), t-distributed Stochastic Neighbor Embedding (t-SNE), NeuroXAI, Explainable Cumulative Fuzzy Class Membership Criterion (X-CFCMC) along with the diseases which can be explained through these methods are provided throughout the paper. The paper also discusses how AI and ML technologies can transform healthcare services. The usability and reliability of the presented methods are summarized, including studies on the usability and reliability of XGBoost for mediastinal cysts and tumors, a 3D brain tumor segmentation network, and the TraCE method for medical image analysis. Overall, this paper aims to contribute to the growing field of XAI in healthcare and provide insights for researchers, practitioners, and decision-makers in the healthcare industry. Finally, we discuss the performance of XAI methods applied in medical health care systems. It is also needed to mention that a brief implemented method is provided in the methodology section.

Towards greater neuroimaging classification transparency via the integration of explainability methods and confidence estimation approaches

The field of neuroimaging has increasingly sought to develop artificial intelligence-based models for neurological and neuropsychiatric disorder automated diagnosis and clinical decision support. However, if these models are to be implemented in a clinical setting, transparency will be vital. Two aspects of transparency are (1) confidence estimation and (2) explainability. Confidence estimation approaches indicate confidence in individual predictions. Explainability methods give insight into the importance of features to model predictions. In this study, we integrate confidence estimation and explainability approaches for the first time. We demonstrate their viability for schizophrenia diagnosis using resting state functional magnetic resonance imaging (rs-fMRI) dynamic functional network connectivity (dFNC) data. We compare two confidence estimation approaches: Monte Carlo dropout (MCD) and MC batch normalization (MCBN). We combine them with two gradient-based explainability approaches, saliency and layer-wise relevance propagation (LRP), and examine their effects upon explanations. We find that MCD often adversely affects model gradients, making it ill-suited for integration with gradient-based explainability methods. In contrast, MCBN does not affect model gradients. Additionally, we find many participant-level differences between regular explanations and the distributions of explanations for combined explainability and confidence estimation approaches. This suggests that a similar confidence estimation approach used in a clinical context with explanations only output for the regular model would likely not yield adequate explanations. We hope that our findings will provide a starting point for the integration of the two fields, provide useful guidance for future studies, and accelerate the development of transparent neuroimaging clinical decision support systems.

Deep Learning–Based Parameter Transfer in Meteorological Data

Abstract Numerical simulations in Earth-system sciences consider a multitude of physical parameters in space and time, leading to severe input/output (I/O) bandwidth requirements and challenges in subsequent data analysis tasks. Deep learning–based identification of redundant parameters and prediction of those from other parameters, that is, variable-to-variable (V2V) transfer, has been proposed as an approach to lessening the bandwidth requirements and streamlining subsequent data analysis. In this paper, we examine the applicability of V2V to meteorological reanalysis data. We find that redundancies within pairs of parameter fields are limited, which hinders application of the original V2V algorithm. Therefore, we assess the predictive strength of reanalysis parameters by analyzing the learning behavior of V2V reconstruction networks in an ablation study. We demonstrate that efficient V2V transfer becomes possible when considering groups of parameter fields for transfer and propose an algorithm to implement this. We investigate further whether the neural networks trained in the V2V process can yield insightful representations of recurring patterns in the data. The interpretability of these representations is assessed via layerwise relevance propagation that highlights field areas and parameters of high importance for the reconstruction model. Applied to reanalysis data, this allows for uncovering mutual relationships between landscape orography and different regional weather situations. We see our approach as an effective means to reduce bandwidth requirements in numerical weather simulations, which can be used on top of conventional data compression schemes. The proposed identification of multiparameter features can spawn further research on the importance of regional weather situations for parameter prediction and also in other kinds of simulation data.

Deciphering multiple sclerosis disability with deep learning attention maps on clinical MRI

The application of convolutional neural networks (CNNs) to MRI data has emerged as a promising approach to achieving unprecedented levels of accuracy when predicting the course of neurological conditions, including multiple sclerosis, by means of extracting image features not detectable through conventional methods. Additionally, the study of CNN-derived attention maps, which indicate the most relevant anatomical features for CNN-based decisions, has the potential to uncover key disease mechanisms leading to disability accumulation.From a cohort of patients prospectively followed up after a first demyelinating attack, we selected those with T1-weighted and T2-FLAIR brain MRI sequences available for image analysis and a clinical assessment performed within the following six months (N = 319). Patients were divided into two groups according to expanded disability status scale (EDSS) score: ≥3.0 and < 3.0. A 3D-CNN model predicted the class using whole-brain MRI scans as input. A comparison with a logistic regression (LR) model using volumetric measurements as explanatory variables and a validation of the CNN model on an independent dataset with similar characteristics (N = 440) were also performed. The layer-wise relevance propagation method was used to obtain individual attention maps.The CNN model achieved a mean accuracy of 79% and proved to be superior to the equivalent LR-model (77%). Additionally, the model was successfully validated in the independent external cohort without any re-training (accuracy = 71%). Attention-map analyses revealed the predominant role of frontotemporal cortex and cerebellum for CNN decisions, suggesting that the mechanisms leading to disability accrual exceed the mere presence of brain lesions or atrophy and probably involve how damage is distributed in the central nervous system.

ULAN: A Universal Local Adversarial Network for SAR Target Recognition Based on Layer-Wise Relevance Propagation

Recent studies have proven that synthetic aperture radar (SAR) automatic target recognition (ATR) models based on deep neural networks (DNN) are vulnerable to adversarial examples. However, existing attacks easily fail in the case where adversarial perturbations cannot be fully fed to victim models. We call this situation perturbation offset. Moreover, since background clutter takes up most of the area in SAR images and has low relevance to recognition results, fooling models with global perturbations is quite inefficient. This paper proposes a semi-white-box attack network called Universal Local Adversarial Network (ULAN) to generate universal adversarial perturbations (UAP) for the target regions of SAR images. In the proposed method, we calculate the model’s attention heatmaps through layer-wise relevance propagation (LRP), which is used to locate the target regions of SAR images that have high relevance to recognition results. In particular, we utilize a generator based on U-Net to learn the mapping from noise to UAPs and craft adversarial examples by adding the generated local perturbations to target regions. Experiments indicate that the proposed method effectively prevents perturbation offset and achieves comparable attack performance to conventional global UAPs by perturbing only a quarter or less of SAR image areas.

Root cause diagnosis for process faults based on multisensor time-series causality discovery

Root cause diagnosis (RCD) focuses on locating the critical root causes and identifying the propagation paths of the industrial process faults, which is superior to conventional fault diagnosis methods and has attracted extensive attention. However, RCD is challenging in terms of dense sensor layout and diversified interconnected subsystems in complex industrial processes, since a fault originating in one unit can propagate throughout the entire plant along with the flow of information and material, obscuring its root causes. Moreover, the previous RCD methods ignore the more crucial temporal information between multisensor time-series, suffering from the disadvantages of requiring expert knowledge and responding untimely. Therefore, a novel data-driven RCD methodology named multisensor time-series causality discovery (MTCD) is proposed for industrial processes fault diagnosis. Firstly, a temporal registration network (TRN) based on the dilated convolutional neural network (DCNN) is proposed to extract the temporal correlation between time-series and implement the time-series prediction. Ulteriorly, a permutation importance causality validation (PICV) is designed to verify the causality between time-series based on the TRN prediction results without prior knowledge or expert experience. Besides, a causal time delay discovery approach based on layerwise relevance propagation (LRP) is presented by interpreting the weights of the trained TRN. Finally, we demonstrate the effectiveness of the proposed method with a stochastic synthesis process and the Tennessee Eastman process. The experimental results show that the proposed method has better performance in both root cause localization and propagation path identification.

Covid-19 detection using chest X-rays: is lung segmentation important for generalization?

PurposeWe evaluated the generalization capability of deep neural networks (DNNs) in the task of classifying chest X-rays as Covid-19, normal or pneumonia, when trained in a relatively small and mixed datasets.MethodsWe proposed a DNN to perform lung segmentation and classification, stacking a segmentation module (U-Net), an original intermediate module and a classification module (DenseNet201). To evaluate generalization capability, we tested the network with an external dataset (from distinct localities) and used Bayesian inference to estimate the probability distributions of performance metrics. Furthermore, we introduce a novel evaluation technique, which uses layer-wise relevance propagation (LRP) and Brixia scores to compare the DNN grounds for decision with radiologists.ResultsThe proposed DNN achieved 0.917 AUC (area under the ROC curve) on the external test dataset, surpassing a DenseNet without segmentation, which showed 0.906 AUC. Bayesian inference indicated mean accuracy of 76.1% and [0.695, 0.826] 95% HDI (high-density interval, which concentrates 95% of the metric’s probability mass) with segmentation and, without segmentation, 71.7% and [0.646, 0.786].ConclusionEmploying an analysis based on LRP and Brixia scores, we discovered that areas where radiologists found strong Covid-19 symptoms are the most important for the stacked DNN classification. External validation showed smaller accuracies than internal, indicating difficulty in generalization, which is positively affected by lung segmentation. Finally, the performance on the external dataset and the analysis with LRP suggest that DNNs can successfully detect Covid-19 even when trained on small and mixed datasets.

Canonical correlation analysis as a feature extraction method to classify active sonar targets with shallow neural networks.

Sonar target recognition remains an active area of research due to the complex entanglement of features from various acoustic scatterers, background clutter, and distortion by waveguide propagation effects. An equally challenging issue is due to different acoustic echoes returned from the target (including different target elements) itself. This work investigates the sonar target classification problem from a statistical perspective and aims to extract salient target feature vectors. Specifically, a multivariate statistical method is employed, canonical correlation analysis (CCA), as a feature extraction technique prior to multi-class classification of active sonar field data. The intuition behind using CCA is that persistent features slowly morph over time due to the changing aspect angles and platform positions and can be represented by maximally correlated projections of consecutive pings. CCA is applied using a sliding window, and the projections are used as feature vectors to train a neural network classifier. The smallest increase in classification accuracy when comparing the projection feature vectors to unprocessed feature vectors was 10%. The largest increase was 34%. The results are further examined through the use of confusion matrices and layer-wise relevance propagation, which distributes the trained networks output score to the input layer.

Higher-Order Explanations of Graph Neural Networks via Relevant Walks.

Graph Neural Networks (GNNs) are a popular approach for predicting graph structured data. As GNNs tightly entangle the input graph into the neural network structure, common explainable AI approaches are not applicable. To a large extent, GNNs have remained black-boxes for the user so far. In this paper, we show that GNNs can in fact be naturally explained using higher-order expansions, i.e., by identifying groups of edges that jointly contribute to the prediction. Practically, we find that such explanations can be extracted using a nested attribution scheme, where existing techniques such as layer-wise relevance propagation (LRP) can be applied at each step. The output is a collection of walks into the input graph that are relevant for the prediction. Our novel explanation method, which we denote by GNN-LRP, is applicable to a broad range of graph neural networks and lets us extract practically relevant insights on sentiment analysis of text data, structure-property relationships in quantum chemistry, and image classification.

Explainable Artificial Intelligence for Bayesian Neural Networks: Toward Trustworthy Predictions of Ocean Dynamics

AbstractThe trustworthiness of neural networks is often challenged because they lack the ability to express uncertainty and explain their skill. This can be problematic given the increasing use of neural networks in high stakes decision‐making such as in climate change applications. We address both issues by successfully implementing a Bayesian Neural Network (BNN), where parameters are distributions rather than deterministic, and applying novel implementations of explainable AI (XAI) techniques. The uncertainty analysis from the BNN provides a comprehensive overview of the prediction more suited to practitioners' needs than predictions from a classical neural network. Using a BNN means we can calculate the entropy (i.e., uncertainty) of the predictions and determine if the probability of an outcome is statistically significant. To enhance trustworthiness, we also spatially apply the two XAI techniques of Layer‐wise Relevance Propagation (LRP) and SHapley Additive exPlanation (SHAP) values. These XAI methods reveal the extent to which the BNN is suitable and/or trustworthy. Using two techniques gives a more holistic view of BNN skill and its uncertainty, as LRP considers neural network parameters, whereas SHAP considers changes to outputs. We verify these techniques using comparison with intuition from physical theory. The differences in explanation identify potential areas where new physical theory guided studies are needed.

LRP‐based network pruning and policy distillation of robust and non‐robust DRL agents for embedded systems

SummaryReinforcement learning (RL) is an effective approach to developing control policies by maximizing the agent's reward. Deep reinforcement learning uses deep neural networks (DNNs) for function approximation in RL, and has achieved tremendous success in recent years. Large DNNs often incur significant memory size and computational overheads, which may impede their deployment into resource‐constrained embedded systems. For deployment of a trained RL agent on embedded systems, it is necessary to compress the policy network of the RL agent to improve its memory and computation efficiency. In this article, we perform model compression of the policy network of an RL agent by leveraging the relevance scores computed by layer‐wise relevance propagation (LRP), a technique for Explainable AI (XAI), to rank and prune the convolutional filters in the policy network, combined with fine‐tuning with policy distillation. Performance evaluation based on several Atari games indicates that our proposed approach is effective in reducing model size and inference time of RL agents. We also consider robust RL agents trained with RADIAL‐RL versus standard RL agents, and show that a robust RL agent can achieve better performance (higher average reward) after pruning than a standard RL agent for different attack strengths and pruning rates.

Explainable Machine-Learning-Based Characterization of Abnormal Cortical Activities for Working Memory of Restless Legs Syndrome Patients.

Restless legs syndrome (RLS) is a sensorimotor disorder accompanied by a strong urge to move the legs and an unpleasant sensation in the legs, and is known to accompany prefrontal dysfunction. Here, we aimed to clarify the neural mechanism of working memory deficits associated with RLS using machine-learning-based analysis of single-trial neural activities. A convolutional neural network classifier was developed to discriminate the cortical activities between RLS patients and normal controls. A layer-wise relevance propagation was applied to the trained classifier in order to determine the critical nodes in the input layer for the output decision, i.e., the time/location of cortical activities discriminating RLS patients and normal controls during working memory tasks. Our method provided high classification accuracy (~94%) from single-trial event-related potentials, which are known to suffer from high inter-trial/inter-subject variation and low signal-to-noise ratio, after strict separation of training/test/validation data according to leave-one-subject-out cross-validation. The determined critical areas overlapped with the cortical substrates of working memory, and the neural activities in these areas were correlated with some significant clinical scores of RLS.

Interpretable deep learning-based hippocampal sclerosis classification.

To evaluate the performance of a deep learning model for hippocampal sclerosis classification on the clinical dataset and suggest plausible visual interpretation for the model prediction. T2-weighted oblique coronal images of the brain MRI epilepsy protocol performed on patients were used. The training set included 320 participants with 160 no, 100 left and 60 right hippocampal sclerosis, and cross-validation was implemented. The test set consisted of 302 participants with 252 no, 25 left and 25 right hippocampal sclerosis. As the test set was imbalanced, we took an average of the accuracy achieved within each group to measure a balanced accuracy for multiclass and binary classifications. The dataset was composed to include not only healthy participants but also participants with abnormalities besides hippocampal sclerosis in the control group. We visualized the reasons for the model prediction using the layer-wise relevance propagation method. When evaluated on the validation of the training set, we achieved multiclass and binary classification accuracy of 87.5% and 88.8% from the voting ensemble of six models. Evaluated on the test sets, we achieved multiclass and binary classification accuracy of 91.5% and 89.76%. The distinctly sparse visual interpretations were provided for each individual participant and group to suggest the contribution of each input voxel to the prediction on the MRI. The current interpretable deep learning-based model is promising for adapting effectively to clinical settings by utilizing commonly used data, such as MRI, with realistic abnormalities faced by neurologists to support the diagnosis of hippocampal sclerosis with plausible visual interpretation.