Global Explanations Research Articles

XRan: Explainable deep learning-based ransomware detection using dynamic analysis

Recently, the frequency and complexity of ransomware attacks have been increasing steadily, posing significant threats to individuals and organizations alike. While traditional signature-based antiransomware systems are effective in the detection of known threats, they struggle to identify new ransomware samples. To address this limitation, many researchers have focused on analyzing the behavior and actions of executables. During this dynamic analysis process, various dynamic-based features emerge, offering different perspectives on the executable's behavior, including Application Program Interface (API) call sequences, dynamic link libraries (DLLs), and mutual exclusions. Existing methods mostly perform machine or deep learning models for feature engineering and detection. These methods usually perform learning according to a single perspective or by combining data from different perspectives into the frequency domain. In this case, they may ignore the information from the other aspects or the sequence relationship between the features. In addition, learning models used in these solutions are mostly incomprehensible to humans, which could be an obstacle in terms of having an insight through the model's mentality and also ransomware's way of work. In this study, we provide XRan (eXplainable deep learning-based RANsomware detection using dynamic analysis), an Explainable Artificial Intelligence (XAI) supported ransomware detection system that combines different dynamic analysis-based sequences, each representing a different view of the executable, in order to enrich the feature space. XRan employs a Convolutional Neural Network (CNN) architecture to detect ransomware and two XAI models as Interpretable Model-Agnostic Explanations (LIME), and SHapley Additive exPlanations (SHAP) to provide local and global explanations for detection. Experimental results demonstrate that XRan provides up to 99.4% True Positive Rate (TPR), and outperforms the state-of-the-art methods.

Prediction and Interpretation of Total N and Its Key Drivers in Cultivated Tropical Peat using Machine Learning and Game Theory

Currently, there is a growing interest among research communities in the development of statistical learning-based pedotransfer functions/PtFs to predict mineral soil nutrients; however, similar studies in peatlands are relatively rare. Moreover, extracting meaningful information from these ‘black-box’ models is crucial, particularly concerning their algorithmic complexity and the non-linear nature of the soil covariate interrelationships. This study employed the Pulunggono (2022a) dataset and the bootstrapping method, to (1) develop and evaluate seven PtF models, including both general linear models (GLM) and machine learning (ML) regressors for estimating total nitrogen (N) in tropical peat that has been drained and cultivated for oil palm (OP) in Riau, Indonesia and (2) explaining model functioning by incorporating Shapley Additive Explanation (SHAP), a tool derived from coalitional game theory. This study demonstrated the superior predictive performance of ML-based PtFs in estimating total N compared to GLM algorithms. The top-performing algorithms for PtF models were identified as GBM, XGB, and Cubist. The SHAP method revealed that sampling depth and organic C were consistently identified as the most important covariates across all models, irrespective of their algorithmic capabilities. Additionally, ML algorithms identified the total Fe, pH, and bulk density (BD) as significant covariates. Local explanations based on Shapley values indicated that the behavior of PtF-based algorithms diverged from their global explanations. This study emphasized the critical role of ML algorithms and game theory in accurately predicting total N in peatlands subjected to drainage and cultivation for OP and explaining their model behavior in relation to soil biogeochemical processes.

DeepSecDrive: An explainable deep learning framework for real-time detection of cyberattack in in-vehicle networks

Autonomous driving (AD) technologies are becoming increasingly popular, promising significant enhancements in road safety and efficiency. However, the susceptibility of autonomous vehicles to cyberattacks highlights the critical need for protecting the security and dependability of in-vehicular networks (IVNs). This study proposes a lightweight, efficient, and explainable deep learning framework, DeepSecDrive, for early detection of security threats against IVN. DeepSecDrive presents robust feature extractor units designed with deformable convolutions and a lightweight non-local network (LNLN). The former promotes dynamic change of receptive field to enhance extraction of complex features from the traffics of in-vehicle networks, while the latter enables learning contextual representations and non-local relationships in vehicular data. DeepSecDrive integrates Shapley Additive exPlanations to construct multi-level interpretability units, providing local and global explanations for model decisions in terms of feature contributions to the overall performance. Proof-of-concept experiments are conducted to evaluate the efficacy and durability of DeepSecDrive on real-world IVN attacks from Car-Hacking dataset, and the experimental results demonstrate the ability of our model to outperform state-of-the-art detection methods.

Explanations based on Item Response Theory (eXirt): A model-specific method to explain tree-ensemble model in trust perspective

Solutions based on tree-ensemble models represent a considerable alternative to real-world prediction problems, but these models are considered black box, thus hindering their applicability in problems of sensitive contexts (such as: health and safety). Explainable Artificial Intelligence (XAI) aims to develop techniques that generate explanations of black box models, since these models are normally not self-explanatory. Methods such as Ciu, Dalex, Eli5, Lofo, Shap and Skater emerged with the proposal to explain black box models through global rankings of feature relevance, which based on different methodologies, generate global explanations that indicate how the model’s inputs explain its predictions. This research aims to present an innovative XAI method, called eXirt, capable of carrying out the process of explaining tree-ensemble models, based on Item Response Theory (IRT). In this context, 41 datasets, 4 tree-ensemble algorithms (Light Gradient Boosting, CatBoost, Random Forest, and Gradient Boosting), and 7 XAI methods (including eXirt) were used to generate explanations. In the first set of analyses, the 164 ranks of global feature relevance generated by eXirt were compared with 984 ranks of the other XAI methods present in the literature, being verified that the new method generated different explanations from other existing methods. In a second analysis, exclusive local and global explanations generated by eXirt were presented that help in understanding the model trust, since in this explanation it is possible to observe particularities of the model regarding difficulty (if the model had difficulty predicting the test dataset), discrimination (if the model understands the test dataset as discriminative) and guesswork (if the model got the test dataset right by chance). Thus, it was verified that eXirt is able to generate global explanations of tree-ensemble models and also local and global explanations of models through IRT, showing how this consolidated theory can be used in machine learning in order to obtain explainable and reliable models.

Explainable AI Evaluation: A Top-Down Approach for Selecting Optimal Explanations for Black Box Models

Explainable Artificial Intelligence (XAI) evaluation has grown significantly due to its extensive adoption, and the catastrophic consequence of misinterpreting sensitive data, especially in the medical field. However, the multidisciplinary nature of XAI research resulted in diverse scholars possessing significant challenges in designing proper evaluation methods. This paper proposes a novel framework of a three-layered top-down approach on how to arrive at an optimal explainer, accenting the persistent need for consensus in XAI evaluation. This paper also investigates a critical comparative evaluation of explanations in both model agnostic and specific explainers including LIME, SHAP, Anchors, and TabNet, aiming to enhance the adaptability of XAI in a tabular domain. The results demonstrate that TabNet achieved the highest classification recall followed by TabPFN, and XGBoost. Additionally, this paper develops an optimal approach by introducing a novel measure of relative performance loss with emphasis on faithfulness and fidelity of global explanations by quantifying the extent to which a model’s capabilities diminish when eliminating topmost features. This addresses a conspicuous gap in the lack of consensus among researchers regarding how global feature importance impacts classification loss, thereby undermining the trust and correctness of such applications. Finally, a practical use case on medical tabular data is provided to concretely illustrate the findings.

Explainability of random survival forests in predicting conversion risk from mild cognitive impairment to Alzheimer’s disease

Random Survival Forests (RSF) has recently showed better performance than statistical survival methods as Cox proportional hazard (CPH) in predicting conversion risk from mild cognitive impairment (MCI) to Alzheimer’s disease (AD). However, RSF application in real-world clinical setting is still limited due to its black-box nature.For this reason, we aimed at providing a comprehensive study of RSF explainability with SHapley Additive exPlanations (SHAP) on biomarkers of stable and progressive patients (sMCI and pMCI) from Alzheimer’s Disease Neuroimaging Initiative. We evaluated three global explanations—RSF feature importance, permutation importance and SHAP importance—and we quantitatively compared them with Rank-Biased Overlap (RBO). Moreover, we assessed whether multicollinearity among variables may perturb SHAP outcome. Lastly, we stratified pMCI test patients in high, medium and low risk grade, to investigate individual SHAP explanation of one pMCI patient per risk group.We confirmed that RSF had higher accuracy (0.890) than CPH (0.819), and its stability and robustness was demonstrated by high overlap (RBO > 90%) between feature rankings within first eight features. SHAP local explanations with and without correlated variables had no substantial difference, showing that multicollinearity did not alter the model. FDG, ABETA42 and HCI were the first important features in global explanations, with the highest contribution also in local explanation. FAQ, mPACCdigit, mPACCtrailsB and RAVLT immediate had the highest influence among all clinical and neuropsychological assessments in increasing progression risk, as particularly evident in pMCI patients’ individual explanation. In conclusion, our findings suggest that RSF represents a useful tool to support clinicians in estimating conversion-to-AD risk and that SHAP explainer boosts its clinical utility with intelligible and interpretable individual outcomes that highlights key features associated with AD prognosis.Graphical

An efficient classification framework for Type 2 Diabetes incorporating feature interactions

Accurate and timely diagnosis of Type 2 Diabetes is a highly challenging task due to its initial asymptomatic nature and complex risk factor composition. Recently, Machine Learning (ML) has been actively used to build improved Type 2 Diabetes classification systems. One important aspect of these systems has been feature selection. Filter feature selection techniques especially based on mutual information have been popularly employed in recent works. However, most of them have focused on selecting relevant features and eliminating redundant ones. A third relationship called feature interaction may exist if input features are highly correlated, as in the case of Type 2 Diabetes. Feature interaction signifies the additional information about the target provided by the interaction between a subset of input features, that may not be relevant to the target individually. Second, many of the existing ML models are black-box, making the model interpretability very difficult. This paper proposes an efficient ML framework for the classification of Prediabetes and Type 2 Diabetes by incorporating feature interactions. It presents a hybrid filter-wrapper technique called Feature Interaction-based Greedy Sequential Feature selection. Agglomerative Feature Clustering and Dendrogram visualization for the analysis of interactive features is performed. A model-agnostic explainability technique of SHapley Additive explanations (SHAP) is augmented to provide local and global explanations of model predictions. The performance of the proposed classification framework was found to be interpretable as well as efficient with an accuracy of 98.8669%, precision of 98.8660%, recall of 98.8665%, and F-score of 0.9886 for Diabetes. For Prediabetes, an accuracy of 90.1187%, precision of 94.6958%, recall of 90.1187%, and F-score of 0.9403 was obtained.

An explainable hybrid deep learning architecture for WiFi-based indoor localization in Internet of Things environment

The indoor positioning service is one of the essential services needed in the Internet of Things ecosystem. Recently, many researchers have focused on the fingerprinting method, which is a method based on signal mapping with the Received Signal Strength Indicator (RSSI) values obtained from the WiFi access points. However, the fingerprinting method is particularly challenging due to some difficulties, such as RSSI variance over time, device diversity, and similarities of fingerprints in indoor networks. For this reason, machine learning and deep learning methods are used for many purposes, such as estimating the location of the building, floor, or the rooms. Detecting the location of a room or more than one reference point in a room becomes a more difficult problem because neighboring reference points’ fingerprints are very similar to each other. This study proposes a WiFi-based XAI-empowered deep learning architecture to predict the reference points in a room or corridor. We present a hybrid deep learning-based method that uses Long-Short-Term Memory to capture long-term dependencies between the signal features, and Convolutional Neural Network to extract local spatial signal patterns. Our deep learning aims to enrich fingerprinting data of each sample to capture more meaningful feature maps coming from different angles. Moreover, the method applies effective filtering and dimension scaling on the data to regulate the RSS values and capture more discriminative patterns using particle filter and sparse autoencoder. To provide local and global explanations for indoor localization estimations, the proposed architecture comprises two Explainable Artificial Intelligence techniques as Interpretable Model-Agnostic Explanations, and SHapley Additive exPlanations. The experimental results demonstrate that the proposed architecture achieves higher accuracy values for all datasets than the baseline deep learning methods.

Cervical Cancer Diagnosis Using Stacked Ensemble Model and Optimized Feature Selection: An Explainable Artificial Intelligence Approach

Cervical cancer affects more than half a million women worldwide each year and causes over 300,000 deaths. The main goals of this paper are to study the effect of applying feature selection methods with stacking models for the prediction of cervical cancer, propose stacking ensemble learning that combines different models with meta-learners to predict cervical cancer, and explore the black-box of the stacking model with the best-optimized features using explainable artificial intelligence (XAI). A cervical cancer dataset from the machine learning repository (UCI) that is highly imbalanced and contains missing values is used. Therefore, SMOTE-Tomek was used to combine under-sampling and over-sampling to handle imbalanced data, and pre-processing steps are implemented to hold missing values. Bayesian optimization optimizes models and selects the best model architecture. Chi-square scores, recursive feature removal, and tree-based feature selection are three feature selection techniques that are applied to the dataset For determining the factors that are most crucial for predicting cervical cancer, the stacking model is extended to multiple levels: Level 1 (multiple base learners) and Level 2 (meta-learner). At Level 1, stacking (training and testing stacking) is employed for combining the output of multi-base models, while training stacking is used to train meta-learner models at level 2. Testing stacking is used to evaluate meta-learner models. The results showed that based on the selected features from recursive feature elimination (RFE), the stacking model has higher accuracy, precision, recall, f1-score, and AUC. Furthermore, To assure the efficiency, efficacy, and reliability of the produced model, local and global explanations are provided.

Interpretable Machine Learning for Predicting Symptomatic Pneumonitis in Locally Advanced Non-Small Cell Lung Cancer Patients Treated with Concurrent Chemoradiotherapy and Immune Checkpoint Inhibitor Consolidation.

The rate of grade 2 and higher pneumonitis has increased with the use of immune checkpoint inhibitors (ICI) following chemoradiotherapy (CRT) for lung cancer, which may alter previously established dose-volume constraints (DVC). In this study, we used an interpretable machine learning model with clinical and dosimetric features to predict grade 2+ pneumonitis and determine DVC associated with pneumonitis for locally advanced non-small cell lung cancer (LA-NSCLC) radiotherapy (RT). Between October 2017 and December 2021, 223 consecutively treated patients with LANSCLC treated with CRT and ICI were retrospectively reviewed. The dataset was split into training and test sets (n = 144/79). Clinical features included age, sex, smoking status, pack-years, BMI, ECOG PS, COPD, tumor location, delivered dose, RT technique, chemotherapy agent and volume of GTVp/GTVn. A total of 228 dosimetric features from the heart, contralateral/ipsilateral lung and lungs-IGTV were extracted, including the minimum/mean dose to the hottest x% volume (Dx%[Gy]/MOHx%[Gy]; x was 5-95 in 5% increments) and minimum/mean/maximum dose and percent volume receiving at least xGy (VxGy [%]; x was 5-60 in 5Gy increments), as well as the overlapping volume of each structure with PTV and the distance from each structure to GTVp/GTVn. Feature selection was performed using Boruta, followed by collinearity removal based on the variance inflation factor. The explainable boosting machine (EBM) was trained on the selected features. The performance of EBM on the test set was evaluated using the area under the receiver operating characteristic curve (AUC) and compared with that of blackbox (BB) models, including extreme gradient boosting (XGB), random forest (RF), and supporting vector machine (SVM). The global explanation of each feature's contribution to the predictions provided by the EBM was used to determine DVC. Shapley additive explanations (SHAP) were used to explain BB predictions. Selected features, ranked in order of EBM's overall feature importance, were V25Gy [%] and MOH65%[Gy] in the ipsilateral lung, the maximum dose in the heart, MOH30%[Gy] in the contralateral lung, and BMI. No dosimetric features in the lungs-IGTV were selected. The SHAP values of three BB models showed similar trends to the feature importance of the EBM. The global explanations of the EBM suggested that to mitigate the risk of pneumonitis, the ipsilateral lung should have V25Gy [%] < 36.8% and MOH65%[Gy] < 39.5Gy, and the heart should have D0.03cc [Gy] < 66.0Gy. Furthermore, an increased risk of pneumonitis was indicated with an increase in BMI, and, surprisingly, a decrease in MOH30%[Gy] in the contralateral lung. The EBM showed the best performance for predicting grade 2+ pneumonitis (AUC = 0.739), followed by RF, SVM, and XGB (AUC = 0.735, 0.733, and 0.717). EBM has the potential to predict grade 2+ pneumonitis in LA-NSCLC patients treated with CRT and ICI, while providing guidance on DVC.

IHCP: interpretable hepatitis C prediction system based on black-box machine learning models

BackgroundHepatitis C is a prevalent disease that poses a high risk to the human liver. Early diagnosis of hepatitis C is crucial for treatment and prognosis. Therefore, developing an effective medical decision system is essential. In recent years, many computational methods have been proposed to identify hepatitis C patients. Although existing hepatitis prediction models have achieved good results in terms of accuracy, most of them are black-box models and cannot gain the trust of doctors and patients in clinical practice. As a result, this study aims to use various Machine Learning (ML) models to predict whether a patient has hepatitis C, while also using explainable models to elucidate the prediction process of the ML models, thus making the prediction process more transparent.ResultWe conducted a study on the prediction of hepatitis C based on serological testing and provided comprehensive explanations for the prediction process. Throughout the experiment, we modeled the benchmark dataset, and evaluated model performance using fivefold cross-validation and independent testing experiments. After evaluating three types of black-box machine learning models, Random Forest (RF), Support Vector Machine (SVM), and AdaBoost, we adopted Bayesian-optimized RF as the classification algorithm. In terms of model interpretation, in addition to using common SHapley Additive exPlanations (SHAP) to provide global explanations for the model, we also utilized the Local Interpretable Model-Agnostic Explanations with stability (LIME_stabilitly) to provide local explanations for the model.ConclusionBoth the fivefold cross-validation and independent testing show that our proposed method significantly outperforms the state-of-the-art method. IHCP maintains excellent model interpretability while obtaining excellent predictive performance. This helps uncover potential predictive patterns of the model and enables clinicians to better understand the model's decision-making process.

Scale-preserving automatic concept extraction (SPACE)

Convolutional Neural Networks (CNN) have become a common choice for industrial quality control, as well as other critical applications in the Industry 4.0. When these CNNs behave in ways unexpected to human users or developers, severe consequences can arise, such as economic losses or an increased risk to human life. Concept extraction techniques can be applied to increase the reliability and transparency of CNNs through generating global explanations for trained neural network models. The decisive features of image datasets in quality control often depend on the feature’s scale; for example, the size of a hole or an edge. However, existing concept extraction methods do not correctly represent scale, which leads to problems interpreting these models as we show herein. To address this issue, we introduce the Scale-Preserving Automatic Concept Extraction (SPACE) algorithm, as a state-of-the-art alternative concept extraction technique for CNNs, focused on industrial applications. SPACE is specifically designed to overcome the aforementioned problems by avoiding scale changes throughout the concept extraction process. SPACE proposes an approach based on square slices of input images, which are selected and then tiled before being clustered into concepts. Our method provides explanations of the models’ decision-making process in the form of human-understandable concepts. We evaluate SPACE on three image classification datasets in the context of industrial quality control. Through experimental results, we illustrate how SPACE outperforms other methods and provides actionable insights on the decision mechanisms of CNNs. Finally, code for the implementation of SPACE is provided.

Establishment and interpretation of the gamma pass rate prediction model based on radiomics for different intensity-modulated radiotherapy techniques in the pelvis

Backgroundand objectives: Implementation of patient-specific quality assurance (PSQA) is a crucial aspect of precise radiotherapy. Various machine learning-based models have showed potential as virtual quality assurance tools, being capable of accurately predicting the dose verification results of fixed-beam intensity-modulated radiation therapy (IMRT) or volumetric modulated arc therapy (VMAT) plans, thereby ensuring safe and efficient treatment for patients. However, there has been no research yet that simultaneously integrates different IMRT techniques to predict the gamma pass rate (GPR) and explain the model.Methods: Retrospective analysis of the 3D dosimetric verification results based on measurements with gamma pass rate criteria of 3%/2 mm and 10% dose threshold of 409 pelvic IMRT and VMAT plans was carried out. Radiomics features were extracted from the dose files, from which the XGBoost algorithm based on SHapley Additive exPlanations (SHAP) values was used to select the optimal feature subset as the input for the prediction model. The study employed four different machine learning algorithms, namely, random forest (RF), adaptive boosting (AdaBoost), extreme gradient boosting (XGBoost), and light gradient boosting machine (LightGBM), to construct predictive models. Sensitivity, specificity, F1 score, and AUC value were calculated to evaluate the classification performance of these models. The SHAP values were utilized to perform a related interpretive analysis on the best performing model.Results: The sensitivities and specificities of the RF, AdaBoost, XGBoost, and LightGBM models were 0.96, 0.82, 0.93, and 0.89, and 0.38, 0.54, 0.62, and 0.62, respectively. The F1 scores and area under the curve (AUC) values were 0.86, 0.81, 0.88, and 0.86, and 0.81, 0.77, 0.85, and 0.83, respectively. The explanation of the model output based on SHAP values can provide a reference basis for medical physicists when adjusting the plan, thereby improving the efficiency and quality of treatment plans.Conclusion: It is feasible to use a machine learning method based on radiomics to establish a gamma pass rate classification prediction model for IMRT and VMAT plans in the pelvis. The XGBoost model performs better in classification than the other three tree-based ensemble models, and global explanations and single-sample explanations of the model output through SHAP values may offer reference for medical physicists to provide high-quality plans, promoting the clinical application and implementation of GPR prediction models, and providing safe and efficient personalized QA management for patients.

A user-centered explainable artificial intelligence approach for financial fraud detection

This paper aims to produce user-centered explanations for financial fraud detection models based on Explainable artificial intelligence (XAI) methods. By combining an ensemble predictive model with an explainable framework based on Shapley values, we develop a financial fraud detection approach that is accurate and explainable at the same time. Our results show that the explainable framework can meet the requirements of different external stakeholders by producing local and global explanations. Local explanations can help understand why a specific prediction is identified as fraud, and global explanations reveal the overall logic of the whole ensemble model.

Predicting and explaining karst spring dissolved oxygen using interpretable deep learning approach

AbstractDissolved oxygen (DO) is one of the most important indicators of water quality and an essential measure for the aquatic organisms and the local ecosystem. DO concentrations in karst spring flow involves complex hydrological processes because of the heterogeneous nature of the karst system. In this study, an interpretable and explainable model that integrates the shapley additive explanations (SHAP) algorithm with the long short‐term memory network model (LSTM) is proposed to evaluate the contributions of karst spring discharge, precipitation, water temperature, and specific conductance to DO concentrations in karst spring flow. The hybrid model can predict the temporal fluctuations of DO levels and provide a robust characterization of DO behaviours. To demonstrate the applicability of the proposed model, we adopted the hydrological and meteorological data from Barton Springs, situated within a highly karstified segment of the Edwards aquifer characterized by extensive fractures and openings. The optimal prediction performance is achieved with a 14‐day time step, which is considered as the effective response time between various hydrological processes and DO concentrations at Barton Springs. It reveals that the influence of karst spring discharge, precipitation, water temperature, and specific conductance in previous 14 days collectively contribute to the current DO concentration in karst spring flow. The SHAP values of input features provide both local and global explanations, demonstrating the magnitude and the direction of each feature's impact on DO levels in karst spring flow. In descending order, the contributions of various hydrological processes to DO are ranked as follows: precipitation, karst spring discharge, temperature, and specific conductance. Precipitation and discharge exhibit positive SHAP values, indicating that increases in these hydrological processes contribute to higher DO levels in the karst flow. Water temperature and specific conductance have negative SHAP values, suggesting that higher water temperature and specific conductance will lead to decreased DO levels in the karst flow.

Generating multi-level explanations for process outcome predictions

Process mining focuses on the analysis of event log data to build various process analytical capabilities. Predictive process analytics has emerged as one of such key capabilities and it uses machine learning techniques to construct process prediction models. In recent years, deep neural networks have gained increasing interest in process prediction since they can handle multi-dimensional sequential inputs with minimal information loss. However, they are considered black-box models and existing studies in explaining deep neural network-based process predictions rely on only event-level features for explanation. In this paper, we propose a new approach for generating explanations for process outcome predictions at multiple levels. The approach is underpinned by three different prediction models: a transparent model for generating global explanations based on case-level features, an attention-based deep neural network for generating local explanations based on event-level features, and a novel eXplainable Dual-learning Deep network (XD2-net) for generating local explanations based on case-level features. Using three publicly available datasets, we have tested the applicability of the approach and further examined the multi-level explanations generated by the approach through an elaborate case study. Unlike others, the design of our approach promotes the idea of leveraging the complementary capabilities of different models and utilizing their strengths, rather than focusing on model performance competition. This will contribute towards generating more comprehensive explanations that meet the needs of different end users and purposes in the future.

On Data-Aware Global Explainability of Graph Neural Networks

Graph Neural Networks (GNNs) have significantly boosted the performance of many graph-based applications, yet they serve as black-box models. To understand how GNNs make decisions, explainability techniques have been extensively studied. While the majority of existing methods focus on local explainability, we propose DAG-Explainer in this work aiming for global explainability. Specifically, we observe three properties of superior explanations for a pretrained GNN: they should be highly recognized by the model, compliant with the data distribution and discriminative among all the classes. The first property entails an explanation to be faithful to the model, as the other two require the explanation to be convincing regarding the data distribution. Guided by these properties, we design metrics to quantify the quality of each single explanation and formulate the problem of finding data-aware global explanations for a pretrained GNN as an optimizing problem. We prove that the problem is NP-hard and adopt a randomized greedy algorithm to find a near optimal solution. Furthermore, we derive an improved bound of the approximation algorithm in our problem over the state-of-the-art (SOTA) best. Experimental results show that DAG-Explainer can efficiently produce meaningful and trustworthy explanations while preserving comparable quantitative evaluation results to the SOTA methods.

Global Concept-Based Interpretability for Graph Neural Networks via Neuron Analysis

Graph neural networks (GNNs) are highly effective on a variety of graph-related tasks; however, they lack interpretability and transparency. Current explainability approaches are typically local and treat GNNs as black-boxes. They do not look inside the model, inhibiting human trust in the model and explanations. Motivated by the ability of neurons to detect high-level semantic concepts in vision models, we perform a novel analysis on the behaviour of individual GNN neurons to answer questions about GNN interpretability. We propose a novel approach for producing global explanations for GNNs using neuron-level concepts to enable practitioners to have a high-level view of the model. Specifically, (i) to the best of our knowledge, this is the first work which shows that GNN neurons act as concept detectors and have strong alignment with concepts formulated as logical compositions of node degree and neighbourhood properties; (ii) we quantitatively assess the importance of detected concepts, and identify a trade-off between training duration and neuron-level interpretability; (iii) we demonstrate that our global explainability approach has advantages over the current state-of-the-art -- we can disentangle the explanation into individual interpretable concepts backed by logical descriptions, which reduces potential for bias and improves user-friendliness.

Interpretable Machine Learning for Choosing Radiation Dose-volume Constraints on Cardio-pulmonary Substructures Associated with Overall Survival in NRG Oncology RTOG 0617

Our objective was to use interpretable machine learning for choosing dose-volume constraints on cardiopulmonary substructures (CPSs) associated with overall survival (OS) in radiation therapy for locally advanced non-small cell lung cancer. A total of 428 patients with non-small cell lung cancer were randomly divided into training/validation/test subsets (n=230/149/49) in Radiation Therapy Oncology Group 0617. Manual or automated contouring was performed to segment CPSs, including heart, atria, ventricles, aorta, left/right ventricle/atrium (LV+RV+LA+RA), inferior/superior vena cava, pulmonary artery, and pericardium. Peri (pericardium-heart), rest (heart-[LV+RV+LA+RA]), clinical target volume (CTV), and lungs-CTV contours were also obtained. Dose-volume histogram features were extracted, including minimum/mean dose to the hottest x% volume (Dx%[Gy]/MOHx%[Gy]), minimum/mean/maximum dose, percent volume receiving at least xGy (VxGy[%]), and overlapping volume of each CPS with planning target volume (PTV_Voverlap[%]). Clinical parameters were collected from the National Clinical Trials Network/Community oncology research program data archive. Feature selection was performed using a series of multiblock sparse partial least squares regression, stability selection supervised principal component analysis, and Boruta. Explainable boosting machine (EBM) was trained using a conditional survival distribution-based approach for imputing censored data, treating survival analysis as a regression problem. Harrell's C-index was used to evaluate OS discrimination performance of EBM, Cox proportional hazards (CPH), random survival forest, extreme gradient boosting survival embeddings, and CPH deep neural network (DeepSurv) models in the test set. Dose-volume constraints were selected using the binary change point detection algorithm in Shapley additive explanations-based partial dependence functions. Selected features included LA_V60Gy(%), pericardium_D30%(Gy), lungs-CTV_PTV_Voverlap(%), RA_V55Gy(%), and received_cons_chemo. All models ranked LA_V60Gy(%) as the most important feature. EBM achieved the best performance for predicting OS, followed by extreme gradient boosting survival embeddings, random survival forest, DeepSurv, and CPH (C-index=0.653, 0.646, 0.642, 0.638, and 0.632). EBM global explanations suggested that LA_V60Gy(%) < 25.6, lungs-CTV_PTV_Voverlap(%) < 1.1, pericardium_D30%(Gy) < 18.9, RA_V55Gy(%) < 19.5, and received_cons_chemo='Yes' for improved OS. EBM can be used to discriminate OS while also guiding dose-volume constraint selection for optimal management of cardiac toxicity in lung cancer radiation therapy.

Explainable AI for Cheating Detection and Churn Prediction in Online Games

Online gaming is a multi-billion dollar industry that entertains a large, global population. Empowering online games with AI has made a great success, however, ignores the explainability of black-box model makes AI less responsible and hinders its further development. In this paper, we introduce and discuss the audience and the concept of XAI (eXplainable AI) in online games. We propose a GXAI workflow which combines the strong expressiveness of multi-view data sources and the clear transparency of multi-view black-box models. We present four specific classifiers and explainers in the character portrait view, the behavior sequence view, the client image view and the social graph view. Experiments conducted on real-world datasets for game cheating detection and player churn prediction show the accuracy of classification and the rationality of explanation. We also discover and present numerous interesting and valuable findings from the individual, local and global explanations. We implement and deploy three practical applications including evidence and reason generation, model debugging and testing, and model compression and comparison in NetEase Games and have received quite positive reviews from user studies. More future work is in progress since this is the first work that introduces XAI in online games.