Global Explanations Research Articles

Explainable Boosting Machine for Choosing Radiation Dose-Volume Constraints on Cardio-Pulmonary Substructures Associated with Overall Survival in the NRG Oncology RTOG 0617 Clinical Trial

<h3>Purpose/Objective(s)</h3> Black box (BB) machine learning models are frequently used for decision making in healthcare. Although posthoc model-agnostic explanations can improve the BB model's interpretability, they are often unreliable. Explainable boosting machine (EBM) is a glass box model that allows for model-specific interpretation of each feature's contribution to outcome prediction. We used EBM for choosing radiation dose-volume constraints (DVCs) on cardio-pulmonary substructures (CPSs) associated with overall survival (OS) in locally advanced non-small cell lung cancer (NSCLC) radiotherapy. <h3>Materials/Methods</h3> A total of 379 NSCLC patients enrolled in RTOG 0617 were randomly split into training/validation subsets (n=230/149). A deep learning autosegmentation model was used for segmenting 12 CPSs: heart, atria, ventricles, aorta, left ventricle/atrium (LV/LA), right ventricle/atrium (RV/RA), inferior/superior vena cava, pulmonary artery, and pericardium. Contours of peri (=pericardium-heart), rest [=heart-(LV|RV|LA|RA)], clinical target volume and lungs were also obtained. From each structure, dose-volume histogram (DVH) features including min/mean dose to the hottest x% volume (Dx%[Gy]/MOHx%[Gy]; x was 5-100 in 5% increments) and min/mean/max dose as well as percent volume receiving at least xGy (VxGy[%]; x was 5-75 in 5Gy increments) were extracted. Unsupervised multivariate multiblock regression was used to select DVH features in each structure that were canonically correlated with 38 clinical/dosimetric/outcome variables taken from the NTCN/NCORP data archive for RTOG 0617. Additional feature selection for OS analysis was performed using stability selection supervised principal component analysis and Boruta. Algorithm for survival stacking was implemented for training EBM by casting survival analysis as a classification problem. Discrimination performance of EBM for OS in the validation set was evaluated by Harrell's c-index and compared to that of Cox proportional hazards (CPH) and 3 BB models: random survival forest (RSF), deep neural network (DeepSurv) and extreme gradient boosting (XGB). Shapley additive explanations (SHAP) were used to explain their predictions. <h3>Results</h3> Selected features included RA_V55Gy[%], aorta_V15Gy[%], LV_D5%[Gy] and pericardium_D20%[Gy]. EBM and all 3 BB models ranked LV_D5%[Gy] as the most important feature, while CPH model ranked second. EBM achieved the best performance for predicting OS, followed by DeepSurv, XGB, RSF and CPH (c-index=0.657, 0.646, 0.643, 0.632 and 0.630). Global explanations of EBM suggested that LV_D5%[Gy]<17.2Gy, pericardium_D20%[Gy]<52.6Gy, RA_V55Gy[%]<12.9% and aorta_V15Gy[%]<49.4% for improved OS. Although SHAP values of CPH and all 3 BB models showed similar trends, they were unable to assist in the precise selection of DVCs. <h3>Conclusion</h3> EBM can be used for modeling OS while also providing guidance on CPS DVC selection for optimal cardiac toxicity management in lung cancer radiotherapy.

Interpretable Machine Learning Model for Predicting Pathologic Complete Response in Patients with Rectal Adenocarcinoma Treated with Chemoradiation Therapy

<h3>Purpose/Objective(s)</h3> Following neoadjuvant chemoradiotherapy (nCRT), pathologic complete response (pCR) strongly influences the decision to proceed with surgery versus "watchful waiting" in rectal cancer (RC) patients. The purpose of this study is to predict pCR without using invasive procedures. An interpretable machine learning model trained with clinical and imaging data from diagnosis and treatment, with extracted radiomics (R) and dosiomics (D) features is utilized to gain insight into contributing factors. <h3>Materials/Methods</h3> This study used multi-institutional datasets, including a training set of 180 patients from our institution and an independent test set of 37 patients from the RTOG 0822 clinical trial. Each patient had a radiotherapy planning CT and the associated contours of the gross tumor volume (GTV) and the organ-at-risks (OARs) including the bladder, bowel_samll, and femur_heads. A total of 296 features including clinical parameters (CP), GTV and OAR dose-volume histogram (DVH), GTV R, and GTV D features were extracted. R and D features were subcategorized into the first- (L1), second- (L2), and higher-order (L3) local texture features. Multiview input data analysis was performed to identify an optimal set of input feature categories by using an exhaustive search. For each input, feature selection was performed using Boruta, followed by collinearity removal based on the variance inflation factor. Explainable boosting machine (EBM), an interpretable glass-box model, was trained using selected features. The performance of EBM on the test set was evaluated using the area under the receiver operating characteristic curve (AUC) and compared with that of 3 state-of-the-art black-box models: extreme gradient boosting (XGB), random forest (RF), and support vector machine (SVM). <h3>Results</h3> Selected features included two shape (elongation, maximum2DDiameterColumn) and one first-order (variance) features in R. Global explanations of EBM showed that tumors with maximum2DDiameterColumn <40 or >90 mm, elongation <0.55, and higher variance of CT intensities were associated with unfavorable outcomes. For all 4 models, the best predictive performance was obtained with an optimal input feature subset of CP+DVH+L1_R+L1_D, which significantly outperformed a full feature set of CP+DVH+R_all+D_all (AUC=0.602 to 0.618). The EBM had the best performance for predicting pCR (AUC=0.855), followed by RF, SVM, and XGB (AUC=0.839, 0.822, and 0.809). <h3>Conclusion</h3> EBM has the potential to improve the predictability of pCR while also enhancing model interpretability, which can aid in radiotherapy decision-making. Using an optimal input feature set selected by multi-view input analysis can lead to improved performance compared to concatenating all available features, which may, in turn, have implications in patient selection for a "watchful-waiting" approach in RC management.

RETRACTED ARTICLE: Prediction of gestational diabetes based on explainable deep learning and fog computing

Gestational diabetes mellitus (GDM) is one of the pregnancy complications that endangers both mothers and babies. GDM is usually diagnosed at 22–26 weeks of gestation. However, early prediction is preferable because it may decrease the risk. The continuous monitoring of the mother’s vital signs helps in predicting any deterioration during pregnancy. The originality of this research is to provide a comprehensive framework for pregnancy women monitoring. The proposed Data Replacement and Prediction Framework consists of three layers, which are: (i) Internet of things (IoT) Layer, (ii) Fog Layer, and (iii) Cloud Layer. The first layer used IoT sensors to aggregate vital signs from pregnancies using invasive and non-invasive sensors. The vital signs are then transmitted to fog nodes to be processed and finally stored in the cloud layer. The main contribution in this research is located in the fog layer producing the GDM module to implement two influential tasks which are as follows: (i) Data Finding Methodology (DFM), and (ii) Explainable Prediction Algorithm (EPM) using DNN. First, the DFM is used to replace the unused data to free up the cache space for new incoming data items. The cache replacement is very important in the case of the healthcare system as the incoming vital signs are frequent and must be replaced continuously. Second, the EPM is used to predict the occurrence of GDM in the second trimester of the pregnancy. To evaluate our model, we extracted data from 16,354 pregnant women from the medical information mart for intensive care (MIMIC III) benchmark dataset. For each woman, vital signs, demographic data, and laboratory tests were aggregated. The results of the prediction model are superior to the state-of-the-art (ACC = 0.957, AUC = 0.942). Regarding explainability, we used Shapley additive explanation (SHAP) framework to provide local and global explanations for the developed models. Overall, the proposed framework is medically intuitive and allows the early prediction of GDM with a cost-effective solution.

Has legislative power declined globally?

ABSTRACT In recent years several scholars have argued that there has been a weakening of the power of national legislatures. This paper addresses two empirical questions: Has global legislative power indeed declined (as has been claimed)? And if so, what factors might explain where legislative power has declined and where it has not? Using data from the Varieties of Democracy (V-dem) project from 1990 to 2020 to measure changes in legislative power, I find that the decline of legislative power has not been monotonic globally, and that there is considerable variation by countries in the world. In this paper, I test a set of institutional, economic, cultural and global explanations for legislative decline that has been proposed by the literature and find that much of this literature is either not supported, or only partially supported.

DDIML: Explainable detection model for drive-by-download attacks

A drive-by download is a method of hackers planting the Web Trojan, which exploits browser vulnerabilities to execute malicious software. Because people usually access web pages with various browsers daily, drive-by downloads have become one of the most common threats in recent years. Most previous studies utilize the abstract syntax tree(AST) with deep learning methods to detect such attacks, which achieved high accuracy but are time-consuming and challenging to explain. Also, some methods use dynamic analysis, which needs a specific environment and is time-consuming with the complex operation. In order to solve these problems, the paper proposes DDIML, an explainable machine learning model based on novel features with static analysis. These features are extracted from five aspects: code obfuscation, URL redirection, special behaviors, encoding characters, and CSS attributes. The most popular machine learning algorithm, Random forest, is applied for building the classifier detection model. In addition, we use both local and global explanations to improve the model and prove that the proposed model could be trusted. The Experimental results show that our proposed model can efficiently detect drive-by downloads with a detection precision of 0.983 and a recall of 0.980. The average detection time for each sample is only 16.07ms in total.

XFDDC: eXplainable Fault Detection Diagnosis and Correction framework for chemical process systems

Industry 4.0 process fault detection and diagnosis(FDD) is built on the foundations of Industrial Internet of Things(IIoT) for sensing and artificial intelligence for recognizing patterns. Despite the performance and application of various learning-based algorithms in multiple sectors, their black-box nature makes industrial experts skeptical. So, this paper proposes an eXplainable Fault Detection, Diagnosis, and Correction(XFDDC) Framework to create best-fit FDD models that are explainable. The XFDDC framework is designed to explain the FDD model predictions using eXplainable Artificial Intelligence(XAI) techniques. The proposed framework was applied to the bench-marked Tennessee Eastman Process(TEP) dataset. On evaluation, an XGBoost model yielded better Fault Detection Rate(FDR) and F1 score against popular transparent and complex models like Naive Bayes, K-Nearest Neighbors, Random Forest and a rule-based version of XGBoost. To explain the predictions of the XGBoost model, the XFDDC framework suggests the use of feature-based XAI techniques. So, the TreeSHAP algorithm is applied on the XGBoost model to generate local and global explanations as part of fault diagnosis. The proposed framework also recommends counterfactual explanations to provide action recommendations for correcting the fault situation. Thus, a best-fit explainable XGBoost Fault Detection, Diagnosis, and Correction (XGBoost-XFDDC) model is created.

Identifying the influence of surface texture waveforms on colors of polished surfaces using an explainable AI approach

An explainable artificial intelligence approach based on consolidating the Local Interpretable and Model-agnostic Explanation (LIME) model outputs was devised to discern the influence of the surface morphology on the colors exhibited by stainless-steel 304 parts polished with a Magnetic Abrasive Finishing (MAF) process. The MAF polishing process was used to create two regions, each appearing either blue or red to the naked eye. The color distribution was microscopically heterogeneous, i.e., some red microscale patches were dispersed in blue regions, and vice versa. The surface morphology was represented in the frequency domain (using a 2D Fourier transform) to capture the harmonic surface patterns, such as the feed and lay marks from the polishing process. A Convolutional Neural Network (CNN) was employed to identify the color of the region from the frequency characteristics of the surface morphology. The CNN was able to predict the observed colors with test accuracies exceeding 99%, suggesting that the frequency characteristics of the surface morphology of the red regions are distinctly different from those of the blue regions. A LIME model was constructed around each small segment within each region of the surface to identify the frequency features that are influential for differentiating between the colors. To deal with the effect of heterogeneity, an algorithm based on the query by experts was used to reconcile the local influences and gather the global explanations of the frequency characteristics that inform the blue versus red regions. We found that the dominant morphological features in the red regions are those that capture the polishing lay patterns underlying surface structure, whereas those in the blue regions capture the non-uniform and high-frequency waveform patterns, such as those result when oxide films form due to the intense polishing conditions.

Trusting deep learning natural-language models via local and global explanations

Despite the high accuracy offered by state-of-the-art deep natural-language models (e.g., LSTM, BERT), their application in real-life settings is still widely limited, as they behave like a black-box to the end-user. Hence, explainability is rapidly becoming a fundamental requirement of future-generation data-driven systems based on deep-learning approaches. Several attempts to fulfill the existing gap between accuracy and interpretability have been made. However, robust and specialized eXplainable Artificial Intelligence solutions, tailored to deep natural-language models, are still missing. We propose a new framework, named T-EBAnO, which provides innovative prediction-local and class-based model-global explanation strategies tailored to deep learning natural-language models. Given a deep NLP model and the textual input data, T-EBAnO provides an objective, human-readable, domain-specific assessment of the reasons behind the automatic decision-making process. Specifically, the framework extracts sets of interpretable features mining the inner knowledge of the model. Then, it quantifies the influence of each feature during the prediction process by exploiting the normalized Perturbation Influence Relation index at the local level and the novel Global Absolute Influence and Global Relative Influence indexes at the global level. The effectiveness and the quality of the local and global explanations obtained with T-EBAnO are proved on an extensive set of experiments addressing different tasks, such as a sentiment-analysis task performed by a fine-tuned BERT model and a toxic-comment classification task performed by an LSTM model. The quality of the explanations proposed by T-EBAnO, and, specifically, the correlation between the influence index and human judgment, has been evaluated by humans in a survey with more than 4000 judgments. To prove the generality of T-EBAnO and its model/task-independent methodology, experiments with other models (ALBERT, ULMFit) on popular public datasets (Ag News and Cola) are also discussed in detail.

An explainable artificial intelligence approach for financial distress prediction

External stakeholders require accurate and explainable financial distress prediction (FDP) models. Complex machine learning algorithms offer high accuracy, but most of them lack explanatory power, resulting in external stakeholders being cautious in adopting them. Therefore, an explainable artificial intelligence approach including a whole process ensemble method and an explainable frame for FDP is here proposed. The ensemble algorithm from feature selection to predictor construction can achieve high accuracy according to the actual case, and the interpretation framework can meet the needs of external users by generating local explanations and global explanations. First, a two-stage scheme integrated with a filter and wrapper technique is designed for feature selection. Second, multiple ensemble models are explored and they are evaluated according to the actual case. Finally, Shapley additive explanations, counterfactual explanations and partial dependence plots are employed to enhance model interpretability. Taking financial data of Chinese listed companies from 2007 to 2020 as a dataset, the highest AUC is ensured by LightGBM with a value of 0.92. Local explanations help individual enterprises identify the key features which lead to their financial distress, and counterfactual explanations are produced to provide improvement strategies. By analyzing the features importance and the impact of feature interaction on the results, global explanations can improve the transparency and credibility of ‘black box’ models.

Radicalization Analyzed by Social Sciences: Can the medium-range concepts already mobilized on urban riots explain the radicalization processes in France?

The purpose of this paper is to question the concepts that social scientists use to apprehend the processes of so-called Islamic radicalization. While some academic disciplines—such as geopolitics or psychology—provide global explanations of radicalizatio

Abnormality Detection and Failure Prediction Using Explainable Bayesian Deep Learning: Methodology and Case Study with Industrial Data

Mistrust, amplified by numerous artificial intelligence (AI) related incidents, is an issue that has caused the energy and industrial sectors to be amongst the slowest adopter of AI methods. Central to this issue is the black-box problem of AI, which impedes investments and is fast becoming a legal hazard for users. Explainable AI (XAI) is a recent paradigm to tackle such an issue. Being the backbone of the industry, the prognostic and health management (PHM) domain has recently been introduced into XAI. However, many deficiencies, particularly the lack of explanation assessment methods and uncertainty quantification, plague this young domain. In the present paper, we elaborate a framework on explainable anomaly detection and failure prognostic employing a Bayesian deep learning model and Shapley additive explanations (SHAP) to generate local and global explanations from the PHM tasks. An uncertainty measure of the Bayesian model is utilized as a marker for anomalies and expands the prognostic explanation scope to include the model’s confidence. In addition, the global explanation is used to improve prognostic performance, an aspect neglected from the handful of studies on PHM-XAI. The quality of the explanation is examined employing local accuracy and consistency properties. The elaborated framework is tested on real-world gas turbine anomalies and synthetic turbofan failure prediction data. Seven out of eight of the tested anomalies were successfully identified. Additionally, the prognostic outcome showed a 19% improvement in statistical terms and achieved the highest prognostic score amongst best published results on the topic.

Explainable Artificial Intelligence for COVID-19 Diagnosis Through Blood Test Variables

This work proposes an explainable artificial intelligence approach to help diagnose COVID-19 patients based on blood test and pathogen variables. Two glass-box models, logistic regression and explainable boosting machine, and two black-box models, random forest and support vector machine, were used to assess the disease diagnosis. Shapley additive explanations were used to explain predictions for the black-box models, while glass-box models feature importance brought insights into the most relevant features. All global explanations show the eosinophils and leukocytes, white blood cells are among the essential features to help diagnose the COVID-19. Moreover, the best model obtained an AUC of 0.87.

“Why Should I Trust Your IDS?”: An Explainable Deep Learning Framework for Intrusion Detection Systems in Internet of Things Networks

Internet of Things (IoT) is an emerging paradigm that is turning and revolutionizing worldwide cities into smart cities. However, this emergence is accompanied with several cybersecurity concerns due mainly to the data sharing and constant connectivity of IoT networks. To address this problem, multiple Intrusion Detection Systems (IDSs) have been designed as security mechanisms, which showed their efficiency in mitigating several IoT-related attacks, especially when using deep learning (DL) algorithms. Indeed, Deep Neural Networks (DNNs) significantly improve the detection rate of IoT-related intrusions. However, DL-based models are becoming more and more complex, and their decisions are hardly interpreted by users, especially companies&#x2019; executive staff and cybersecurity experts. Hence, the corresponding users cannot neither understand and trust DL models decisions, nor optimize their decisions (users) based on DL models outputs. To overcome these limits, Explainable Artificial Intelligence (XAI) is an emerging paradigm of Artificial Intelligence (AI), that provides a set of techniques to help interpreting and understanding predictions made by DL models. Thus, XAI enables to explain the decisions of DL-based IDSs to make them interpretable by cybersecurity experts In this paper, we design a new XAI-based framework to give explanations to any critical DL-based decisions for IoT-related IDSs. Our framework relies on a novel IDS for IoT networks, that we also develop by leveraging deep neural network, to detect IoT-related intrusions. In addition, our framework uses three main XAI techniques (i.e., RuleFit, Local Interpretable Model-Agnostic Explanations (LIME), and SHapley Additive exPlanations (SHAP)), on top of our DNN-based model. Our framework can provide both local and global explanations to optimize the interpretation of DL-based decisions. The local explanations target a single/particular DL output, while global explanations focus on deducing the most important features that have conducted to each made decision (e.g., intrusion detection). Thus, our proposed framework introduces more transparency and trust between the decisions made by our DL-based IDS model and cybersecurity experts. Both NSL-KDD and UNSW-NB15 datasets are used to validate the feasibility of our XAI framework. The experimental results show the efficiency of our framework to improve the interpretability of the IoT IDS against well-known IoT attacks, and help the cybersecurity experts get a better understanding of IDS decisions.

Enhancing Cluster Analysis With Explainable AI and Multidimensional Cluster Prototypes

Explainable Artificial Intelligence (XAI) aims to introduce transparency and intelligibility into the decision-making process of AI systems. Most often, its application concentrates on supervised machine learning problems such as classification and regression. Nevertheless, in the case of unsupervised algorithms like clustering, XAI can also bring satisfactory results. In most cases, such application is based on the transformation of an unsupervised clustering task into a supervised one and providing generalised global explanations or local explanations based on cluster centroids. However, in many cases, the global explanations are too coarse, while the centroid-based local explanations lose information about cluster shape and distribution. In this paper, we present a novel approach called CLAMP (Cluster Analysis with Multidimensional Prototypes) that aids experts in cluster analysis with human-readable rule-based explanations. The developed state-of-the-art explanation mechanism is based on cluster prototypes represented by multidimensional bounding boxes. This allows representing of arbitrary shaped clusters and combines the strengths of local explanations with the generality of global ones.We demonstrate and evaluate the use of our approach in a real-life industrial case study from the domain of steel manufacturing as well as on the benchmark datasets. The explanations generated with CLAMP were more precise than either centroid-based or global ones.

SurvNAM: The machine learning survival model explanation

An extension of the Neural Additive Model (NAM) called SurvNAM and its modifications are proposed to explain predictions of a black-box machine learning survival model. The method is based on applying the original NAM to solving the explanation problem in the framework of survival analysis. The basic idea behind SurvNAM is to train the network by means of a specific expected loss function which takes into account peculiarities of the survival model predictions. Moreover, the loss function approximates the black-box model by the extension of the Cox proportional hazards model, which uses the well-known Generalized Additive Model (GAM) in place of the simple linear relationship of covariates. The proposed method SurvNAM allows performing local and global explanations. The global explanation uses the whole training dataset. In contrast to the global explanation, a set of synthetic examples around the explained example are randomly generated for the local explanation. The proposed modifications of SurvNAM are based on using the Lasso-based regularization for functions from GAM and for a special representation of the GAM functions using their weighted linear and non-linear parts, which is implemented as a shortcut connection. Many numerical experiments illustrate efficiency of SurvNAM.

User‐guidedglobal explanations for deep image recognition: A user study

AbstractWe study a user‐guided approach for producing global explanations of deep networks for image recognition. The global explanations are produced with respect to a test data set and give the overall frequency of different “recognition reasons” across the data. Each reason corresponds to a small number of the most significant human‐recognizable visual concepts used by the network. The key challenge is that the visual concepts cannot be predetermined and those concepts will often not correspond to existing vocabulary or have labeled data sets. We address this issue via an interactive‐naming interface, which allows users to freely cluster significant image regions in the data into visually similar concepts. Our main contribution is a user study on two visual recognition tasks. The results show that the participants were able to produce a small number of visual concepts sufficient for explanation and that there was significant agreement among the concepts, and hence global explanations, produced by different participants.

Feature-Based Interpretation of the Deep Neural Network

The significant advantage of deep neural networks is that the upper layer can capture the high-level features of data based on the information acquired from the lower layer by stacking layers deeply. Since it is challenging to interpret what knowledge the neural network has learned, various studies for explaining neural networks have emerged to overcome this problem. However, these studies generate the local explanation of a single instance rather than providing a generalized global interpretation of the neural network model itself. To overcome such drawbacks of the previous approaches, we propose the global interpretation method for the deep neural network through features of the model. We first analyzed the relationship between the input and hidden layers to represent the high-level features of the model, then interpreted the decision-making process of neural networks through high-level features. In addition, we applied network pruning techniques to make concise explanations and analyzed the effect of layer complexity on interpretability. We present experiments on the proposed approach using three different datasets and show that our approach could generate global explanations on deep neural network models with high accuracy and fidelity.

Modelling local and global explanations for shape aftereffects with naturalistic novel stimuli

Modelling local and global explanations for shape aftereffects with naturalistic novel stimuli

Toward Learning Trustworthily from Data Combining Privacy, Fairness, and Explainability: An Application to Face Recognition.

In many decision-making scenarios, ranging from recreational activities to healthcare and policing, the use of artificial intelligence coupled with the ability to learn from historical data is becoming ubiquitous. This widespread adoption of automated systems is accompanied by the increasing concerns regarding their ethical implications. Fundamental rights, such as the ones that require the preservation of privacy, do not discriminate based on sensible attributes (e.g., gender, ethnicity, political/sexual orientation), or require one to provide an explanation for a decision, are daily undermined by the use of increasingly complex and less understandable yet more accurate learning algorithms. For this purpose, in this work, we work toward the development of systems able to ensure trustworthiness by delivering privacy, fairness, and explainability by design. In particular, we show that it is possible to simultaneously learn from data while preserving the privacy of the individuals thanks to the use of Homomorphic Encryption, ensuring fairness by learning a fair representation from the data, and ensuring explainable decisions with local and global explanations without compromising the accuracy of the final models. We test our approach on a widespread but still controversial application, namely face recognition, using the recent FairFace dataset to prove the validity of our approach.

Local and global explanations of agent behavior: Integrating strategy summaries with saliency maps

With advances in reinforcement learning (RL), agents are now being developed in high-stakes application domains such as healthcare and transportation. Explaining the behavior of these agents is challenging, as the environments in which they act have large state spaces, and their decision-making can be affected by delayed rewards, making it difficult to analyze their behavior. To address this problem, several approaches have been developed. Some approaches attempt to convey the global behavior of the agent, describing the actions it takes in different states. Other approaches devised local explanations which provide information regarding the agent's decision-making in a particular state. In this paper, we combine global and local explanation methods, and evaluate their joint and separate contributions, providing (to the best of our knowledge) the first user study of combined local and global explanations for RL agents. Specifically, we augment strategy summaries that extract important trajectories of states from simulations of the agent with saliency maps which show what information the agent attends to. Our results show that the choice of what states to include in the summary (global information) strongly affects people's understanding of agents: participants shown summaries that included important states significantly outperformed participants who were presented with agent behavior in a set of world-states that are likely to appear during gameplay. We find mixed results with respect to augmenting demonstrations with saliency maps (local information), as the addition of saliency maps, in the form of raw heat maps, did not significantly improve performance in most cases. However, we do find some evidence that saliency maps can help users better understand what information the agent relies on during its decision-making, suggesting avenues for future work that can further improve explanations of RL agents.