Explainable AI Models Research Articles

Explainable AI identifies diagnostic cells of genetic AML subtypes.

Explainable AI is deemed essential for clinical applications as it allows rationalizing model predictions, helping to build trust between clinicians and automated decision support tools. We developed an inherently explainable AI model for the classification of acute myeloid leukemia subtypes from blood smears and found that high-attention cells identified by the model coincide with those labeled as diagnostically relevant by human experts. Based on over 80,000 single white blood cell images from digitized blood smears of 129 patients diagnosed with one of four WHO-defined genetic AML subtypes and 60 healthy controls, we trained SCEMILA, a single-cell based explainable multiple instance learning algorithm. SCEMILA could perfectly discriminate between AML patients and healthy controls and detected the APL subtype with an F1 score of 0.86±0.05 (mean±s.d., 5-fold cross-validation). Analyzing a novel multi-attention module, we confirmed that our algorithm focused with high concordance on the same AML-specific cells as human experts do. Applied to classify single cells, it is able to highlight subtype specific cells and deconvolve the composition of a patient's blood smear without the need of single-cell annotation of the training data. Our large AML genetic subtype dataset is publicly available, and an interactive online tool facilitates the exploration of data and predictions. SCEMILA enables a comparison of algorithmic and expert decision criteria and can present a detailed analysis of individual patient data, paving the way to deploy AI in the routine diagnostics for identifying hematopoietic neoplasms.

A survey of ASER members on artificial intelligence in emergency radiology: trends, perceptions, and expectations.

There is a growing body of diagnostic performance studies for emergency radiology-related artificial intelligence/machine learning (AI/ML) tools; however, little is known about user preferences, concerns, experiences, expectations, and the degree of penetration of AI tools in emergency radiology. Our aim is to conduct a survey of the current trends, perceptions, and expectations regarding AI among American Society of Emergency Radiology (ASER) members. An anonymous and voluntary online survey questionnaire was e-mailed to all ASER members, followed by two reminder e-mails. A descriptive analysis of the data was conducted, and results summarized. A total of 113 members responded (response rate 12%). The majority were attending radiologists (90%) with greater than 10years' experience (80%) and from an academic practice (65%). Most (55%) reported use of commercial AI CAD tools in their practice. Workflow prioritization based on pathology detection, injury or disease severity grading and classification, quantitative visualization, and auto-population of structured reports were identified as high-value tasks. Respondents overwhelmingly indicated a need for explainable and verifiable tools (87%) and the need for transparency in the development process (80%). Most respondents did not feel that AI would reduce the need for emergency radiologists in the next two decades (72%) or diminish interest in fellowship programs (58%). Negative perceptions pertained to potential for automation bias (23%), over-diagnosis (16%), poor generalizability (15%), negative impact on training (11%), and impediments to workflow (10%). ASER member respondents are in general optimistic about the impact of AI in the practice of emergency radiology and its impact on the popularity of emergency radiology as a subspecialty. The majority expect to see transparent and explainable AI models with the radiologist as the decision-maker.

Explainable machine learning for long-term outcome prediction in two-center stroke patients after intravenous thrombolysis.

Neurological outcome prediction in patients with ischemic stroke is very critical in treatment strategy and post-stroke management. Machine learning techniques with high accuracy are increasingly being developed in the medical field. We studied the application of machine learning models to predict long-term neurological outcomes in patients with after intravenous thrombolysis. A retrospective cohort study was performed to review all stroke patients with intravenous thrombolysis. Patients with modified Rankin Score (mRs) less than two at three months post-thrombolysis were considered as good outcome. The clinical features between stroke patients with good and with poor outcomes were compared using three different machine learning models (Random Forest, Support Vector Machine and Logistic Regression) to identify which performed best. Two datasets from the other stroke center were included accordingly for external verification and performed with explainable AI models. Of the 488 patients enrolled in this study, and 374 (76.6%) patients had favorable outcomes. Patients with higher mRs at 3 months had increased systolic pressure, blood glucose, cholesterol (TC), and 7-day National Institute of Health Stroke Scale (NIHSS) score compared to those with lower mRs. The predictability and the areas under the curves (AUC) for the random forest model was relatively higher than support vector machine and LR models. These findings were further validated in the external dataset and similar results were obtained. The explainable AI model identified the risk factors as well. Explainable AI model is able to identify NIHSS_Day7 is independently efficient in predicting neurological outcomes in patients with ischemic stroke after intravenous thrombolysis.

Explainable AI Enabled Infant Mortality Prediction Based on Neonatal Sepsis

Neonatal sepsis is the third most common cause of neonatal mortality and a serious public health problem, especially in developing countries. There have been researches on human sepsis, vaccine response, and immunity. Also, machine learning methodologies were used for predicting infant mortality based on certain features like age, birth weight, gestational weeks, and Appearance, Pulse, Grimace, Activity and Respiration (APGAR) score. Sepsis, which is considered the most determining condition towards infant mortality, has never been considered for mortality prediction. So, we have deployed a deep neural model which is the state of art and performed a comparative analysis of machine learning models to predict the mortality among infants based on the most important features including sepsis. Also, for assessing the prediction reliability of deep neural model which is a black box, Explainable AI models like Dalex and Lime have been deployed. This would help any non-technical personnel like doctors and practitioners to understand and accordingly make decisions.

Explainable Misinformation Detection Across Multiple Social Media Platforms

Web Information Processing (WIP) has enormously impacted modern society since a huge percentage of the population relies on the internet to acquire information. Social Media platforms provide a channel for disseminating information and a breeding ground for spreading misinformation, creating confusion and fear among the population. One of the techniques for the detection of misinformation is machine learning-based models. However, due to the availability of multiple social media platforms, developing and training AI-based models has become a tedious job. Despite multiple efforts to develop machine learning-based methods for identifying misinformation, there has been very limited work on developing an explainable generalized detector capable of robust detection and generating explanations beyond black-box outcomes. Knowing the reasoning behind the outcomes is essential to make the detector trustworthy. Hence employing explainable AI techniques is of utmost importance. In this work, the integration of two machine learning approaches, namely domain adaptation and explainable AI, is proposed to address these two issues of generalized detection and explainability. Firstly the Domain Adversarial Neural Network (DANN) develops a generalized misinformation detector across multiple social media platforms. DANN is employed to generate the classification results for test domains with relevant but unseen data. The DANN-based model, a traditional black-box model, cannot justify and explain its outcome, i.e., the labels for the target domain. Hence a Local Interpretable Model-Agnostic Explanations (LIME) explainable AI model is applied to explain the outcome of the DANN model. To demonstrate these two approaches and their integration for effective explainable generalized detection, COVID-19 misinformation is considered a case study. We experimented with two datasets and compared results with and without DANN implementation. It is observed that using DANN significantly improves the F1 score of classification and increases the accuracy by 5% and AUC by 11%. The results show that the proposed framework performs well in the case of domain shift and can learn domain-invariant features while explaining the target labels with LIME implementation. This can enable trustworthy information processing and extraction to combat misinformation effectively.

Explainable AI and Interpretable Model for Insurance Premium Prediction

Explainable AI and Interpretable Model for Insurance Premium Prediction

Influential factors of intercity patient mobility and its network structure in China

Intercity patient mobility reflects the geographic mismatch between healthcare resources and the population, and has rarely been studied with big data at large spatial scales. In this paper, we investigated the patterns of intercity patient mobility and factors influencing this behavior based on >4 million hospitalization records of patients with chronic kidney disease in China. To provide practical policy recommendations, a role identification framework informed by complex network theory was proposed considering the strength and distribution of connections of cities in mobility networks. Such a mobility network features multiscale community structure with “universal administrative constraints and a few boundary breaches”. We discovered that cross-module visits which accounted for only 20 % of total visits, accounted for >50 % of the total travel distance. The explainable machine learning modeling results revealed that distance has a power-law-like effect on flow volume, and high-quality healthcare resources are an important driving factor. This paper provides not only a methodological reference for patient mobility studies but also valuable insights into public health policies. • More than 4 million multi-year mobility records from a national database were investigated. • The distribution of travel distance of patients differs from the results of general human mobility studies. • The network features multiscale community structure with “universal administrative constraints and a few boundary breaches”. • An identification framework was proposed to understand cities’ roles in the mobility network. • The explainable AI model revealed that distance and high-quality healthcare resources are two critical factors.

Solving transparency in drought forecasting using attention models.

Droughts are one of the most devastating and recurring natural disaster due to a multitude of reasons. Among the different drought studies, drought forecasting is one of the key aspects of effective drought management. The occurrence of droughts is related to a multitude of factors which is a combination of hydro-meteorological and climatic factors. These variables are non-linear in nature, and neural networks have been found to effectively forecast drought. However, classical neural nets often succumb to over-fitting due to various lag components among the variables and therefore, the emergence of new deep learning and explainable models can effectively solve this problem. The present study uses an Attention-based model to forecast meteorological droughts (Standard Precipitation Index) at short-term forecast range (1-3 months) for five sites situated in Eastern Australia. The main aim of the work is to interpret the model outcomes and examine how a deep neural network achieves the forecasting results. The plots show the importance of the variables along with its short-term and long-term dependencies at different lead times. The results indicate the importance of large-scale climatic indices at different sequence dependencies specific to the study site, thus providing an example of the necessity to build a spatio-temporal explainable AI model for drought forecasting. The use of such interpretable models would help the decision-makers and planners to use data-driven models as an effective measure to forecast droughts as they provide transparency and trust while using these models.

Image Embeddings Extracted from CNNs Outperform Other Transfer Learning Approaches in Classification of Chest Radiographs.

To identify the best transfer learning approach for the identification of the most frequent abnormalities on chest radiographs (CXRs), we used embeddings extracted from pretrained convolutional neural networks (CNNs). An explainable AI (XAI) model was applied to interpret black-box model predictions and assess its performance. Seven CNNs were trained on CheXpert. Three transfer learning approaches were thereafter applied to a local dataset. The classification results were ensembled using simple and entropy-weighted averaging. We applied Grad-CAM (an XAI model) to produce a saliency map. Grad-CAM maps were compared to manually extracted regions of interest, and the training time was recorded. The best transfer learning model was that which used image embeddings and random forest with simple averaging, with an average AUC of 0.856. Grad-CAM maps showed that the models focused on specific features of each CXR. CNNs pretrained on a large public dataset of medical images can be exploited as feature extractors for tasks of interest. The extracted image embeddings contain relevant information that can be used to train an additional classifier with satisfactory performance on an independent dataset, demonstrating it to be the optimal transfer learning strategy and overcoming the need for large private datasets, extensive computational resources, and long training times.

Federated Learning of Explainable AI Models in 6G Systems: Towards Secure and Automated Vehicle Networking

This article presents the concept of federated learning (FL) of eXplainable Artificial Intelligence (XAI) models as an enabling technology in advanced 5G towards 6G systems and discusses its applicability to the automated vehicle networking use case. Although the FL of neural networks has been widely investigated exploiting variants of stochastic gradient descent as the optimization method, it has not yet been adequately studied in the context of inherently explainable models. On the one side, XAI permits improving user experience of the offered communication services by helping end users trust (by design) that in-network AI functionality issues appropriate action recommendations. On the other side, FL ensures security and privacy of both vehicular and user data across the whole system. These desiderata are often ignored in existing AI-based solutions for wireless network planning, design and operation. In this perspective, the article provides a detailed description of relevant 6G use cases, with a focus on vehicle-to-everything (V2X) environments: we describe a framework to evaluate the proposed approach involving online training based on real data from live networks. FL of XAI models is expected to bring benefits as a methodology for achieving seamless availability of decentralized, lightweight and communication efficient intelligence. Impacts of the proposed approach (including standardization perspectives) consist in a better trustworthiness of operations, e.g., via explainability of quality of experience (QoE) predictions, along with security and privacy-preserving management of data from sensors, terminals, users and applications.

An Explainable AI Model for Hate Speech Detection on Indonesian Twitter

To avoid citizen disputes, hate speech on social media, such as Twitter, must be automatically detected. The current research in Indonesian Twitter focuses on developing better hate speech detection models. However, there is limited study on the explainability aspects of hate speech detection. The research aims to explain issues that previous researchers have not detailed and attempt to answer the shortcomings of previous researchers. There are 13,169 tweets in the dataset with labels like “hate speech” and “abusive language”. The dataset also provides binary labels on whether hate speech is directed to individual, group, religion, race, physical disability, and gender. In the research, classification is performed by using traditional machine learning models, and the predictions are evaluated using an Explainable AI model, such as Local Interpretable Model-Agnostic Explanations (LIME), to allow users to comprehend why a tweet is regarded as a hateful message. Moreover, models that perform well in classification perceive incorrect words as contributing to hate speech. As a result, such models are unsuitable for deployment in the real world. In the investigation, the combination of XGBoost and logical LIME explanations produces the most logical results. The use of the Explainable AI model highlights the importance of choosing the ideal model while maintaining users’ trust in the deployed model.

Mitigating the Backdoor Attack by Federated Filters for Industrial IoT Applications

The federated learning provides an effective solution to train collaborative models over a large scale of participated Industrial Internet of Things (IIoT) applications with the help of a global server, building an intelligent life. However, the federated learning is vulnerable to the backdoor attack from strong malicious participants. The backdoor attack is inconspicuous and may result in devastating consequences. To resist the attack on IIoT applications, we propose the federated backdoor filter defense that can identify backdoor inputs and restore the data to availability by the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">blur-label-flipping</i> strategy. We build multiple filters with eXplainable AI models on the server and send them to clients randomly, preventing advanced attackers from evading the defense. Our backdoor filters show significant backdoor recognition with the accuracy up to 99%. After the implementation of the blur-label-flipping strategy, victim's local model on suspicious backdoor samples can achieve the accuracy up to 88%.

IHAR—A fog‐driven interpretable human activity recognition system

AbstractHuman activity recognition (HAR) is an active research area that is currently being applied to various healthcare applications such as fall detection, assisted living, etc. These applications make use of the Internet of Things which is widespread across today's world. One of the major challenges in these applications is the need for quick intelligent decisions. The deployment of hierarchical edge‐fog‐cloud computing architecture is the potential solution to address the latency issue. In this article, we proposed an interpretable human activity recognition (IHAR) framework that supports HAR, leveraging the advantages of the fog layer. The proposed framework used a deep learning (DL) model at the cloud infrastructure to classify the activities of humans. The trained DL model is made to run on local fog nodes. However, the DL model being black‐box does not provide any explanations for the output to make them acceptable by the users and physicians. Hence, this article also incorporated an explainable artificial intelligence model at the fog layer, to gain insights into the classified output of the DL model. The effective model‐level insights emphasized the need for explainable HAR. The article also included an analysis of the results of the existing explainable AI models such as LIME and SHAP to understand which model leads to better performance in the domain of HAR. Results show that the SHAP model has a 33% higher success rate in generating explanations when compared to the LIME model.

Accurate auto-labeling of chest X-ray images based on quantitative similarity to an explainable AI model

The inability to accurately, efficiently label large, open-access medical imaging datasets limits the widespread implementation of artificial intelligence models in healthcare. There have been few attempts, however, to automate the annotation of such public databases; one approach, for example, focused on labor-intensive, manual labeling of subsets of these datasets to be used to train new models. In this study, we describe a method for standardized, automated labeling based on similarity to a previously validated, explainable AI (xAI) model-derived-atlas, for which the user can specify a quantitative threshold for a desired level of accuracy (the probability-of-similarity, pSim metric). We show that our xAI model, by calculating the pSim values for each clinical output label based on comparison to its training-set derived reference atlas, can automatically label the external datasets to a user-selected, high level of accuracy, equaling or exceeding that of human experts. We additionally show that, by fine-tuning the original model using the automatically labelled exams for retraining, performance can be preserved or improved, resulting in a highly accurate, more generalized model.

Investigation of a Web-Based Explainable AI Screening for Prolonged Grief Disorder

Losing a loved one through death is known to be one of the most challenging life events. To help the bereaved and their therapists monitor and better understand the factors that contribute to Prolonged Grief Disorder (PGD), we co-designed and studied a web-based explainable AI screening system named “Grief Inquiries Following Tragedy (GIFT).” We used an initial iteration of the system to collect PGD-related data from 611 participants. Using this data, we developed a model that could be used to screen and explain the different factors contributing to PGD. Our results showed that a Random Forest model using Bereavement risk and outcome features performed best in detecting PGD (AUC=0.772), with features such as a negative intepretation of grief and the ability to integrate stressful life events contributing strongly to the model. Afterwards, five grief experts were asked to provide feedback on a mock-up of the results generated by the GIFT model, and discuss the potential value of the explanatory AI model in real-world PGD care. Overall, the grief experts were generally receptive towards using such a tool in a clinical setting and acknowledged the benefit of offering a personalized result to the users based on the explainable AI model. Our results also showed that, in addition to the explainability of the model, the grief experts also preferred a more “empathetic” and “actionable” AI system, especially, when designing for patient end-users.

A Multimodal Approach for the Risk Prediction of Intensive Care and Mortality in Patients with COVID-19.

Background: Although several studies have been launched towards the prediction of risk factors for mortality and admission in the intensive care unit (ICU) in COVID-19, none of them focuses on the development of explainable AI models to define an ICU scoring index using dynamically associated biological markers. Methods: We propose a multimodal approach which combines explainable AI models with dynamic modeling methods to shed light into the clinical features of COVID-19. Dynamic Bayesian networks were used to seek associations among cytokines across four time intervals after hospitalization. Explainable gradient boosting trees were trained to predict the risk for ICU admission and mortality towards the development of an ICU scoring index. Results: Our results highlight LDH, IL-6, IL-8, Cr, number of monocytes, lymphocyte count, TNF as risk predictors for ICU admission and survival along with LDH, age, CRP, Cr, WBC, lymphocyte count for mortality in the ICU, with prediction accuracy 0.79 and 0.81, respectively. These risk factors were combined with dynamically associated biological markers to develop an ICU scoring index with accuracy 0.9. Conclusions: to our knowledge, this is the first multimodal and explainable AI model which quantifies the risk of intensive care with accuracy up to 0.9 across multiple timepoints.

An explainable artificial intelligence approach for decoding the enhancer histone modifications code and identification of novel enhancers in Drosophila

BackgroundEnhancers are non-coding regions of the genome that control the activity of target genes. Recent efforts to identify active enhancers experimentally and in silico have proven effective. While these tools can predict the locations of enhancers with a high degree of accuracy, the mechanisms underpinning the activity of enhancers are often unclear.ResultsUsing machine learning (ML) and a rule-based explainable artificial intelligence (XAI) model, we demonstrate that we can predict the location of known enhancers in Drosophila with a high degree of accuracy. Most importantly, we use the rules of the XAI model to provide insight into the underlying combinatorial histone modifications code of enhancers. In addition, we identified a large set of putative enhancers that display the same epigenetic signature as enhancers identified experimentally. These putative enhancers are enriched in nascent transcription, divergent transcription and have 3D contacts with promoters of transcribed genes. However, they display only intermediary enrichment of mediator and cohesin complexes compared to previously characterised active enhancers. We also found that 10–15% of the predicted enhancers display similar characteristics to super enhancers observed in other species.ConclusionsHere, we applied an explainable AI model to predict enhancers with high accuracy. Most importantly, we identified that different combinations of epigenetic marks characterise different groups of enhancers. Finally, we discovered a large set of putative enhancers which display similar characteristics with previously characterised active enhancers.

SHAP and LIME: An Evaluation of Discriminative Power in Credit Risk.

In credit risk estimation, the most important element is obtaining a probability of default as close as possible to the effective risk. This effort quickly prompted new, powerful algorithms that reach a far higher accuracy, but at the cost of losing intelligibility, such as Gradient Boosting or ensemble methods. These models are usually referred to as “black-boxes”, implying that you know the inputs and the output, but there is little way to understand what is going on under the hood. As a response to that, we have seen several different Explainable AI models flourish in recent years, with the aim of letting the user see why the black-box gave a certain output. In this context, we evaluate two very popular eXplainable AI (XAI) models in their ability to discriminate observations into groups, through the application of both unsupervised and predictive modeling to the weights these XAI models assign to features locally. The evaluation is carried out on real Small and Medium Enterprises data, obtained from official italian repositories, and may form the basis for the employment of such XAI models for post-processing features extraction.

Interpretable and explainable AI (XAI) model for spatial drought prediction

Accurate prediction of any type of natural hazard is a challenging task. Of all the various hazards, drought prediction is challenging as it lacks a universal definition and is getting adverse with climate change impacting drought events both spatially and temporally. The problem becomes more complex as drought occurrence is dependent on a multitude of factors ranging from hydro-meteorological to climatic variables. A paradigm shift happened in this field when it was found that the inclusion of climatic variables in the data-driven prediction model improves the accuracy. However, this understanding has been primarily using statistical metrics used to measure the model accuracy. The present work tries to explore this finding using an explainable artificial intelligence (XAI) model. The explainable deep learning model development and comparative analysis were performed using known understandings drawn from physical-based models. The work also tries to explore how the model achieves specific results at different spatio-temporal intervals, enabling us to understand the local interactions among the predictors for different drought conditions and drought periods. The drought index used in the study is Standard Precipitation Index (SPI) at 12 month scales applied for five different regions in New South Wales, Australia, with the explainable algorithm being SHapley Additive exPlanations (SHAP). The conclusions drawn from SHAP plots depict the importance of climatic variables at a monthly scale and varying ranges of annual scale. We observe that the results obtained from SHAP align with the physical model interpretations, thus suggesting the need to add climatic variables as predictors in the prediction model.

Explainable AI methods in cyber risk management

AbstractArtificial intelligence (AI) methods are becoming widespread, especially when data are not sufficient to build classical statistical models, as is the case for cyber risk management. However, when applied to regulated industries, such as energy, finance, and health, AI methods lack explainability. Authorities aimed at validating machine learning models in regulated fields will not consider black‐box models, unless they are supplemented with further methods that explain why certain predictions have been obtained, and which are the variables that mostly concur to such predictions. Recently, Shapley values have been introduced for this purpose: They are model agnostic, and powerful, but are not normalized and, therefore, cannot become a standardized procedure. In this paper, we provide an explainable AI model that embeds Shapley values with a statistical normalization, based on Lorenz Zonoids, particularly suited for ordinal measurement variables that can be obtained to assess cyber risk.