Explainable AI: A Review of Machine Learning Interpretability Methods

This study investigates the use of machine learning (ML) models for wastewater treatment plant (WWTP) sludge predictions and explainable artificial intelligence (XAI) techniques for understanding the impact of variables behind the prediction. Three ML models, random forest (RF), gradient boosting machine (GBM), and gradient boosting tree (GBT), were evaluated for their performance using statistical indicators. Input variable combinations were selected through different feature selection (FS) methods. XAI techniques were employed to enhance the interpretability and transparency of ML models. The results suggest that prediction accuracy depends on the choice of model and the number of variables. XAI techniques were found to be effective in interpreting the decisions made by each ML model. This study provides an example of using ML models in sludge production prediction and interpreting models applying XAI to understand the factors influencing it. Understandable interpretation of ML model prediction can facilitate targeted interventions for process optimization and improve the efficiency and sustainability of wastewater treatment processes. PRACTITIONER POINTS: Explainable artificial intelligence can play a crucial role in promoting trust between machine learning models and their real-world applications. Widely practiced machine learning models were used to predict sludge production of a United States wastewater treatment plant. Feature selection methods can reduce the required number of input variables without compromising model accuracy. Explainable artificial intelligence techniques can explain driving variables behind machine learning prediction.

Thyroid disease classification plays a crucial role in early diagnosis and effective treatment of thyroid disorders. Machine learning (ML) techniques have demonstrated remarkable potential in this domain, offering accurate and efficient diagnostic tools. Most of the real-life datasets have imbalanced characteristics that hamper the overall performance of the classifiers. Existing data balancing techniques process the whole dataset at a time that sometimes causes overfitting and underfitting. However, the complexity of some ML models, often referred to as "black boxes," raises concerns about their interpretability and clinical applicability. This paper presents a comprehensive study focused on the analysis and interpretability of various ML models for classifying thyroid diseases. In our work, we first applied a new data-balancing mechanism using a clustering technique and then analyzed the performance of different ML algorithms. To address the interpretability challenge, we explored techniques for model explanation and feature importance analysis using eXplainable Artificial Intelligence (XAI) tools globally as well as locally. Finally, the XAI results are validated with the domain experts. Experimental results have shown that our proposed mechanism is efficient in diagnosing thyroid disease and can explain the models effectively. The findings can contribute to bridging the gap between adopting advanced ML techniques and the clinical requirements of transparency and accountability in diagnostic decision-making.

Explainable artificial intelligence (XAI) is a growing field that aims to increase the transparency and interpretability of machine learning (ML) models. The aim of this work is to use the categorical properties of learning algorithms in conjunction with the categorical perspective of the information in the datasets to give a framework for explainability. In order to achieve this, we are going to define the enriched categories, with decorated morphisms, LearnLearn , ParaPara and MNetMNet of learners, parameterized functions, and neural networks over metric spaces respectively. The main idea is to encode information from the dataset via categorical methods, see how it propagates, and lastly, interpret the results thanks again to categorical (metric) information. This means that we can attach numerical (computable) information via enrichment to the structural information of the category. With this, we can translate theoretical information into parameters that are easily understandable. We will make use of different categories of enrichment to keep track of different kinds of information. That is, to see how differences in attributes of the data are modified by the algorithm to result in differences in the output to achieve better separation. In that way, the categorical framework gives us an algorithm to interpret what the learning algorithm is doing. Furthermore, since it is designed with generality in mind, it should be applicable in various different contexts. There are three main properties of category theory that help with the interpretability of ML models: formality, the existence of universal properties, and compositionality. The last property offers a way to combine smaller, simpler models that are easily understood to build larger ones. This is achieved by formally representing the structure of ML algorithms and information contained in the model. Finally, universal properties are a cornerstone of category theory. They help us characterize an object, not by its attributes, but by how it interacts with other objects. Thus, we can formally characterize an algorithm by how it interacts with the data. The main advantage of the framework is that it can unify under the same language different techniques used in XAI. Thus, using the same language and concepts we can describe a myriad of techniques and properties of ML algorithms, streamlining their explanation and making them easier to generalize and extrapolate.

Artificial Intelligence (AI) is widely used in decision making procedures in myriads of real-world applications across important practical areas such as finance, healthcare, education, and safety critical systems. Due to its ubiquitous use in safety and privacy critical domains, it is often vital to understand the reasoning behind the AI decisions, which motivates the need for explainable AI (XAI). One of the major approaches to XAI is represented by computing so-called interpretable machine learning (ML) models, such as decision trees (DT), decision lists (DL) and decision sets (DS). These models build on the use of if-then rules and are thus deemed to be easily understandable by humans. A number of approaches have been proposed in the recent past to devising all kinds of interpretable ML models, the most prominent of which involve encoding the problem into a logic formalism, which is then tackled by invoking a reasoning or discrete optimization procedure. This paper overviews the recent advances of the reasoning and constraints based approaches to learning interpretable ML models and discusses their advantages and limitations.

Local interpretable model-agnostic explanation approach for medical imaging analysis: A systematic literature review.

In recent years, the field of Artificial Intelligence (AI) and Machine Learning (ML) has witnessed remarkable advancements, revolutionizing various industries and domains. The proliferation of data availability, computational power, and algorithmic innovations has propelled the development of highly sophisticated AI models, particularly in the realm of Deep Learning (DL). These DL models have demonstrated unprecedented levels of accuracy and performance across a wide range of tasks, including image recognition, natural language processing, and complex decision-making. However, amidst these impressive achievements, a critical challenge has emerged - the lack of interpretability.Highly accurate AI models, including DL models, are often referred to as black boxes because their internal workings and decision-making processes are not readily understandable to humans. While these models excel in generating accurate predictions or classifications, they do not provide clear explanations for their reasoning, leaving users and stakeholders in the dark about how and why specific decisions are made. This lack of interpretability raises concerns and limits the trust that humans can place in these models, particularly in safety-critical or high-stakes applications where accountability, transparency, and understanding are paramount.To address the challenge of interpretability, Explainable AI (xAI) has emerged as a multidisciplinary field that aims to bridge the gap in understanding between machines and humans. xAI encompasses a collection of methods and techniques designed to shed light on the decision-making processes of AI models, making their outputs more transparent, interpretable, and comprehensible to human users.The main objective of this paper is to enhance the explainability of AI-based systems that involve user interaction by employing various xAI methods. The proposed approach revolves around a comprehensive ML workflow, beginning with the utilization of real-world data to train a machine learning model that learns the behavior of a simulated driver. The training process encompasses a diverse range of real-world driving scenarios, ensuring that the model captures the intricacies and nuances of different driving situations. This training data serves as the foundation for the subsequent phases of the workflow, where the model's predictive performance is evaluated.Following the training and testing phases, the predictions generated by the ML model are subjected to explanation using different xAI methods, such as LIME (Local Interpretable Model-Agnostic Explanations) and SHAP (SHapley Additive exPlanations). These xAI methods operate at both the global and local levels, providing distinct perspectives on the model's decision-making process. Global explanations offer insights into the overall behavior of the ML model, enabling a broader understanding of the patterns, relationships, and features that the model deems significant across different instances. These global explanations contribute to a deeper comprehension of the decision-making process employed by the model, allowing users to gain insights into the underlying factors driving its predictions.In contrast, local explanations offer detailed insights into specific instances or predictions made by the model. By analyzing these local explanations, users can better understand why the model made a particular prediction in a given case. This granular analysis facilitates the identification of potential weaknesses, biases, or areas for improvement in the model's performance. By pinpointing the specific features or factors that contribute to the model's decision in individual instances, local explanations offer valuable insights for refining the model and enhancing its accuracy and reliability.In conclusion, the lack of explainability in AI models, particularly in the realm of DL, presents a significant challenge that hinders trust and understanding between machines and humans. Explainable AI (xAI) has emerged as a vital field of research and practice, aiming to address this challenge by providing methods and techniques to enhance the interpretability and transparency of AI models. This paper focuses on enhancing the explainability of AI-based systems involving user interaction by employing various xAI methods. The proposed ML workflow, coupled with global and local explanations, offers valuable insights into the decision-making processes of the model. By unraveling the scenario-based behavior of a self-learning function with user interaction, this paper aims to contribute to the understanding and interpretability of AI-based systems. The insights gained from this research can pave the way for enhanced user trust, improved model performance, and further advancements in the field of explainable AI.

The rapid advancements in artificial intelligence and machine learning have led to the development of highly sophisticated models capable of superhuman performance in a variety of tasks. However, the increasing complexity of these models has also resulted in them becoming "black boxes", where the internal decision-making process is opaque and difficult to interpret. This lack of transparency and explainability has become a significant barrier to the widespread adoption of these models, particularly in sensitive domains such as healthcare and finance. To address this challenge, the field of Explainable AI has emerged, focusing on developing new methods and techniques to improve the interpretability and explainability of machine learning models. This review paper aims to provide a comprehensive overview of the research exploring the combination of Explainable AI and traditional machine learning approaches, known as "hybrid models". This paper discusses the importance of explainability in AI, and the necessity of combining interpretable machine learning models with black-box models to achieve the desired trade-off between accuracy and interpretability. It provides an overview of key methods and applications, integration techniques, implementation frameworks, evaluation metrics, and recent developments in the field of hybrid AI models. The paper also delves into the challenges and limitations in implementing hybrid explainable AI systems, as well as the future trends in the integration of explainable AI and traditional machine learning. Altogether, this paper will serve as a valuable reference for researchers and practitioners working on developing explainable and interpretable AI systems. Keywords: Explainable AI (XAI), Traditional Machine Learning (ML), Hybrid Models, Interpretability, Transparency, Predictive Accuracy, Neural Networks, Ensemble Methods, Decision Trees, Linear Regression, SHAP (Shapley Additive Explanations), LIME (Local Interpretable Model-agnostic Explanations), Healthcare Analytics, Financial Risk Management, Autonomous Systems, Predictive Maintenance, Quality Control, Integration Techniques, Evaluation Metrics, Regulatory Compliance, Ethical Considerations, User Trust, Data Quality, Model Complexity, Future Trends, Emerging Technologies, Attention Mechanisms, Transformer Models, Reinforcement Learning, Data Visualization, Interactive Interfaces, Modular Architectures, Ensemble Learning, Post-Hoc Explainability, Intrinsic Explainability, Combined Models

Interpretable vs. noninterpretable machine learning models for data-driven hydro-climatological process modeling

Machine Learning Reimagined: The Promise of Interpretability to Combat Bias.

Cardiac disease is a major health issue worldwide and a leading cause of mortality. Recent advancements in machine learning (ML) show potential for early heart disease detection using patient data and electrocardiograms (ECGs). Early identification can significantly reduce death rates and mitigate the impact of heart disease. Delayed and misdiagnosis therapeutic are two major issues with the traditional diagnostic approaches that can worsen diseases and shoot up the healthcare expenses. To cut these medical costs and avoid any faulty diagnostic ML techniques provides a promising solution in health care sector. The most noninvasive and affordable technique adopted for heart diagnostic by healthcare professionals is ECG. In medical ECG is widely used for the diagnosis, detection, and prevention of many cardiac problems. Disregard of many advantages, there are still issues, such as the lack of qualified cardiologists, comorbidities, and the resemblance of heart disease symptoms in ECG readings. Additionally, patient data and ECGs are often unbalanced, complicating the impartial performance of classical ML models. Traditional ECG diagnostics has improved by applying ML techniques and, helping doctors in interpreting complex cardiac disease processes and boosting computer-assisted treatments. Despite their potential, ML models face skepticism from medical professionals due to their “black box” nature and poor explainability. Many ML models for ECG-based heart diseases detection suffer from bias and lack transparency, raising ethical, legal, and social concerns. To address this, interpretable machine learning (IML) models can boost doctor confidence by providing evidence-based, understandable diagnoses. The detection of cardiac diseases using ECG data, investigating bias and fairness in IML models, and suggesting strategies to guarantee equitable model performance across heterogeneous patient populations are the main focus of this systematic literature review.

The growing interest in explainable artificial intelligence(XAI) for critical decision making motivates the need for interpretable machine learning (ML) models. In fact, due to their structure (especially with small sizes), these models are inherently understandable by humans. Recently, several exact methods for computing such models are proposed to overcome weaknesses of traditional heuristic methods by providing more compact models or better prediction quality. Despite their compressed representation of Boolean functions, Binary decision diagrams (BDDs) did not gain enough interest as other interpretable ML models. In this paper, we first propose SAT-based models for learning optimal BDDs (in terms of the number of features) that classify all input examples. Then, we lift the encoding to a MaxSAT model to learn optimal BDDs in limited depths, that maximize the number of examples correctly classified. Finally, we tackle the fragmentation problem by introducing a method to merge compatible subtrees for the BDDs found via the MaxSAT model. Our empirical study shows clear benefits of the proposed approach in terms of prediction quality and interpretability (i.e., lighter size) compared to the state-of-the-art approaches.

ObjectiveTo develop a computed tomography (CT) radiomics-based interpretable machine learning (ML) model to preoperatively predict human epidermal growth factor receptor 2 (HER2) status in bladder cancer (BCa) with multicenter validation.MethodsIn this retrospective study, 207 patients with pathologically confirmed BCa were enrolled and divided into the training set (n = 154) and test set (n = 53). Least absolute shrinkage and selection operator (LASSO) regression was used to identify the most discriminative features in the training set. Five radiomics-based ML models, namely logistic regression (LR), support vector machine (SVM), k-nearest neighbors (KNN), eXtreme Gradient Boosting (XGBoost) and random forest (RF), were developed. The predictive performance of established ML models was evaluated by the area under the receiver operating characteristic curve (AUC). The Shapley additive explanation (SHAP) was used to analyze the interpretability of ML models.ResultsA total of 1218 radiomics features were extracted from the nephrographic phase CT images, and 11 features were filtered for constructing ML models. In the test set, the AUCs of LR, SVM, KNN, XGBoost, and RF were 0.803, 0.709, 0.679, 0.794, and 0.815, with corresponding accuracies of 71.7%, 69.8%, 60.4%, 75.5%, and 75.5%, respectively. RF was identified as the optimal classifier. SHAP analysis showed that texture features (gray level size zone matrix and gray level co-occurrence matrix) were significant predictors of HER2 status.ConclusionsThe radiomics-based interpretable ML model provides a noninvasive tool to predict the HER2 status of BCa with satisfactory discriminatory performance.Critical relevance statementAn interpretable radiomics-based machine learning model can preoperatively predict HER2 status in bladder cancer, potentially aiding in the clinical decision-making process.Key PointsThe CT radiomics model could identify HER2 status in bladder cancer.The random forest model showed a more robust and accurate performance.The model demonstrated favorable interpretability through SHAP method.Graphical

The use of machine learning (ML) models in oncology is growing, but their limited interpretability, where the inner workings can be seen as a ‘black box’, is a barrier to their acceptance. The goal of explainable artificial intelligence (XAI) frameworks, such as Local Interpretable Model-agnostic Explanations and SHapley Additive exPlanations, is to provide light on how the ‘black box’ functions. This chapter examines a number of oncology research areas, including prognostication, diagnostics, radiomics, pathology, treatment selection, radiation treatment processes, and epidemiology, where ML models and XAI have been used to enhance interpretability. When the intricacy of such models would ordinarily make adoption in the clinic difficult, these examples serve as a starting point for further research into how XAI can help facilitate their implementation.

Optimisation and interpretation of machine and deep learning models for improved water quality management in Lake Loktak

BACKGROUND: Depression is highly prevalent in elderly patients with gastrointestinal (GID) or chronic liver diseases (CLD), significantly impairing quality of life and treatment outcomes. This study aimed to develop and validate an interpretable machine learning (ML) model to identify depression risk in this population, overcoming the “black box” limitation of conventional ML. MATERIALS AND METHODS: This prospective analysis utilized data from the baseline (2018) and follow-up (2020) waves of the China Health and Retirement Longitudinal Study (CHARLS). Potential predictors measured at baseline were selected via Least Absolute Shrinkage and Selection Operator (LASSO) regression. The outcome was incident depression at the 2020 follow-up, defined by a CES-D-10 score ≥ 10 among participants free of depression at baseline. Ten ML algorithms were employed to construct models. Performance was evaluated using the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, precision, F1-score, calibration curves, and decision curve analysis. The SHapley Additive exPlanations (SHAP) framework interpreted feature contributions. RESULTS: Among 1,353 participants (424 with depression), LASSO identified 10 key predictors. The Logistic Regression (LR) model demonstrated optimal discriminative performance, with an AUC of 0.723 (95% CI: 0.674–0.772). SHAP analysis revealed the top five predictors: self-reported health, life satisfaction, gender, education, and memory scores. CONCLUSIONS: We developed an interpretable ML model for predicting depression risk in elderly patients with GID or CLD. This tool aids early detection and intervention, potentially improving clinical outcomes in this vulnerable population.