Explainable artificial intelligence techniques for interpretation of food models: a review

Early detection and accurate identification of melanoma, the most lethal form of skin cancer, are critical for improving patient survival rates. However, conventional diagnostic tools often fail to detect early-stage melanoma due to limited sensitivity. This research introduces an advanced Vision Transformer (ViT)-based model equipped with explainability features, specifically designed to address the complex challenge of early-stage melanoma detection. Self-attention-based Melanoma Analysis using Reliable Transformers with Explainable Artificial Intelligence (SMART-XAI) combines ViT with self-attention methods and explainable AI techniques like Attention Rollout and Self-Attention Attribution to create a melanoma diagnostic system. The ViT architecture leverages multi-scale feature extraction and self-attention mechanisms to capture both local and global patterns within skin lesion images. By segmenting images into patches, the model effectively identifies critical melanoma features across various scales, enhancing classification accuracy and interpretability. To ensure clinical transparency, the model incorporates explainable artificial intelligence (AI) techniques, namely Attention Rollout and Self-Attention Attribution, which enable the visualization of image regions that influence the modelís decisions. This interpretability allows clinicians to understand the diagnostic rationale behind AI, thereby fostering greater trust and usability in clinical settings. Experimental validation demonstrated strong performance, achieving 96.0 ± 0.87% accuracy, 95.2 ± 0.75% sensitivity, and 94.1 ± 1.05% specificity across 5-fold cross-validation. These results confirm the system’s reliability as an early detection tool for melanoma, while providing interpretable insights for informed clinical decision-making. Overall, the integration of deep learning with explainable AI represents a significant advancement in dermatological diagnostics, particularly for the management of early-stage melanoma.

This study investigates the use of machine learning (ML) models for wastewater treatment plant (WWTP) sludge predictions and explainable artificial intelligence (XAI) techniques for understanding the impact of variables behind the prediction. Three ML models, random forest (RF), gradient boosting machine (GBM), and gradient boosting tree (GBT), were evaluated for their performance using statistical indicators. Input variable combinations were selected through different feature selection (FS) methods. XAI techniques were employed to enhance the interpretability and transparency of ML models. The results suggest that prediction accuracy depends on the choice of model and the number of variables. XAI techniques were found to be effective in interpreting the decisions made by each ML model. This study provides an example of using ML models in sludge production prediction and interpreting models applying XAI to understand the factors influencing it. Understandable interpretation of ML model prediction can facilitate targeted interventions for process optimization and improve the efficiency and sustainability of wastewater treatment processes. PRACTITIONER POINTS: Explainable artificial intelligence can play a crucial role in promoting trust between machine learning models and their real-world applications. Widely practiced machine learning models were used to predict sludge production of a United States wastewater treatment plant. Feature selection methods can reduce the required number of input variables without compromising model accuracy. Explainable artificial intelligence techniques can explain driving variables behind machine learning prediction.

Due to the nature of composites, the ability to accurately locate low-energy impacts on structures is crucial for Structural Health Monitoring (SHM) in the aerospace sector. For this purpose, several techniques have been developed in the past, and, among them, Artificial Intelligence (AI) has demonstrated promising results with high performance. The non-linear behavior of AI-based solutions has made them able to withstand scenarios where complex structures and different impact configurations have been introduced, making accurate location predictions. However, the black-box nature of AI poses a challenge in the aerospace field, where reliability, trustworthiness, and validation capability are paramount. To overcome this problem, Explainable Artificial Intelligence (XAI) techniques emerge as a solution, enhancing model transparency, trust, and validation. This research presents a case study: a previously trained Impact-Locator-AI model is, initially, demonstrating a promising location accuracy; however, its behavior in real-life scenarios is unknown, and before embedding it in an aerospace structure as an SHM system its reliability must be tested. By applying XAI methodologies, the Impact-Locator-AI model can be critically evaluated to assess its reliability and potential suitability for aerospace applications, while also laying the groundwork for future research at the intersection of XAI and impact location in SHM.

Abstract: Deep learning models have demonstrated remarkable capabilities across various domains, but their inherent complexity often leads to challenges in understanding and interpreting their decisions. The demand for transparent and interpretable artificial intelligence (AI) systems is particularly crucial in fields such as healthcare, finance, and autonomous systems. This research paper presents a comprehensive study on the application of Explainable AI (XAI) techniques to enhance transparency and interpretability in deep learning models. Keywords: Explainable AI (XAI), artificial intelligence (AI).

Brain tumors pose significant health risks due to their high mortality rates and challenges in early diagnosis. Advances in medical imaging, particularly MRI, combined with artificial intelligence (AI), have revolutionized tumor detection, segmentation, and classification. Despite the high accuracy of models such as Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), their clinical adoption is hampered by a lack of interpretability. This study provides a comprehensive analysis of machine learning, deep learning, and explainable AI (XAI) techniques in brain tumor diagnosis, emphasizing their strengths, limitations, and potential to improve transparency and clinical trust. By reviewing 53 peer-reviewed articles published between 2017 and 2024, we assess the current state of research, identify gaps, and provide practical recommendations for clinicians, regulators, and AI developers. The findings reveal that while XAI techniques, such as Grad-CAM, SHAP, and LIME, significantly enhance model interpretability, challenges remain in terms of generalizability, computational complexity, and dataset quality. Future research should focus on addressing these limitations to fully realize the potential of AI in brain tumor diagnostics.

Plant diseases pose a significant threat to global food security, affecting crop yield, quality, and overall agricultural productivity. Traditionally, diagnosing plant diseases has relied on time-consuming visual inspections by experts, which can often lead to errors. Machine learning (ML) and artificial intelligence (AI), particularly Vision Transformers (ViTs), and Convolutional Neural Networks, offer a faster, automated alternative for identifying plant diseases through leaf image analysis. However, these models are often criticized for their “black box” nature, limiting trust in their predictions due to a lack of transparency. Our findings show that incorporating Explainable AI (XAI) techniques, such as Grad-CAM, Integrated Gradients, and LIME, significantly improves model interpretability, making it easier for practitioners to identify the underlying symptoms of plant diseases. This study not only contributes to the field of plant disease detection but also offers a novel perspective on improving AI transparency in real-world agricultural applications through the use of XAI techniques. With training accuracies of 100.00% for ViT, 96.88% for EfficientNetB7, 93.75% for EfficientNetB0, and 87.50% for ResNet50, and corresponding validation accuracies of 96.39% for ViT, 86.98% for EfficientNetB7, and 82.00% for EfficientNetB0, our proposed models outperform earlier research on the same dataset. This demonstrates a notable improvement in model performance while maintaining transparency and trustworthiness through interpretable and reliable decision-making.

The development of medical imaging artificial intelligence (AI) systems for evaluating COVID‐19 patients has demonstrated potential for improving clinical decision making and assessing patient outcomes during the recent COVID‐19 pandemic. These have been applied to many medical imaging tasks, including disease diagnosis and patient prognosis, as well as augmented other clinical measurements to better inform treatment decisions. Because these systems are used in life‐or‐death decisions, clinical implementation relies on user trust in the AI output. This has caused many developers to utilize explainability techniques in an attempt to help a user understand when an AI algorithm is likely to succeed as well as which cases may be problematic for automatic assessment, thus increasing the potential for rapid clinical translation. AI application to COVID‐19 has been marred with controversy recently. This review discusses several aspects of explainable and interpretable AI as it pertains to the evaluation of COVID‐19 disease and it can restore trust in AI application to this disease. This includes the identification of common tasks that are relevant to explainable medical imaging AI, an overview of several modern approaches for producing explainable output as appropriate for a given imaging scenario, a discussion of how to evaluate explainable AI, and recommendations for best practices in explainable/interpretable AI implementation. This review will allow developers of AI systems for COVID‐19 to quickly understand the basics of several explainable AI techniques and assist in the selection of an approach that is both appropriate and effective for a given scenario.

Exploring transparency: A comparative analysis of explainable artificial intelligence techniques in retinography images to support the diagnosis of glaucoma.

Artificial Intelligence (AI) techniques of deep learning have revolutionized the disease diagnosis with their outstanding image classification performance. In spite of the outstanding results, the widespread adoption of these techniques in clinical practice is still taking place at a moderate pace. One of the major hindrance is that a trained Deep Neural Networks (DNN) model provides a prediction, but questions about why and how that prediction was made remain unanswered. This linkage is of utmost importance for the regulated healthcare domain to increase the trust in the automated diagnosis system by the practitioners, patients and other stakeholders. The application of deep learning for medical imaging has to be interpreted with caution due to the health and safety concerns similar to blame attribution in the case of an accident involving autonomous cars. The consequences of both a false positive and false negative cases are far reaching for patients' welfare and cannot be ignored. This is exacerbated by the fact that the state-of-the-art deep learning algorithms comprise of complex interconnected structures, millions of parameters, and a ‘black box’ nature, offering little understanding of their inner working unlike the traditional machine learning algorithms. Explainable AI (XAI) techniques help to understand model predictions which help develop trust in the system, accelerate the disease diagnosis, and meet adherence to regulatory requirements.This survey provides a comprehensive review of the promising field of XAI for biomedical imaging diagnostics. We also provide a categorization of the XAI techniques, discuss the open challenges, and provide future directions for XAI which would be of interest to clinicians, regulators and model developers.

Gut microbiome research has made tremendous progress, especially with the integration of machine learning and artificial intelligence that can provide new insights from complex microbiome data and its impact on human health. The use of explainable artificial intelligence is becoming critical in medicine and adopting it in precision medicine-models leveraging gut microbiome data is appealing for providing more transparency and trustworthiness in clinical research. This scoping review evaluates the use of machine learning and explainable artificial intelligence techniques and identifies existing gaps in knowledge in this research area to suggest future research directions. Online databases (PubMed and Scopus) were searched to retrieve papers published between 2018-2024, and from which we selected 76 publications. Different clinical applications of machine learning and artificial intelligence techniques in gut microbiome studies were explored in the reviewed articles. We observed a high prevalence in the use of black box models in the field, with Random Forest being the most used algorithm. The explainability remains somewhat limited in the field, but it appears to be improving. Researchers showed interest in SHAP applications as an explainable technique. Finally, not enough attention was paid to the reproducibility of the research work published. This review highlights opportunities for advancing research on explainable artificial intelligence models in the field of microbiome, supporting future applications of microbiome-based precision medicine.

The aim of this retrospective study is to assess any association between abdominal CT findings and the radiological stage of COVID-19 pneumonia, pulmonary embolism and patient outcomes. We included 158 adult hospitalized COVID-19 patients between 1 March 2020 and 1 March 2021 who underwent 206 abdominal CTs. Two radiologists reviewed all CT images. Pathological findings were classified as acute or not. A subset of patients with inflammatory pathology in ACE2 organs (bowel, biliary tract, pancreas, urinary system) was identified. The radiological stage of COVID pneumonia, pulmonary embolism, overall days of hospitalization, ICU admission and outcome were registered. Univariate statistical analysis coupled with explainable artificial intelligence (AI) techniques were used to discover associations between variables. The most frequent acute findings were bowel abnormalities (n = 58), abdominal fluid (n = 42), hematomas (n = 28) and acute urologic conditions (n = 8). According to univariate statistical analysis, pneumonia stage > 2 was significantly associated with increased frequency of hematomas, active bleeding and fluid-filled colon. The presence of at least one hepatobiliary finding was associated with all the COVID-19 stages > 0. Free abdominal fluid, acute pathologies in ACE2 organs and fluid-filled colon were associated with ICU admission; free fluid also presented poor patient outcomes. Hematomas and active bleeding with at least a progressive stage of COVID pneumonia. The explainable AI techniques find no strong relationship between variables.

Alzheimer's disease (AD) is a progressive neurodegenerative disorder characterized by cognitive decline, posing a significant challenge for individuals and society. Early detection and treatment are essential for effective disease management. The objective of this research is to develop a novel and interpretable deep learning model for rapid and accurate Alzheimer's disease detection, incorporating Explainable Artificial Intelligence (XAI) techniques. The model aims to ensure generalizability through cross-validation and data augmentation, while enhancing interpretability and transparency by using Explainable Artificial Intelligence methods such as Grad-CAM, SHAP, and LIME, alongside an Enhanced Fuzzy C-Means (FCM) algorithm to clarify feature categorization and improve understanding of the model's decision-making process. The proposed model employs a multi-stage approach. Initially, MRI scans are transformed into feature vectors suitable for input into a Deep Convolutional Neural Network (CNN). Subsequently, an Enhanced Fuzzy C-Mean (FCM) algorithm, incorporating spatial information, refines these features to improve clustering precision. The model integrates Explainable Artificial Intelligence techniques, including Grad-CAM, SHAP, and LIME, to elucidate critical features and regions influencing classification outcomes. The performance metrics such as Accuracy, Recall and Specificity are used for assessing the performance of the model. The XAI-DEF Alzheimer's disease detection model consistently demonstrated exceptional performance across both the ADNI and OASIS datasets. On ADNI, the model achieved an accuracy of 99.39%, recall of 99.47%, and specificity of 99.3%. Similarly, on OASIS, the model attained an accuracy of 99.36%, recall of 99.53%, and specificity of 99.15%. These results underscore the model's effectiveness in accurately classifying Alzheimer's disease cases while minimizing false positives and negatives. Through the development of this model, we contribute to the advancement of dependable diagnostic tools tailored for the detection and management of Alzheimer's disease. By prioritizing interpretability alongside accuracy, our approach provides valuable insights into the decisionmaking process of the model, ultimately improving patient outcomes and facilitating further research in neurodegenerative disorders.

There is a growing trend of using artificial intelligence, particularly deep learning algorithms, in medical diagnostics, revolutionizing healthcare by improving efficiency, accuracy, and patient outcomes. However, the use of artificial intelligence in medical diagnostics comes with the critical need to explain the reasoning behind artificial intelligence‐based predictions and ensure transparency in decision‐making. Explainable artificial intelligence has emerged as a crucial research area to address the need for transparency and interpretability in medical diagnostics. Explainable artificial intelligence techniques aim to provide insights into the decision‐making process of artificial intelligence systems, enabling clinicians to understand the factors the algorithms consider in reaching their predictions. This paper presents a detailed review of saliency‐based (visual) methods, such as class activation methods, which have gained popularity in medical imaging as they provide visual explanations by highlighting the regions of an image most influential in the artificial intelligence's decision. We also present the literature on non‐visual methods, but the focus will be on visual methods. We also use the existing literature to experiment with infrared breast images for detecting breast cancer. Towards the end of this paper, we also propose an “attention guided Grad‐CAM” that enhances the visualizations for explainable artificial intelligence. The existing literature shows that explainable artificial intelligence techniques are not explored in the context of infrared medical images and opens up a wide range of opportunities for further research to make clinical thermography into assistive technology for the medical community.

Text classification plays an essential role in the medical domain by organizing and categorizing vast amounts of textual data through machine learning (ML) and deep learning (DL). The adoption of Artificial Intelligence (AI) technologies in healthcare has raised concerns about the interpretability of AI models, often perceived as "black boxes." Explainable AI (XAI) techniques aim to mitigate this issue by elucidating AI model decision-making process. In this paper, we present a scoping review exploring the application of different XAI techniques in medical text classification, identifying two main types: model-specific and model-agnostic methods. Despite some positive feedback from developers, formal evaluations with medical end users of these techniques remain limited. The review highlights the necessity for further research in XAI to enhance trust and transparency in AI-driven decision-making processes in healthcare.

Debilitating hearing loss (HL) affects ~6% of the human population. Only 20% of the people in need of a hearing assistive device will eventually seek and acquire one. The number of people that are satisfied with their Hearing Aids (HAids) and continue using them in the long term is even lower. Understanding the personal, behavioral, environmental, or other factors that correlate with the optimal HAid fitting and with users' experience of HAids is a significant step in improving patient satisfaction and quality of life, while reducing societal and financial burden. In SMART BEAR we are addressing this need by making use of the capacity of modern HAids to provide dynamic logging of their operation and by combining this information with a big amount of information about the medical, environmental, and social context of each HAid user. We are studying hearing rehabilitation through a 12-month continuous monitoring of HL patients, collecting data, such as participants' demographics, audiometric and medical data, their cognitive and mental status, their habits, and preferences, through a set of medical devices and wearables, as well as through face-to-face and remote clinical assessments and fitting/fine-tuning sessions. Descriptive, AI-based analysis and assessment of the relationships between heterogeneous data and HL-related parameters will help clinical researchers to better understand the overall health profiles of HL patients, and to identify patterns or relations that may be proven essential for future clinical trials. In addition, the future state and behavioral (e.g., HAids Satisfiability and HAids usage) of the patients will be predicted with time-dependent machine learning models to assist the clinical researchers to decide on the nature of the interventions. Explainable Artificial Intelligence (XAI) techniques will be leveraged to better understand the factors that play a significant role in the success of a hearing rehabilitation program, constructing patient profiles. This paper is a conceptual one aiming to describe the upcoming data collection process and proposed framework for providing a comprehensive profile for patients with HL in the context of EU-funded SMART BEAR project. Such patient profiles can be invaluable in HL treatment as they can help to identify the characteristics making patients more prone to drop out and stop using their HAids, using their HAids sufficiently long during the day, and being more satisfied by their HAids experience. They can also help decrease the number of needed remote sessions with their Audiologist for counseling, and/or HAids fine tuning, or the number of manual changes of HAids program (as indication of poor sound quality and bad adaptation of HAids configuration to patients' real needs and daily challenges), leading to reduced healthcare cost.