Interpretation Of Neural Networks Research Articles

Agglomerator++: Interpretable part-whole hierarchies and latent space representations in neural networks

Deep neural networks achieve outstanding results in a large variety of tasks, often outperforming human experts. However, a known limitation of current neural architectures is the poor accessibility in understanding and interpreting the network’s response to a given input. This is directly related to the huge number of variables and the associated non-linearities of neural models, which are often used as black boxes. This lack of transparency, particularly in crucial areas like autonomous driving, security, and healthcare, can trigger skepticism and limit trust, despite the networks’ high performance. In this work, we want to advance the interpretability in neural networks. We present Agglomerator++, a framework capable of providing a representation of part-whole hierarchies from visual cues and organizing the input distribution to match the conceptual-semantic hierarchical structure between classes. We evaluate our method on common datasets, such as SmallNORB, MNIST, FashionMNIST, CIFAR-10, and CIFAR-100, showing that our solution delivers a more interpretable model compared to other state-of-the-art approaches. Our code is available at https://mmlab-cv.github.io/Agglomeratorplusplus/.

A comprehensive review of deep neural network interpretation using topological data analysis

Deep neural networks have achieved significant success across various fields, but their intrinsic black-box nature hinders the further development. Addressing the interpretability challenges, topological data analysis has emerged as a promising tool to reveal these complex models. In this work, we present a review of the emerging field of interpreting deep neural networks using topological data analysis. We organize the existing body of work into distinct analytical categories, highlighting interpretations based on the topology of data, network structural characteristics, network functional characteristics, and techniques derived from Mapper. The objective of this paper is to extract the research pattern of this area, and point out the future research direction.

On the interpretability of quantum neural networks

Interpretability of artificial intelligence (AI) methods, particularly deep neural networks, is of great interest. This heightened focus stems from the widespread use of AI-backed systems. These systems, often relying on intricate neural architectures, can exhibit behavior that is challenging to explain and comprehend. The interpretability of such models is a crucial component of building trusted systems. Many methods exist to approach this problem, but they do not apply straightforwardly to the quantum setting. Here, we explore the interpretability of quantum neural networks using local model-agnostic interpretability measures commonly utilized for classical neural networks. Following this analysis, we generalize a classical technique called LIME, introducing Q-LIME, which produces explanations of quantum neural networks. A feature of our explanations is the delineation of the region in which data samples have been given a random label, likely subjects of inherently random quantum measurements. We view this as a step toward understanding how to build responsible and accountable quantum AI models.

Interpretability of deep neural networks: A review of methods, classification and hardware

Artificial intelligence, and especially deep neural networks, have evolved substantially in the recent years, infiltrating numerous domains of applications, often greatly impactful to society’s well-being. As a result, the need to understand how these models operate in depth and to access explanations of their decisions has become more vital than ever. Tending to this demand, the following paper aims to provide a thorough overview of the methods that have so far been developed to explain deep neural networks. Key aspects of explainability are defined and a straightforward classification of existing approaches is introduced, along with numerous examples. The task of realizing these methods on hardware is also discussed to complete the understanding of their application.

Decoupling Deep Learning for Enhanced Image Recognition Interpretability

The quest for enhancing the interpretability of neural networks has become a prominent focus in recent research endeavors. Prototype-based neural networks have emerged as a promising avenue for imbuing models with interpretability by gauging the similarity between image components and category prototypes to inform decision-making. However, these networks face challenges as they share similarity activations during both the inference and explanation processes, creating a trade-off between accuracy and interpretability. To address this issue and ensure that a network achieves high accuracy and robust interpretability in the classification process, this paper introduces a groundbreaking prototype-based neural network termed the “Decoupling Prototypical Network” (DProtoNet). This novel architecture comprises encoder, inference, and interpretation modules. In the encoder module, we introduce decoupling feature masks to facilitate the generation of feature vectors and prototypes, enhancing the generalization capabilities of the model. The inference module leverages these feature vectors and prototypes to make predictions based on similarity comparisons, thereby preserving an interpretable inference structure. Meanwhile, the interpretation module advances the field by presenting a novel approach: a “multiple dynamic masks decoder” that replaces conventional upsampling similarity activations. This decoder operates by perturbing images with mask vectors of varying sizes and learning saliency maps through consistent activation. This methodology offers a precise and innovative means of interpreting prototype-based networks. DProtoNet effectively separates the inference and explanation components within prototype-based networks. By eliminating the constraints imposed by shared similarity activations during the inference and explanation phases, our approach concurrently elevates accuracy and interpretability. Experimental evaluations on diverse public natural datasets, including CUB-200-2011, Stanford Cars, and medical datasets like RSNA and iChallenge-PM, corroborate the substantial enhancements achieved by our method compared to previous state-of-the-art approaches. Furthermore, ablation studies are conducted to provide additional evidence of the effectiveness of our proposed components.

A detailed analysis of the interpretability of Convolutional Neural Networks for text classification

Abstract Convolutional Neural Networks (CNNs) have become ubiquitous in many NLP tasks. However, understanding its process is still an area with much to be done. In this paper, we introduce a method to study the interpretability of CNNs when used for text classification. More specifically, we work on the interpretability of the convolutional filters in the context of sentiment analysis. The framework used in this paper has allowed the understanding of the mechanics of the network when applied to the task of sentiment analysis. The results of the experiments reveal how certain parts of speech (POS) tags are more relevant than others to the classification of a sentence. Furthermore, we also observed a preference for shorter $n$-grams when classifying negative sentiment sequences. Additionally, we detected a certain amount of redundancy among the convolutional filters, thus allowing us to conclude that smaller architectures would have worked for this particular task. This was achieved by computing the relevant metrics to measure the influence of the pertinent property on the desired class. In our case, the concepts treated were POS tags that carry semantic information and properties were related to the convolutional filters.

Fdstnet & TransCAM: A 1D CNN temperature interpretation model for temperature indication paint and model interpretability method

The color of temperature indicating paint (TIP) is commonly used in temperature measurement testing of aircraft engine hot end components. In this paper, an end-to-end convolutional neural network utilizing DSTNet, is proposed for accurate temperature interpretation of TIP. Furthermore, the DSTNet is optimized by fusion of reflectance spectrum and sub-band statistical features, resulting in the creation of FDSTNet. The FDSTNet achieves highly accurate temperature interpretation. When the temperature error is 0℃, the interpretation accuracy on KN3A, KN6 and KN8 samples is 98 %, 97 % and 94 %, respectively. When the temperature error is ±10℃, the accuracy on the engine blade is 88 %. In addition, a 1D convolutional neural network interpretability method based on TransCAM, is proposed. This approach analyzes the process of learning the reflectance spectrum of DSTNet, and localizes the characteristic wavelength by visualizing the class activation mapping curves of each convolutional layer of DSTNet.

Analysing the Performance and Interpretability of CNN-Based Architectures for Plant Nutrient Deficiency Identification

Early detection of plant nutrient deficiency is crucial for agricultural productivity. This study investigated the performance and interpretability of Convolutional Neural Networks (CNNs) for this task. Using the rice and banana datasets, we compared three CNN architectures (CNN, VGG-16, Inception-V3). Inception-V3 achieved the highest accuracy (93% for rice and banana), but simpler models such as VGG-16 might be easier to understand. To address this trade-off, we employed Explainable AI (XAI) techniques (SHAP and Grad-CAM) to gain insights into model decision-making. This study emphasises the importance of both accuracy and interpretability in agricultural AI and demonstrates the value of XAI for building trust in these models.

B-Cos Alignment for Inherently Interpretable CNNs and Vision Transformers.

We present a new direction for increasing the interpretability of deep neural networks (DNNs) by promoting weight-input alignment during training. For this, we propose to replace the linear transformations in DNNs by our novel B-cos transformation. As we show, a sequence (network) of such transformations induces a single linear transformation that faithfully summarises the full model computations. Moreover, the B-cos transformation is designed such that the weights align with relevant signals during optimisation. As a result, those induced linear transformations become highly interpretable and highlight task-relevant features. Importantly, the B-cos transformation is designed to be compatible with existing architectures and we show that it can easily be integrated into virtually all of the latest state of the art models for computer vision-e.g. ResNets, DenseNets, ConvNext models, as well as Vision Transformers-by combining the B-cos-based explanations with normalisation and attention layers, all whilst maintaining similar accuracy on ImageNet. Finally, we show that the resulting explanations are of high visual quality and perform well under quantitative interpretability metrics.

Enhancing prediction accuracy of Remaining Useful Life in lithium-ion batteries: A deep learning approach with Bat optimizer

Remaining Useful Life (RUL) prediction in lithium-ion batteries is crucial for assessing battery performance. Despite the popularity of deep learning methods for RUL prediction, their complex architectures often pose challenges in interpretation and resource consumption. We propose a novel approach that combines the interpretability of a convolutional neural network (CNN) with the efficiency of a bat-based optimizer. CNN extracts battery data features and characterizes degradation kinetics, while the optimizer refines CNN parameters. Tested on NASA PCoE data, our method achieves exceptional results with minimal computational burden and fewer parameters. It outperforms traditional approaches, yielding an R2-score of 0.9987120, an MAE of 0.004397067 Ah, and a low RMSE of 0.00656 Ah. The proposed model outperforms traditional deep learning models, as confirmed by comparative analysis.

Enhancing interpretability in neural networks for nuclear power plant fault diagnosis: A comprehensive analysis and improvement approach

Deep neural networks, as applied in the field of nuclear power fault diagnosis, have garnered significant attention alongside advancements in artificial intelligence technology. However, the “black box” nature of deep learning models has raised concerns regarding their deployment in scenarios demanding high safety standards, such as nuclear power plants. In this paper, we innovatively propose the utilization of an explainable artificial intelligence method, grounded in game theory, to conduct a detailed analysis of the diagnostic behavior of neural network models applied to nuclear power plants. By leveraging SHAP, the decision-making processes of these opaque models are demystified, offering insights into how and why they arrive at their predictions. In this study, two of the most widely utilized neural network frameworks are applied across six representative fault diagnosis cases for a comprehensive analysis. Based on its analysis results, a novel SHAP-Enhanced Feature Selection for Efficient Neural Network Fault Diagnosis strategy is proposed, significantly reducing model complexity without sacrificing diagnostic performance. This research breaks through the limitations of data-driven models used in the field of nuclear power fault diagnosis, conducts an interpretable analysis of their behavior, and proposes an improved strategy based on the analysis results, contributing to the practical application of data-driven models in nuclear power.

Local Interpretations for Explainable Natural Language Processing: A Survey

As the use of deep learning techniques has grown across various fields over the past decade, complaints about the opaqueness of the black-box models have increased, resulting in an increased focus on transparency in deep learning models. This work investigates various methods to improve the interpretability of deep neural networks for Natural Language Processing (NLP) tasks, including machine translation and sentiment analysis. We provide a comprehensive discussion on the definition of the term interpretability and its various aspects at the beginning of this work. The methods collected and summarised in this survey are only associated with local interpretation and are specifically divided into three categories: (1) interpreting the model’s predictions through related input features; (2) interpreting through natural language explanation; (3) probing the hidden states of models and word representations.

Artificial Intelligence, Lymphoid Neoplasms, and Prediction of MYC, BCL2, and BCL6 Gene Expression Using a Pan-Cancer Panel in Diffuse Large B-Cell Lymphoma

Background: Artificial intelligence in medicine is a field that is rapidly evolving. Machine learning and deep learning are used to improve disease identification and diagnosis, personalize disease treatment, analyze medical images, evaluate clinical trials, and speed drug development. Methods: First, relevant aspects of AI are revised in a comprehensive manner, including the classification of hematopoietic neoplasms, types of AI, applications in medicine and hematological neoplasia, generative pre-trained transformers (GPTs), and the architecture and interpretation of feedforward neural net-works (multilayer perceptron). Second, a series of 233 diffuse large B-cell lymphoma (DLBCL) patients treated with rituximab-CHOP from the Lymphoma/Leukemia Molecular Profiling Project (LLMPP) was analyzed. Results: Using conventional statistics, the high expression of MYC and BCL2 was associated with poor survival, but high BCL6 was associated with a favorable overall survival of the patients. Then, a neural network predicted MYC, BCL2, and BCL6 with high accuracy using a pan-cancer panel of 758 genes of immuno-oncology and translational research that includes clinically relevant actionable genes and pathways. A comparable analysis was performed using gene set enrichment analysis (GSEA). Conclusions: The mathematical way in which neural networks reach conclusions has been considered a black box, but a careful understanding and evaluation of the architectural design allows us to interpret the results logically. In diffuse large B-cell lymphoma, neural networks are a plausible data analysis approach.

IBPGNET: lung adenocarcinoma recurrence prediction based on neural network interpretability.

Lung adenocarcinoma (LUAD) is the most common histologic subtype of lung cancer. Early-stage patients have a 30-50% probability of metastatic recurrence after surgical treatment. Here, we propose a new computational framework, Interpretable Biological Pathway Graph Neural Networks (IBPGNET), based on pathway hierarchy relationships to predict LUAD recurrence and explore the internal regulatory mechanisms of LUAD. IBPGNET can integrate different omics data efficiently and provide global interpretability. In addition, our experimental results show that IBPGNET outperforms other classification methods in 5-fold cross-validation. IBPGNET identified PSMC1 and PSMD11 as genes associated with LUAD recurrence, and their expression levels were significantly higher in LUAD cells than in normal cells. The knockdown of PSMC1 and PSMD11 in LUAD cells increased their sensitivity to afatinib and decreased cell migration, invasion and proliferation. In addition, the cells showed significantly lower EGFR expression, indicating that PSMC1 and PSMD11 may mediate therapeutic sensitivity through EGFR expression.

Adaptive coded phase mask design and high-quality image reconstruction for interference-less coded aperture correlation holography.

The interference-less coded aperture correlation holography is a non-scanning, motionless, and incoherent technique for imaging three-dimensional objects without two-wave interference. Nevertheless, a challenge lies in that the coded phase mask encodes the system noise, while traditional reconstruction algorithms often introduce unwanted surplus background components during reconstruction. A deep learning-based method is proposed to mitigate system noise and background components simultaneously. Specifically, this method involves two sub-networks: a coded phase mask design sub-network and an image reconstruction sub-network. The former leverages the object's frequency distribution to generate an adaptive coded phase mask that encodes the object wave-front precisely without being affected by the superfluous system noise. The latter establishes a mapping between the autocorrelations of the hologram and the object, effectively suppresses the background components by embedding a prior physical knowledge and improves the neural network's adaptability and interpretability. Experimental results demonstrate the effectiveness of the proposed method in suppressing system noise and background components, thereby significantly improving the signal-to-noise ratio of the reconstructed images.

Convolutional Neural Network with Attention Mechanism and Visual Vibration Signal Analysis for Bearing Fault Diagnosis.

Bearings, as widely employed supporting components, frequently work in challenging working conditions, leading to diverse fault types. Traditional methods for diagnosing bearing faults primarily center on time-frequency analysis, but this often requires expert experience for accurate fault identification. Conversely, intelligent fault recognition and classification methods frequently lack interpretability. To address this challenge, this paper introduces a convolutional neural network with an attention mechanism method, denoted as CBAM-CNN, for bearing fault diagnosis. This approach incorporates an attention mechanism, creating a Convolutional Block Attention Module (CBAM), to enhance the fault feature extraction capability of the network in the time-frequency domain. In addition, the proposed method integrates a weight visualization module known as the Gradient-Weighted Class Activation Map (Grad-CAM), enhancing the interpretability of the convolutional neural network by generating visual heatmaps on fault time-frequency graphs. The experimental results demonstrate that utilizing the dataset employed in this study, the CBAM-CNN achieves an accuracy of 99.81%, outperforming the Base-CNN with enhanced convergence speed. Furthermore, the analysis of attention weights reveals that this method exhibits distinct focus of attention under various fault types and degrees. The interpretability experiments indicate that the CBAM module balances the weight allocation, emphasizing signal frequency distribution rather than amplitude distribution. Consequently, this mitigates the impact of the signal amplitude on the diagnostic model to some extent.

Leveraging saliency priors and explanations for enhanced consistent interpretability

Deep neural networks have emerged as highly effective tools for computer vision systems, showcasing remarkable performance. However, the intrinsic opacity, potential biases, and vulnerability to shortcut learning in these models pose significant concerns regarding their practical application. To tackle these issues, this work employs saliency prior and explanations to enhance the credibility, reliability, and interpretability of neural networks. Specifically, we employ a salient object detection algorithm to extract human-consistent priors from images for data augmentation. The identified saliency priors, along with explanations, serve as supervision signals directing the network’s focus to salient regions within the image. Additionally, contrastive self-supervised learning is incorporated to enable the model to discern the most discriminative concepts. Experimental results confirm the algorithm’s capability to align model explanations with human priors, thereby improving interpretability. Moreover, the proposed approach enhances model performance in data-limited and fine-grained classification scenarios. Importantly, our algorithm is label-independent, allowing for the integration of unlabeled data during training. In practice, this method contributes to improving the reliability and interpretability of intelligent models for downstream tasks. Our code is available here: https://github.com/DLAIResearch/SGC.

Use of Autoencoders for Improving the Performance of Classification and Interpretation of Convolutional Neural Networks in Histopathological Images

Breast cancer is one of the most common types of cancer and it presents itself as being the leading cause of death among women. If its diagnosis occur soon enough, the probability to achieve the cure of the patient can be increased. Recently, it has been more common the use of deep neural network techniques to aid pathologists in their prognosis, but they still do not fully trust them because they lack interpretability. In light of that, this work investigates if previous training of the models as encoders could enhance their accuracy in both classification and interpretability. There were implemented three models to the BreakHis and BreCaHAD dataset: NASNet Mobile, DenseNET201, and MobileNetV2. The experiments have shown that the three models increased the classification performance and two models improved their interpretability using the proposed strategy. DenseNet201 encoder has performed almost 23% better than its vanilla version in classifying a tumor and the NASNet Mobile encoder has improved 28.5% in its tumor interpretation.

Research on image classification leveraging deep convolutional neural networks and visual cognition

The field of image classification has experienced remarkable improvements with the advent of deep learning techniques, especially Deep Convolutional Neural Networks. The present study provides an extensive exploration of the junction where image classification based on Deep Convolutional Neural Networks meets human visual cognition. Utilizing the inherent ability of these networks to automatically learn hierarchical features from raw pixel data, this research examines their potential in classifying images from diverse complex datasets, emphasizing predominantly on the extensively utilized ImageNet dataset. The initial aspect of this study involves training and evaluating models based on Deep Convolutional Neural Networks on the ImageNet dataset, which comprises millions of labeled images spanning across thousands of categories. Well-established network architectures such as AlexNet, VGGNet, GoogLeNet, and ResNet are employed, and their performance in the challenging task of image classification is assessed. Rigorous experiments highlight the strengths and weaknesses of each model while emphasizing the prospects of transfer learning and fine-tuning. Following this, the interpretability of Deep Convolutional Neural Networks is explored by using visualization techniques to comprehend the learned feature representations. By visualizing activation maps and class-specific saliency maps, valuable insights are gained into the regions of interest that guide the decision-making of these models. Moreover, the correlation between the features extracted by these models and human visual attention mechanisms is examined to shed light on the focus of attention of the models. The study also addresses the difficulties that adversarial attacks, data bias, and generalization capabilities present to Deep Convolutional Neural Networks. Strategies to enhance the robustness and adaptability of the models across various domains are examined, linking these observations to human cognitive behavior.

Interpretability of Neural Networks Based on Game-theoretic Interactions

Interpretability of Neural Networks Based on Game-theoretic Interactions