Interpretation Of Neural Networks Research Articles

Evidential Deep Learning: Enhancing Predictive Uncertainty Estimation for Earth System Science Applications

Abstract Robust quantification of predictive uncertainty is a critical addition needed for machine learning applied to weather and climate problems to improve understanding of what is driving prediction sensitivity. Ensembles of machine learning models provide predictive uncertainty estimates in a conceptually simple way but require multiple models for training and prediction, increasing computational cost and latency. Parametric deep learning can estimate uncertainty with one model by predicting the parameters of a probability distribution but does not account for epistemic uncertainty. Evidential deep learning, a technique that extends parametric deep learning to higher-order distributions, can account for both aleatoric and epistemic uncertainty with one model. This study compares the uncertainty derived from evidential neural networks to that obtained from ensembles. Through applications of classification of winter precipitation type and regression of surface layer fluxes, we show evidential deep learning models attaining predictive accuracy rivaling standard methods, while robustly quantifying both sources of uncertainty. We evaluate the uncertainty in terms of how well the predictions are calibrated and how well the uncertainty correlates with prediction error. Analyses of uncertainty in the context of the inputs reveal sensitivities to underlying meteorological processes, facilitating interpretation of the models. The conceptual simplicity, interpretability, and computational efficiency of evidential neural networks make them highly extensible, offering a promising approach for reliable and practical uncertainty quantification in Earth system science modeling. In order to encourage broader adoption of evidential deep learning, we have developed a new Python package, MILES-GUESS (https://github.com/ai2es/miles-guess), that enables users to train and evaluate both evidential and ensemble deep learning.

Mapping Data to Concepts: Enhancing Quantum Neural Network Transparency with Concept-Driven Quantum Neural Networks

We introduce the concept-driven quantum neural network (CD-QNN), an innovative architecture designed to enhance the interpretability of quantum neural networks (QNNs). CD-QNN merges the representational capabilities of QNNs with the transparency of self-explanatory models by mapping input data into a human-understandable concept space and making decisions based on these concepts. The algorithmic design of CD-QNN is comprehensively analyzed, detailing the roles of the concept generator, feature extractor, and feature integrator in improving and balancing model expressivity and interpretability. Experimental results demonstrate that CD-QNN maintains high predictive accuracy while offering clear and meaningful explanations of its decision-making process. This paradigm shift in QNN design underscores the growing importance of interpretability in quantum artificial intelligence, positioning CD-QNN and its derivative technologies as pivotal in advancing reliable and interpretable quantum intelligent systems for future research and applications.

Decoupling Deep Learning for Enhanced Image Recognition Interpretability

The quest for enhancing the interpretability of neural networks has become a prominent focus in recent research endeavors. Prototype-based neural networks have emerged as a promising avenue for imbuing models with interpretability by gauging the similarity between image components and category prototypes to inform decision-making. However, these networks face challenges as they share similarity activations during both the inference and explanation processes, creating a tradeoff between accuracy and interpretability. To address this issue and ensure that a network achieves high accuracy and robust interpretability in the classification process, this article introduces a groundbreaking prototype-based neural network termed the “Decoupling Prototypical Network” (DProtoNet). This novel architecture comprises encoder, inference, and interpretation modules. In the encoder module, we introduce decoupling feature masks to facilitate the generation of feature vectors and prototypes, enhancing the generalization capabilities of the model. The inference module leverages these feature vectors and prototypes to make predictions based on similarity comparisons, thereby preserving an interpretable inference structure. Meanwhile, the interpretation module advances the field by presenting a novel approach: a “multiple dynamic masks decoder” that replaces conventional upsampling similarity activations. This decoder operates by perturbing images with mask vectors of varying sizes and learning saliency maps through consistent activation. This methodology offers a precise and innovative means of interpreting prototype-based networks. DProtoNet effectively separates the inference and explanation components within prototype-based networks. By eliminating the constraints imposed by shared similarity activations during the inference and explanation phases, our approach concurrently elevates accuracy and interpretability. Experimental evaluations on diverse public natural datasets, including CUB-200-2011, Stanford Cars, and medical datasets like RSNA and iChallenge-PM, corroborate the substantial enhancements achieved by our method compared to previous state-of-the-art approaches. Furthermore, ablation studies are conducted to provide additional evidence of the effectiveness of our proposed components.

Massive experimental quantification of amyloid nucleation allows interpretable deep learning of protein aggregation.

Protein aggregation is a pathological hallmark of more than fifty human diseases and a major problem for biotechnology. Methods have been proposed to predict aggregation from sequence, but these have been trained and evaluated on small and biased experimental datasets. Here we directly address this data shortage by experimentally quantifying the amyloid nucleation of >100,000 protein sequences. This unprecedented dataset reveals the limited performance of existing computational methods and allows us to train CANYA, a convolution-attention hybrid neural network that accurately predicts amyloid nucleation from sequence. We adapt genomic neural network interpretability analyses to reveal CANYA's decision-making process and learned grammar. Our results illustrate the power of massive experimental analysis of random sequence-spaces and provide an interpretable and robust neural network model to predict amyloid nucleation.

Deep Learning for Stylometry and Authorship Attribution: a Review of Literature

The application of deep learning techniques to stylometry and authorship attribution has emerged as a promising frontier in computational linguistics, offering new possibilities for understanding literary style and authorship in both historical and contemporary contexts. This review paper synthesizes recent advances in the use of deep learning models, such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and transformer architectures, for identifying and attributing authorship based on stylistic analysis. We examine the effectiveness of these models in comparison to traditional statistical methods, highlighting their ability to capture complex linguistic patterns and nuances that are often overlooked by conventional approaches. Furthermore, we explore how deep learning models handle challenges such as multilingual texts, limited data, and variations across genres and periods. This review also addresses the interpretability of neural networks in the context of stylometry and discusses the implications of these methods for fields ranging from literary studies to digital forensics. By providing a comprehensive overview of the current state of research, this paper identifies key trends, challenges, and future directions for the application of deep learning to stylometry and authorship attribution.

НЕЙРОМЕРЕЖЕВІ МАТЕМАТИЧНІ МОДЕЛІ ЛЮДИНОМАШИННОГО СПІЛКУВАННЯ ТА РОЗПІЗНАВАННЯ

The study is devoted to the neural network interpretation of the task of human-machine communication and recognition by multiple criteria, considered as a task of assignment. The main goal is to reduce this problem to a standard form, where the number of criteria groups is equal to the number of ranked documents. The study defined the architecture of a neural network and proposed the use of a network of binary neurons, which is a matrix of a certain dimension. The proposed ranking model is based on a neural network that contains arbitrary feedback. This allows the excitation to be transmitted back to the neuron, which contributes to the repeated performance of its function. However, in dynamic neural networks instability occurs, which is manifested in a random change in the states of neurons without reaching stationary states. The question of stability of the dynamics of such systems remains open. The considered discrete Hopfield neural network has the following characteristics: one layer of elements, each element is connected to all others, but not to itself; only one element is updated per stage; elements are updated in random order, but each is updated with the same frequency; the output function is binary (value "0" or "1"). A Hopfield neural network is recurrent: the output of the network is reused as input until a steady state is reached. After starting, the neural network changes its state, gradually moving to a stable mode, which allows identifying a plan for evaluating the process of human-machine communication according to a set of criteria. Random search procedures are used to refine the results. The proposed energy function is minimized to ensure that the constraints are met and the problem is solved. The constructed function reaches a minimum only in the states corresponding to the assignment plans. The definition of the network parameters is carried out by comparing the obtained functions with the energy function in general form. The practical implementation of the model demonstrated that Hopfield's neural network can be successfully applied to document ranking in human-machine communication and recognition systems, providing high accuracy and efficiency in solving ranking problems.

Agglomerator++: Interpretable part-whole hierarchies and latent space representations in neural networks

Deep neural networks achieve outstanding results in a large variety of tasks, often outperforming human experts. However, a known limitation of current neural architectures is the poor accessibility in understanding and interpreting the network’s response to a given input. This is directly related to the huge number of variables and the associated non-linearities of neural models, which are often used as black boxes. This lack of transparency, particularly in crucial areas like autonomous driving, security, and healthcare, can trigger skepticism and limit trust, despite the networks’ high performance. In this work, we want to advance the interpretability in neural networks. We present Agglomerator++, a framework capable of providing a representation of part-whole hierarchies from visual cues and organizing the input distribution to match the conceptual-semantic hierarchical structure between classes. We evaluate our method on common datasets, such as SmallNORB, MNIST, FashionMNIST, CIFAR-10, and CIFAR-100, showing that our solution delivers a more interpretable model compared to other state-of-the-art approaches. Our code is available at https://mmlab-cv.github.io/Agglomeratorplusplus/.

On feature extraction using distances from reference points

Abstract Feature extraction is the key to a successfully trained classifier. Although many automatic methods exist for traditional data, other data types (e.g., sequences, graphs) usually require dedicated approaches. In this paper, we study a universal feature extraction method based on distance from reference points. First, we formalize this process and provide an instantiation based on network centrality. To reliably select the best reference points, we introduce the notion of θ-neighborhood which allows us to navigate the topography of fully connected graphs. Our experiments show that the proposed peak selection method is significantly better than a traditional top-k approach for centrality-based reference points and that the quality of the reference points is much less important than their quantity. Finally, we provide an alternative, neural network interpretation of reference points, which paves a path to optimization-based selection methods, together with a new type of neuron, called the Euclidean neuron, and the necessary modifications to backpropagation.

A comprehensive review of deep neural network interpretation using topological data analysis

Deep neural networks have achieved significant success across various fields, but their intrinsic black-box nature hinders the further development. Addressing the interpretability challenges, topological data analysis has emerged as a promising tool to reveal these complex models. In this work, we present a review of the emerging field of interpreting deep neural networks using topological data analysis. We organize the existing body of work into distinct analytical categories, highlighting interpretations based on the topology of data, network structural characteristics, network functional characteristics, and techniques derived from Mapper. The objective of this paper is to extract the research pattern of this area, and point out the future research direction.

On the interpretability of quantum neural networks

Interpretability of artificial intelligence (AI) methods, particularly deep neural networks, is of great interest. This heightened focus stems from the widespread use of AI-backed systems. These systems, often relying on intricate neural architectures, can exhibit behavior that is challenging to explain and comprehend. The interpretability of such models is a crucial component of building trusted systems. Many methods exist to approach this problem, but they do not apply straightforwardly to the quantum setting. Here, we explore the interpretability of quantum neural networks using local model-agnostic interpretability measures commonly utilized for classical neural networks. Following this analysis, we generalize a classical technique called LIME, introducing Q-LIME, which produces explanations of quantum neural networks. A feature of our explanations is the delineation of the region in which data samples have been given a random label, likely subjects of inherently random quantum measurements. We view this as a step toward understanding how to build responsible and accountable quantum AI models.

Propagating Transparency: A Deep Dive into the Interpretability of Neural Networks

In the rapidly evolving landscape of deep learning (DL), understanding the inner workings of neural networks remains a significant challenge. This need for transparency and accountability from DL models assumes particular importance as DL models become increasingly prevalent in decision-making processes. Interpreting these models is key to addressing this challenge. This paper offers a comprehensive overview of interpretable deep learning methods. It emphasizes gradient-based propagation techniques that shed light on the complex mechanisms driving neural network predictions. Through a systematic review, we categorize gradient-based interpretability approaches, delve into the theory of notable methods, and compare their strengths and weaknesses. Furthermore, we investigate various evaluation metrics for interpretable systems, often generalized under the term eXplainable Artificial Intelligence (XAI). We highlight their significance in assessing the faithfulness, robustness, localization, complexity, randomization, and adherence to the axiomatic principles of XAI methods. We aim to help researchers and practitioners work towards a more transparent future for artificial intelligence by providing an overview of the most recent developments in the field.

Unifying Adversarial Robustness and Interpretability in Deep Neural Networks: A Comprehensive Framework for Explainable and Secure Machine Learning Models

Unifying Adversarial Robustness and Interpretability in Deep Neural Networks: A Comprehensive Framework for Explainable and Secure Machine Learning Models

Interpretability of deep neural networks: A review of methods, classification and hardware

Artificial intelligence, and especially deep neural networks, have evolved substantially in the recent years, infiltrating numerous domains of applications, often greatly impactful to society’s well-being. As a result, the need to understand how these models operate in depth and to access explanations of their decisions has become more vital than ever. Tending to this demand, the following paper aims to provide a thorough overview of the methods that have so far been developed to explain deep neural networks. Key aspects of explainability are defined and a straightforward classification of existing approaches is introduced, along with numerous examples. The task of realizing these methods on hardware is also discussed to complete the understanding of their application.

A detailed analysis of the interpretability of Convolutional Neural Networks for text classification

Abstract Convolutional Neural Networks (CNNs) have become ubiquitous in many NLP tasks. However, understanding its process is still an area with much to be done. In this paper, we introduce a method to study the interpretability of CNNs when used for text classification. More specifically, we work on the interpretability of the convolutional filters in the context of sentiment analysis. The framework used in this paper has allowed the understanding of the mechanics of the network when applied to the task of sentiment analysis. The results of the experiments reveal how certain parts of speech (POS) tags are more relevant than others to the classification of a sentence. Furthermore, we also observed a preference for shorter $n$-grams when classifying negative sentiment sequences. Additionally, we detected a certain amount of redundancy among the convolutional filters, thus allowing us to conclude that smaller architectures would have worked for this particular task. This was achieved by computing the relevant metrics to measure the influence of the pertinent property on the desired class. In our case, the concepts treated were POS tags that carry semantic information and properties were related to the convolutional filters.

Fdstnet & TransCAM: A 1D CNN temperature interpretation model for temperature indication paint and model interpretability method

The color of temperature indicating paint (TIP) is commonly used in temperature measurement testing of aircraft engine hot end components. In this paper, an end-to-end convolutional neural network utilizing DSTNet, is proposed for accurate temperature interpretation of TIP. Furthermore, the DSTNet is optimized by fusion of reflectance spectrum and sub-band statistical features, resulting in the creation of FDSTNet. The FDSTNet achieves highly accurate temperature interpretation. When the temperature error is 0℃, the interpretation accuracy on KN3A, KN6 and KN8 samples is 98 %, 97 % and 94 %, respectively. When the temperature error is ±10℃, the accuracy on the engine blade is 88 %. In addition, a 1D convolutional neural network interpretability method based on TransCAM, is proposed. This approach analyzes the process of learning the reflectance spectrum of DSTNet, and localizes the characteristic wavelength by visualizing the class activation mapping curves of each convolutional layer of DSTNet.

Analysing the Performance and Interpretability of CNN-Based Architectures for Plant Nutrient Deficiency Identification

Early detection of plant nutrient deficiency is crucial for agricultural productivity. This study investigated the performance and interpretability of Convolutional Neural Networks (CNNs) for this task. Using the rice and banana datasets, we compared three CNN architectures (CNN, VGG-16, Inception-V3). Inception-V3 achieved the highest accuracy (93% for rice and banana), but simpler models such as VGG-16 might be easier to understand. To address this trade-off, we employed Explainable AI (XAI) techniques (SHAP and Grad-CAM) to gain insights into model decision-making. This study emphasises the importance of both accuracy and interpretability in agricultural AI and demonstrates the value of XAI for building trust in these models.

B-Cos Alignment for Inherently Interpretable CNNs and Vision Transformers.

We present a new direction for increasing the interpretability of deep neural networks (DNNs) by promoting weight-input alignment during training. For this, we propose to replace the linear transformations in DNNs by our novel B-cos transformation. As we show, a sequence (network) of such transformations induces a single linear transformation that faithfully summarises the full model computations. Moreover, the B-cos transformation is designed such that the weights align with relevant signals during optimisation. As a result, those induced linear transformations become highly interpretable and highlight task-relevant features. Importantly, the B-cos transformation is designed to be compatible with existing architectures and we show that it can easily be integrated into virtually all of the latest state of the art models for computer vision-e.g. ResNets, DenseNets, ConvNext models, as well as Vision Transformers-by combining the B-cos-based explanations with normalisation and attention layers, all whilst maintaining similar accuracy on ImageNet. Finally, we show that the resulting explanations are of high visual quality and perform well under quantitative interpretability metrics.

Enhancing prediction accuracy of Remaining Useful Life in lithium-ion batteries: A deep learning approach with Bat optimizer

Remaining Useful Life (RUL) prediction in lithium-ion batteries is crucial for assessing battery performance. Despite the popularity of deep learning methods for RUL prediction, their complex architectures often pose challenges in interpretation and resource consumption. We propose a novel approach that combines the interpretability of a convolutional neural network (CNN) with the efficiency of a bat-based optimizer. CNN extracts battery data features and characterizes degradation kinetics, while the optimizer refines CNN parameters. Tested on NASA PCoE data, our method achieves exceptional results with minimal computational burden and fewer parameters. It outperforms traditional approaches, yielding an R2-score of 0.9987120, an MAE of 0.004397067 Ah, and a low RMSE of 0.00656 Ah. The proposed model outperforms traditional deep learning models, as confirmed by comparative analysis.

Enhancing interpretability in neural networks for nuclear power plant fault diagnosis: A comprehensive analysis and improvement approach

Deep neural networks, as applied in the field of nuclear power fault diagnosis, have garnered significant attention alongside advancements in artificial intelligence technology. However, the “black box” nature of deep learning models has raised concerns regarding their deployment in scenarios demanding high safety standards, such as nuclear power plants. In this paper, we innovatively propose the utilization of an explainable artificial intelligence method, grounded in game theory, to conduct a detailed analysis of the diagnostic behavior of neural network models applied to nuclear power plants. By leveraging SHAP, the decision-making processes of these opaque models are demystified, offering insights into how and why they arrive at their predictions. In this study, two of the most widely utilized neural network frameworks are applied across six representative fault diagnosis cases for a comprehensive analysis. Based on its analysis results, a novel SHAP-Enhanced Feature Selection for Efficient Neural Network Fault Diagnosis strategy is proposed, significantly reducing model complexity without sacrificing diagnostic performance. This research breaks through the limitations of data-driven models used in the field of nuclear power fault diagnosis, conducts an interpretable analysis of their behavior, and proposes an improved strategy based on the analysis results, contributing to the practical application of data-driven models in nuclear power.

Local Interpretations for Explainable Natural Language Processing: A Survey

As the use of deep learning techniques has grown across various fields over the past decade, complaints about the opaqueness of the black-box models have increased, resulting in an increased focus on transparency in deep learning models. This work investigates various methods to improve the interpretability of deep neural networks for Natural Language Processing (NLP) tasks, including machine translation and sentiment analysis. We provide a comprehensive discussion on the definition of the term interpretability and its various aspects at the beginning of this work. The methods collected and summarised in this survey are only associated with local interpretation and are specifically divided into three categories: (1) interpreting the model’s predictions through related input features; (2) interpreting through natural language explanation; (3) probing the hidden states of models and word representations.