Interpretation Of Neural Networks Research Articles

B-Cos Alignment for Inherently Interpretable CNNs and Vision Transformers.

We present a new direction for increasing the interpretability of deep neural networks (DNNs) by promoting weight-input alignment during training. For this, we propose to replace the linear transformations in DNNs by our novel B-cos transformation. As we show, a sequence (network) of such transformations induces a single linear transformation that faithfully summarises the full model computations. Moreover, the B-cos transformation is designed such that the weights align with relevant signals during optimisation. As a result, those induced linear transformations become highly interpretable and highlight task-relevant features. Importantly, the B-cos transformation is designed to be compatible with existing architectures and we show that it can easily be integrated into virtually all of the latest state of the art models for computer vision-e.g. ResNets, DenseNets, ConvNext models, as well as Vision Transformers-by combining the B-cos-based explanations with normalisation and attention layers, all whilst maintaining similar accuracy on ImageNet. Finally, we show that the resulting explanations are of high visual quality and perform well under quantitative interpretability metrics.

Enhancing interpretability in neural networks for nuclear power plant fault diagnosis: A comprehensive analysis and improvement approach

Deep neural networks, as applied in the field of nuclear power fault diagnosis, have garnered significant attention alongside advancements in artificial intelligence technology. However, the “black box” nature of deep learning models has raised concerns regarding their deployment in scenarios demanding high safety standards, such as nuclear power plants. In this paper, we innovatively propose the utilization of an explainable artificial intelligence method, grounded in game theory, to conduct a detailed analysis of the diagnostic behavior of neural network models applied to nuclear power plants. By leveraging SHAP, the decision-making processes of these opaque models are demystified, offering insights into how and why they arrive at their predictions. In this study, two of the most widely utilized neural network frameworks are applied across six representative fault diagnosis cases for a comprehensive analysis. Based on its analysis results, a novel SHAP-Enhanced Feature Selection for Efficient Neural Network Fault Diagnosis strategy is proposed, significantly reducing model complexity without sacrificing diagnostic performance. This research breaks through the limitations of data-driven models used in the field of nuclear power fault diagnosis, conducts an interpretable analysis of their behavior, and proposes an improved strategy based on the analysis results, contributing to the practical application of data-driven models in nuclear power.

Local Interpretations for Explainable Natural Language Processing: A Survey

As the use of deep learning techniques has grown across various fields over the past decade, complaints about the opaqueness of the black-box models have increased, resulting in an increased focus on transparency in deep learning models. This work investigates various methods to improve the interpretability of deep neural networks for Natural Language Processing (NLP) tasks, including machine translation and sentiment analysis. We provide a comprehensive discussion on the definition of the term interpretability and its various aspects at the beginning of this work. The methods collected and summarised in this survey are only associated with local interpretation and are specifically divided into three categories: (1) interpreting the model’s predictions through related input features; (2) interpreting through natural language explanation; (3) probing the hidden states of models and word representations.

Artificial Intelligence, Lymphoid Neoplasms, and Prediction of MYC, BCL2, and BCL6 Gene Expression Using a Pan-Cancer Panel in Diffuse Large B-Cell Lymphoma

Background: Artificial intelligence in medicine is a field that is rapidly evolving. Machine learning and deep learning are used to improve disease identification and diagnosis, personalize disease treatment, analyze medical images, evaluate clinical trials, and speed drug development. Methods: First, relevant aspects of AI are revised in a comprehensive manner, including the classification of hematopoietic neoplasms, types of AI, applications in medicine and hematological neoplasia, generative pre-trained transformers (GPTs), and the architecture and interpretation of feedforward neural net-works (multilayer perceptron). Second, a series of 233 diffuse large B-cell lymphoma (DLBCL) patients treated with rituximab-CHOP from the Lymphoma/Leukemia Molecular Profiling Project (LLMPP) was analyzed. Results: Using conventional statistics, the high expression of MYC and BCL2 was associated with poor survival, but high BCL6 was associated with a favorable overall survival of the patients. Then, a neural network predicted MYC, BCL2, and BCL6 with high accuracy using a pan-cancer panel of 758 genes of immuno-oncology and translational research that includes clinically relevant actionable genes and pathways. A comparable analysis was performed using gene set enrichment analysis (GSEA). Conclusions: The mathematical way in which neural networks reach conclusions has been considered a black box, but a careful understanding and evaluation of the architectural design allows us to interpret the results logically. In diffuse large B-cell lymphoma, neural networks are a plausible data analysis approach.

IBPGNET: lung adenocarcinoma recurrence prediction based on neural network interpretability.

Lung adenocarcinoma (LUAD) is the most common histologic subtype of lung cancer. Early-stage patients have a 30-50% probability of metastatic recurrence after surgical treatment. Here, we propose a new computational framework, Interpretable Biological Pathway Graph Neural Networks (IBPGNET), based on pathway hierarchy relationships to predict LUAD recurrence and explore the internal regulatory mechanisms of LUAD. IBPGNET can integrate different omics data efficiently and provide global interpretability. In addition, our experimental results show that IBPGNET outperforms other classification methods in 5-fold cross-validation. IBPGNET identified PSMC1 and PSMD11 as genes associated with LUAD recurrence, and their expression levels were significantly higher in LUAD cells than in normal cells. The knockdown of PSMC1 and PSMD11 in LUAD cells increased their sensitivity to afatinib and decreased cell migration, invasion and proliferation. In addition, the cells showed significantly lower EGFR expression, indicating that PSMC1 and PSMD11 may mediate therapeutic sensitivity through EGFR expression.

Adaptive coded phase mask design and high-quality image reconstruction for interference-less coded aperture correlation holography.

The interference-less coded aperture correlation holography is a non-scanning, motionless, and incoherent technique for imaging three-dimensional objects without two-wave interference. Nevertheless, a challenge lies in that the coded phase mask encodes the system noise, while traditional reconstruction algorithms often introduce unwanted surplus background components during reconstruction. A deep learning-based method is proposed to mitigate system noise and background components simultaneously. Specifically, this method involves two sub-networks: a coded phase mask design sub-network and an image reconstruction sub-network. The former leverages the object's frequency distribution to generate an adaptive coded phase mask that encodes the object wave-front precisely without being affected by the superfluous system noise. The latter establishes a mapping between the autocorrelations of the hologram and the object, effectively suppresses the background components by embedding a prior physical knowledge and improves the neural network's adaptability and interpretability. Experimental results demonstrate the effectiveness of the proposed method in suppressing system noise and background components, thereby significantly improving the signal-to-noise ratio of the reconstructed images.

Convolutional Neural Network with Attention Mechanism and Visual Vibration Signal Analysis for Bearing Fault Diagnosis.

Bearings, as widely employed supporting components, frequently work in challenging working conditions, leading to diverse fault types. Traditional methods for diagnosing bearing faults primarily center on time-frequency analysis, but this often requires expert experience for accurate fault identification. Conversely, intelligent fault recognition and classification methods frequently lack interpretability. To address this challenge, this paper introduces a convolutional neural network with an attention mechanism method, denoted as CBAM-CNN, for bearing fault diagnosis. This approach incorporates an attention mechanism, creating a Convolutional Block Attention Module (CBAM), to enhance the fault feature extraction capability of the network in the time-frequency domain. In addition, the proposed method integrates a weight visualization module known as the Gradient-Weighted Class Activation Map (Grad-CAM), enhancing the interpretability of the convolutional neural network by generating visual heatmaps on fault time-frequency graphs. The experimental results demonstrate that utilizing the dataset employed in this study, the CBAM-CNN achieves an accuracy of 99.81%, outperforming the Base-CNN with enhanced convergence speed. Furthermore, the analysis of attention weights reveals that this method exhibits distinct focus of attention under various fault types and degrees. The interpretability experiments indicate that the CBAM module balances the weight allocation, emphasizing signal frequency distribution rather than amplitude distribution. Consequently, this mitigates the impact of the signal amplitude on the diagnostic model to some extent.

Leveraging saliency priors and explanations for enhanced consistent interpretability

Deep neural networks have emerged as highly effective tools for computer vision systems, showcasing remarkable performance. However, the intrinsic opacity, potential biases, and vulnerability to shortcut learning in these models pose significant concerns regarding their practical application. To tackle these issues, this work employs saliency prior and explanations to enhance the credibility, reliability, and interpretability of neural networks. Specifically, we employ a salient object detection algorithm to extract human-consistent priors from images for data augmentation. The identified saliency priors, along with explanations, serve as supervision signals directing the network’s focus to salient regions within the image. Additionally, contrastive self-supervised learning is incorporated to enable the model to discern the most discriminative concepts. Experimental results confirm the algorithm’s capability to align model explanations with human priors, thereby improving interpretability. Moreover, the proposed approach enhances model performance in data-limited and fine-grained classification scenarios. Importantly, our algorithm is label-independent, allowing for the integration of unlabeled data during training. In practice, this method contributes to improving the reliability and interpretability of intelligent models for downstream tasks. Our code is available here: https://github.com/DLAIResearch/SGC.

Use of Autoencoders for Improving the Performance of Classification and Interpretation of Convolutional Neural Networks in Histopathological Images

Breast cancer is one of the most common types of cancer and it presents itself as being the leading cause of death among women. If its diagnosis occur soon enough, the probability to achieve the cure of the patient can be increased. Recently, it has been more common the use of deep neural network techniques to aid pathologists in their prognosis, but they still do not fully trust them because they lack interpretability. In light of that, this work investigates if previous training of the models as encoders could enhance their accuracy in both classification and interpretability. There were implemented three models to the BreakHis and BreCaHAD dataset: NASNet Mobile, DenseNET201, and MobileNetV2. The experiments have shown that the three models increased the classification performance and two models improved their interpretability using the proposed strategy. DenseNet201 encoder has performed almost 23% better than its vanilla version in classifying a tumor and the NASNet Mobile encoder has improved 28.5% in its tumor interpretation.

Research on image classification leveraging deep convolutional neural networks and visual cognition

The field of image classification has experienced remarkable improvements with the advent of deep learning techniques, especially Deep Convolutional Neural Networks. The present study provides an extensive exploration of the junction where image classification based on Deep Convolutional Neural Networks meets human visual cognition. Utilizing the inherent ability of these networks to automatically learn hierarchical features from raw pixel data, this research examines their potential in classifying images from diverse complex datasets, emphasizing predominantly on the extensively utilized ImageNet dataset. The initial aspect of this study involves training and evaluating models based on Deep Convolutional Neural Networks on the ImageNet dataset, which comprises millions of labeled images spanning across thousands of categories. Well-established network architectures such as AlexNet, VGGNet, GoogLeNet, and ResNet are employed, and their performance in the challenging task of image classification is assessed. Rigorous experiments highlight the strengths and weaknesses of each model while emphasizing the prospects of transfer learning and fine-tuning. Following this, the interpretability of Deep Convolutional Neural Networks is explored by using visualization techniques to comprehend the learned feature representations. By visualizing activation maps and class-specific saliency maps, valuable insights are gained into the regions of interest that guide the decision-making of these models. Moreover, the correlation between the features extracted by these models and human visual attention mechanisms is examined to shed light on the focus of attention of the models. The study also addresses the difficulties that adversarial attacks, data bias, and generalization capabilities present to Deep Convolutional Neural Networks. Strategies to enhance the robustness and adaptability of the models across various domains are examined, linking these observations to human cognitive behavior.

Interpretability of Neural Networks Based on Game-theoretic Interactions

Interpretability of Neural Networks Based on Game-theoretic Interactions

Engravings, Secrets, and Interpretability of Neural Networks

Engravings, Secrets, and Interpretability of Neural Networks

NN2Poly: A Polynomial Representation for Deep Feed-Forward Artificial Neural Networks.

Interpretability of neural networks (NNs) and their underlying theoretical behavior remain an open field of study even after the great success of their practical applications, particularly with the emergence of deep learning. In this work, NN2Poly is proposed: a theoretical approach to obtain an explicit polynomial model that provides an accurate representation of an already trained fully connected feed-forward artificial NN a multilayer perceptron (MLP). This approach extends a previous idea proposed in the literature, which was limited to single hidden layer networks, to work with arbitrarily deep MLPs in both regression and classification tasks. NN2Poly uses a Taylor expansion on the activation function, at each layer, and then applies several combinatorial properties to calculate the coefficients of the desired polynomials. Discussion is presented on the main computational challenges of this method, and the way to overcome them by imposing certain constraints during the training phase. Finally, simulation experiments as well as applications to real tabular datasets are presented to demonstrate the effectiveness of the proposed method.

AD-CAM: Enhancing Interpretability of Convolutional Neural Networks with a Lightweight Framework - From Black Box to Glass Box.

In the realm of machine vision, the convolutional neural network (CNN) is a frequently used and significant deep learning method. It is challenging to comprehend how predictions are formed since the inner workings of CNNs are sometimes seen as a black box. As a result, there has been an increase in interest among AI experts in creating AI systems that are easier to understand. Many strategies have shown promise in improving the interpretability of CNNs, including Class Activation Map (CAM), Grad-CAM, LIME, and other CAM-based approaches. These methods do, however, have certain drawbacks, such as architectural constraints or the requirement for gradient computations. We provide a simple framework termed Adaptive Learning based CAM (Adaptive-CAM) to take advantage of the connection between activation maps and network predictions. This framework includes temporarily masking particular feature maps. According to the Average Drop-Coherence-Complexity (ADCC) metrics, our method outperformed Score-CAM and another CAM-based activation map strategy in Residual Network-based models. With the exception of the VGG16 model, which witnessed a 1.94% decline in performance, the performance improvement spans from 3.78% to 7.72%. Additionally, Adaptive-CAM generates saliency maps that are on par with CAM-based methods and around 153 times superior to other CAM-based methods.

An Analysis Method for Interpretability of Convolutional Neural Network in Bearing Fault Diagnosis

An Analysis Method for Interpretability of Convolutional Neural Network in Bearing Fault Diagnosis

Making Phase-Picking Neural Networks More Consistent and Interpretable

Abstract Improving the interpretability of phase-picking neural networks remains an important task to facilitate their deployment to routine, real-time seismic monitoring. The popular phase-picking neural networks published in the literature lack interpretability because their output prediction scores do not necessarily correspond with the reliability of phase picks and can even be highly inconsistent depending on how we window the waveform data. Here, we show that systematically shifting the waveforms during training and using an antialiasing filter within the neural network architecture can substantially improve the consistency of the output prediction scores and can even make them scale with the signal-to-noise ratios of the waveforms. We demonstrate the improvements by applying these approaches to a commonly used phase-picking neural network architecture and using waveform data from the 2019 Ridgecrest earthquake sequence.

Adversarial Attack Model Based on Deep Neural Network Interpretability and Artificial Fish Swarm Algorithm

Adversarial Attack Model Based on Deep Neural Network Interpretability and Artificial Fish Swarm Algorithm

Semi-White-Box Strategy: Enhancing Data Efficiency and Interpretability of Convolutional Neural Networks in Image Processing

Data-hunger is a persistent challenge in machine learning, particularly in the field of image processing based on convolutional neural networks (CNNs). This study systematically investigates the factors contributing to data-hunger in machine-learning-based image-processing algorithms. The results revealed that the proliferation of model parameters, the lack of interpretability, and the complexity of model structure are significant factors influencing data-hunger. Based on these findings, this paper introduces a novel semi-white-box neural network model construction strategy. This approach effectively reduces the number of model parameters while enhancing the interpretability of model components. It accomplishes this by constraining uninterpretable processes within the model and leveraging prior knowledge of image processing for model. Rather than relying on a single all-in-one model, a semi-white-box model is composed of multiple smaller models, each responsible for extracting fundamental semantic features. The final output is derived from these features and prior knowledge. The proposed strategy holds the potential to substantially decrease data requirements under specific data source conditions while improving the interpretability of model components. Validation experiments are conducted on well-established datasets, including MNIST, Fashion MNIST, CIFAR, and generated data. The results demonstrate the superiority of the semi-white-box strategy over the traditional all-in-one approach in terms of accuracy when trained with equivalent data volumes. Impressively, on the tested datasets, a simplified semi-white-box model achieves performance close to that of ResNet while utilizing a small number of parameters. Furthermore, the semi-white-box strategy offers improved interpretability and parameter reusability features that are challenging to achieve with the all-in-one approach. In conclusion, this paper contributes to mitigating data-hunger challenges in machine-learning-based image processing through the introduction of a novel semi-white-box model construction strategy, backed by empirical evidence of its effectiveness.

Learnable PM diffusion coefficients and reformative coordinate attention network for low dose CT denoising

Objective. Various deep learning methods have recently been used for low dose CT (LDCT) denoising. Aggressive denoising may destroy the edge and fine anatomical structures of CT images. Therefore a key issue in LDCT denoising tasks is the difficulty of balancing noise/artifact suppression and edge/structure preservation. Approach. We proposed an LDCT denoising network based on the encoder-decoder structure, namely the Learnable PM diffusion coefficient and efficient attention network (PMA-Net). First, using the powerful feature modeling capability of partial differential equations, we constructed a multiple learnable edge module to generate precise edge information, incorporating the anisotropic image processing idea of Perona–Malik (PM) model into the neural network. Second, a multiscale reformative coordinate attention module was designed to extract multiscale information. Non-overlapping dilated convolution capturing abundant contextual content was combined with coordinate attention which could embed the spatial location information of important features into the channel attention map. Finally, we imposed additional constraints on the edge information using edge-enhanced multiscale perceptual loss to avoid structure loss and over-smoothing. Main results. Experiments are conducted on simulated and real datasets. The quantitative and qualitative results show that the proposed method has better performance in suppressing noise/artifacts and preserving edges/structures. Significance. This work proposes a novel edge feature extraction method that unfolds partial differential equation into neural networks, which contributes to the interpretability and clinical application value of neural network.

Improving the Accuracy of the Neural Network Models Interpretation of Nonlinear Dynamic Objects

The paper is devoted to the problem of neural network interpretation in the tasks of modeling nonlinear dynamic objects. The purpose of the work is to improve the accuracy of the neural network models interpretation of nonlinear dynamic objects and to determine the scope of their effective application. This goal is achieved by applying analytical models in the form of integral-power series based on multidimensional weight functions. The scientific novelty of the work lies in the use of nonlinear dynamic models in the form of integral-power series based on multidimensional weight functions instead of linear surrogate models. It allows to improve modeling accuracy. The practical usefulness of the work is determination of the effective application area of analytical interpretive models. The practical significance of the obtained results lies in the application of the proposed models for the interpretation of neural network models of nonlinear dynamic objects, which allows to increase the accuracy of interpretation models compared to linear surrogate models.