Black-box Behavior Research Articles

Uncertainty-aware regression model to predict post-operative visual acuity in patients with macular holes

Full-thickness macular holes are a relatively common and visually disabling condition with a prevalence of approximately 0.5% in the over-40-year-old age group. If left untreated, the hole typically enlarges, reducing visual acuity (VA) below the definition of blindness in the eye affected. They are now routinely treated with surgery, which can close the hole and improve vision in most cases. The extent of improvement, however, is variable and dependent on the size of the hole and other features which can be discerned in spectral-domain optical coherence tomography imaging, which is now routinely available in eye clinics globally. Artificial intelligence (AI) models have been developed to enable surgical decision-making and have achieved relatively high predictive performance. However, their black-box behaviour is opaque to users and uncertainty associated with their predictions is not typically stated, leading to a lack of trust among clinicians and patients. In this paper, we describe an uncertainty-aware regression model (U-ARM) for predicting VA for people undergoing macular hole surgery using preoperative spectral-domain optical coherence tomography images, achieving an MAE of 6.07, RMSE of 9.11 and R2 of 0.47 in internal tests, and an MAE of 6.49, RMSE of 9.49, and R2 of 0.42 in external tests. In addition to predicting VA following surgery, U-ARM displays its associated uncertainty, a p-value of ¡0.005 in internal and external tests, showing the predictions are not due to random chance. We then qualitatively evaluated the performance of U-ARM. Lastly, we demonstrate out-of-sample data performance, generalising well to data outside the training distribution, low-quality images, and unseen instances not encountered during training. The results show that U-ARM outperforms commonly used methods in terms of prediction and reliability. U-ARM is thus a promising approach for clinical settings and can improve the reliability of AI models in predicting VA.

Leveraging CAM Algorithms for Explaining Medical Semantic Segmentation

Convolutional neural networks (CNNs) achieve prevailing results in segmentation tasks nowadays and represent the state-of-the-art for image-based analysis. However, the under- standing of the accurate decision-making process of a CNN is rather unknown. The research area of explainable artificial intelligence (xAI) primarily revolves around understanding and interpreting this black-box behavior. One way of interpreting a CNN is the use of class ac- tivation maps (CAMs) that represent heatmaps to indicate the importance of image areas for the prediction of the CNN. For classification tasks, a variety of CAM algorithms exist. But for segmentation tasks, only one CAM algorithm for the interpretation of the output of a CNN exist. We propose a transfer between existing classification- and segmentation- based methods for more detailed, explainable, and consistent results which show salient pixels in semantic segmentation tasks. The resulting Seg-HiRes-Grad CAM is an exten- sion of the segmentation-based Seg-Grad CAM with the transfer to the classification-based HiRes CAM. Our method improves the previously-mentioned existing segmentation-based method by adjusting it to recently published classification-based methods. Especially for medical image segmentation, this transfer solves existing explainability disadvantages. The code is available at <a href='https://github.com/TillmannRheude/SegHiResGrad_CAM'>https://github.com/TillmannRheude/SegHiResGrad_CAM</a>

Exploring machine learning applications in chemical production through valorization of biomass, plastics, and petroleum resources: A comprehensive review

Machine learning (ML) is a subtype of artificial intelligence that uses a computer's ability to learn from a given set of accessible data. ML is becoming prominent in almost every business, including the domain of chemical engineering, where there have been numerous researches and investigations. This article provides a detailed overview of the use of ML in the production and characterization study of biomass, polymers, and petroleum products. Categories of ML, including classification, regression, and clustering, are also investigated to get a deeper understanding of ML. From this review, it can be concluded that ML has aided in numerous domains, such as the prediction of biomass energy, the stability of crude oil based on NMR spectroscopy, the calculation of gasoline's octane number, the estimation of fuel oil's kinematic viscosity, the classification of waste plastics, and the estimation of drilling efficiency in petroleum reservoirs, among others. Apart from this, ML has also been playing a significant role in the microwave-assisted pyrolysis of biomass, polymers, and petroleum resources. ML substantially influences chemical engineering and is especially useful for enhancing system efficiency and monitoring processes that are difficult to understand manually. Although several obstacles are associated with ML, such as black box behavior, the need for a large amount of data, and the difficulty of understanding the predictions, deploying the model in the future is uncomplicated once the learning program has been trained.

An eXplainable Artificial Intelligence Methodology on Big Data Architecture

Although artificial intelligence has become part of everyone’s real life, a trust crisis against such systems is occurring, thus increasing the need to explain black-box predictions, especially in the military, medical, and financial domains. Modern eXplainable Artificial Intelligence (XAI) techniques focus on benchmark datasets, but the cognitive applicability of such solutions under big data settings is still unclear due to memory or computation constraints. In this paper, we extend a model-agnostic XAI methodology, named Cluster-Aided Space Transformation for Local Explanation (CASTLE), to be able to deal with high-volume datasets. CASTLE aims to explain the black-box behavior of predictive models by combining both local (i.e., based on the input sample) and global (i.e., based on the whole scope for action of the model) information. In particular, the local explanation provides a rule-based explanation for the prediction of a target instance as well as the directions to update the likelihood of the predicted class. Our extension leverages modern big data technologies (e.g., Apache Spark) to handle the high volume, variety, and velocity of huge datasets. We have evaluated the framework on five datasets, in terms of temporal efficiency, explanation quality, and model significance. Our results indicate that the proposed approach retains the high-quality explanations associated with CASTLE while efficiently handling large datasets. Importantly, it exhibits a sub-linear, rather than exponential, dependence on dataset size, making it a scalable solution for massive datasets or in any big data scenario.

Explainability in AI-based behavioral malware detection systems

Nowadays, our security and privacy are strongly threatened by malware programs which aim to steal our confidential data and make our systems out of service, among other things. While traditional signature-based malware detection methods or statistical analysis have proven to be ineffective and time-consuming, recently data-driven Artificial Intelligence (AI) techniques, i.e. Machine Learning (ML) and Deep Learning (DL) approaches, have been successfully applied leveraging the behavior of malware in terms of API calls, and achieving promising performances. However, their black-box behavior leads to a lack of explainability thus preventing their application in real world scenarios. In light of this, eXplainable Artificial Intelligence (XAI) methodologies and tools can be effectively embedded within an AI-based malware detection process in order to make more understandable the produced results. In this paper, we propose a XAI framework for behavioral malware detection problems and evaluate the usefulness of four XAI methods (SHAP, LIME, LRP and Attention mechanism) on three datasets with different size, sequence length and number of classes, by which we could evaluate the strengths and weaknesses – from effectiveness and efficiency point of views – of recurrent deep architectures (i.e. Long-Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) models), and their applicability in the modern Cyber Security (CS) scenarios.

Toward Interpretable Sleep Stage Classification Using Cross-Modal Transformers.

Accurate sleep stage classification is significant for sleep health assessment. In recent years, several machine-learning based sleep staging algorithms have been developed, and in particular, deep-learning based algorithms have achieved performance on par with human annotation. Despite improved performance, a limitation of most deep-learning based algorithms is their black-box behavior, which have limited their use in clinical settings. Here, we propose a cross-modal transformer, which is a transformer-based method for sleep stage classification. The proposed cross-modal transformer consists of a cross-modal transformer encoder architecture along with a multi-scale one-dimensional convolutional neural network for automatic representation learning. The performance of our method is on-par with the state-of-the-art methods and eliminates the black-box behavior of deep-learning models by utilizing the interpretability aspect of the attention modules. Furthermore, our method provides considerable reductions in the number of parameters and training time compared to the state-of-the-art methods. Our code is available at https://github.com/Jathurshan0330/Cross-Modal-Transformer. A demo of our work can be found at https://bit.ly/Cross_modal_transformer_demo.

A Process Mining-based unsupervised Anomaly Detection technique for the Industrial Internet of Things

Industrial Internet of Things (IIoT) applications in Industry 4.0 collect and process Time Series (TS) originating from heterogeneous sources. Many data-driven techniques have been proposed over the years for unsupervised collective Anomaly Detection (AD) to detect anomalous TS and improve quality of service. These techniques build statistical and/or behavioral models through data-driven algorithms often based on machine and deep learning. However, these algorithms may have black-box behavior, may require too much computational effort, and may not be trustworthy. In order to address these challenges, this paper proposes and evaluates an unsupervised AD technique that pre-processes noisy and complex TS and detects anomalous patterns. Pre-processing handles TS complexity through Feature Extraction and Dimensionality Reduction based on Autoencoders (AEs), whereas anomalous patterns are classified through Process Mining (PM), which unsupervisedly captures TS patterns and compares unknown behavior to such patterns for collective AD. We apply the technique to a scale replica of an assembly plant adopted in a smart automotive factory to evaluate our technique with respect to several configurations to analyze their impact on both detection and timing performance. Our contribution improves and extends the existing state-of-the-art work in the literature regarding the application of PM to IIoT for collective AD in TS. Moreover, it compares the obtained results with a baseline approach based on AEs.

Towards transparent deep learning for surface water detection from SAR imagery

Water detection from SAR imagery has significant values, such as the flood monitoring and environmental protection. Nowadays, significant progress has been achieved in water detection using deep neural network (DNN) methods, but the blackbox behavior incurs many doubts in the performance of deep learning techniques, which undermines its trustworthiness in water detection from SAR imagery. By integrating SAR domain knowledge, DNN and eXplainable Artificial Intelligence (XAI), an explainable DNN framework for surface water detection is proposed for the first time. This framework includes three parts: the water extraction network containing four backbone networks, the Local and Global Mixed Attribution (LGMA) module for performance evaluation of backbone network, and the Semantic Specific-class Activation Mapping (SSAM) module, which performs geo-visualization for the output layers of high-level features. In the experiment, SAR images from different resolutions and frequency-bands are utilized, which are from millimeter-wave and Sentinel-1 systems. The attribution maps and heatmaps of four backbone networks are assessed towards the final water extraction results. The experiment indicates that the proposed framework can glass-box the decision-making process of DNN in water detection and offer corresponding attribution analytics for given input SAR imagery. This work encourages other scholars to conduct extensive research on the explanation of DNN in SAR domain, gradually establish the trustworthiness of DNN, and promote the development of DNN in SAR images analytics.

Explaining deep convolutional models by measuring the influence of interpretable features in image classification

The accuracy and flexibility of Deep Convolutional Neural Networks (DCNNs) have been highly validated over the past years. However, their intrinsic opaqueness is still affecting their reliability and limiting their application in critical production systems, where the black-box behavior is difficult to be accepted. This work proposes EBAnO, an innovative explanation framework able to analyze the decision-making process of DCNNs in image classification by providing prediction-local and class-based model-wise explanations through the unsupervised mining of knowledge contained in multiple convolutional layers. EBAnO provides detailed visual and numerical explanations thanks to two specific indexes that measure the features’ influence and their influence precision in the decision-making process. The framework has been experimentally evaluated, both quantitatively and qualitatively, by (i) analyzing its explanations with four state-of-the-art DCNN architectures, (ii) comparing its results with three state-of-the-art explanation strategies and (iii) assessing its effectiveness and easiness of understanding through human judgment, by means of an online survey. EBAnO has been released as open-source code and it is freely available online.

A scalable, black-box hybrid genetic algorithm for continuous multimodal optimization in moderate dimensions

Optimization problems can be found in many areas of scienceand technology. Not only the global optimum, but also a(large) number of near-optima are often of interest. Thisgives rise to what are referred to as multimodal optimizationproblems. In most cases, the number and quality of the optimaare unknown and assumptions cannot be made about theobjective functions. In this paper, we focus on continuous,unconstrained optimization in moderately high-dimensionalcontinuous spaces (d ≤ 10).We present a scalablealgorithm with virtually no parameters, which performs wellfor general objective functions (non-convex, discontinuous).It is based on two well-established algorithms (CMA-ES,deterministic crowding). Novel elements of the algorithminclude the detection of seed points for local searches andcollision avoidance, both based on nearest neighbors, and astrategy for semi-sequential optimization to realize scalability.The performance of the proposed algorithm is numericallyevaluated using the CEC2013 niching benchmark suite for1 − 20 dimensional functions, and a 9 dimensional real-worldproblem from constraint optimization in climate research.The algorithm performs well in relation to the CEC2013benchmarks and only falls short on higher dimensional andstrongly inisotropic problems. In the case of the climate-related problem, the algorithm is able to find a high number(> 150) of optima of relevance to climate research. Theproposed algorithm does not require special configuration forthe optimization problems considered in this paper, i.e., itshows good black-box behavior.

Stable and actionable explanations of black-box models through factual and counterfactual rules

Recent years have witnessed the rise of accurate but obscure classification models that hide the logic of their internal decision processes. Explaining the decision taken by a black-box classifier on a specific input instance is therefore of striking interest. We propose a local rule-based model-agnostic explanation method providing stable and actionable explanations. An explanation consists of a factual logic rule, stating the reasons for the black-box decision, and a set of actionable counterfactual logic rules, proactively suggesting the changes in the instance that lead to a different outcome. Explanations are computed from a decision tree that mimics the behavior of the black-box locally to the instance to explain. The decision tree is obtained through a bagging-like approach that favors stability and fidelity: first, an ensemble of decision trees is learned from neighborhoods of the instance under investigation; then, the ensemble is merged into a single decision tree. Neighbor instances are synthetically generated through a genetic algorithm whose fitness function is driven by the black-box behavior. Experiments show that the proposed method advances the state-of-the-art towards a comprehensive approach that successfully covers stability and actionability of factual and counterfactual explanations.

Erythemato-Squamous Diseases Prediction and Interpretation Using Explainable AI

Erythemato-squamous diseases (ESD) diagnosis is a significant challenge in dermatology. It is divided into six categories. Artificial intelligence models have been applied to categorize these categories. Artificial intelligent models are black boxes in nature. The objective of this study is to unbox the black-box behavior and interpret the decision-making. Random Forest and XGBoost models are applied on a standard dataset with SHAP value to get interpretability and causability of decision. The Random Forest model had a classification accuracy of 98.21%. Integration of explainability increase the transparency of result and identify the root cause of the disease in the subject. A comprehensive quantitative study will help to adopt artificial intelligence in healthcare with ethical issues like transparency, causability, and interpretability of diagnosis.

An experimental and comparative benchmark study examining resource utilization in managed Hadoop context

Transitioning cloud-based Hadoop frameworks from IaaS to PaaS, which are commercially conceptualized as pay-as-you-go or pay-per-use, often reduces the associated system costs. However, the managed Hadoop systems obscure the inner performance dynamics of the platform and present a black-box behavior to the end-users. The aim of this study was to investigate the resource utilization of current managed Hadoop platforms. Thus, we explored three prominent Hadoop-on-PaaS proposals as they come out-of-the-box and conducted Hadoop-specific workloads using the HiBench Benchmark Suite. During the benchmark executions, the system resource utilization data from the worker nodes were collected and analyzed. The results indicated that the same property specifications among cloud services neither do guarantee similar performance outputs, nor produce consistent results based on different workloads within themselves. We anticipate that the managed systems’ architectures and pre-configurations play a crucial role in the performance outcomes.

Optimization Workflows for Linking Model-Based Systems Engineering (MBSE) and Multidisciplinary Analysis and Optimization (MDAO)

Developing modern products involves numerous domains (controlling, production, engineering, etc.) and disciplines (mechanics, electronics, software, etc.). The products have become increasingly complex while their time to market has decreased. These challenges can be overcome by Model-Based Systems Engineering (MBSE), where all development data (requirements, architecture, etc.) is stored and linked in a system model. In an MBSE system model, product requirements at the system level can lead to numerous technical variants with conflicting objectives at the parameter level. To determine the best technical variants or tradeoffs, Multidisciplinary Analysis and Optimization (MDAO) is already being used today. Linking MBSE and MDAO allows for mutually beneficial synergies to be expected that have not yet been fully exploited. In this paper, a new approach to link MBSE and MDAO is proposed. The novelty compared to existing approaches is the reuse of existing MBSE system model data. Models developed during upstream design and test activities already linked to the MBSE system model were integrated into an MDAO problem. Benefits are reduced initial and reconfiguration efforts and the resolution of the MDAO black-box behavior. For the first time, the MDAO problem was modeled as a workflow using activity diagrams in the MBSE system model. For a given system architecture, this workflow finds the design variable values that allow for the best tradeoff of objectives. The structure and behavior of the workflow were formally described in the MBSE system model with SysML. The presented approach for linking MBSE and MDAO is demonstrated using an example of an electric coolant pump.

Learning interpretable word embeddings via bidirectional alignment of dimensions with semantic concepts

We propose bidirectional imparting or BiImp, a generalized method for aligning embedding dimensions with concepts during the embedding learning phase. While preserving the semantic structure of the embedding space, BiImp makes dimensions interpretable, which has a critical role in deciphering the black-box behavior of word embeddings. BiImp separately utilizes both directions of a vector space dimension: each direction can be assigned to a different concept. This increases the number of concepts that can be represented in the embedding space. Our experimental results demonstrate the interpretability of BiImp embeddings without making compromises on the semantic task performance. We also use BiImp to reduce gender bias in word embeddings by encoding gender-opposite concepts (e.g., male–female) in a single embedding dimension. These results highlight the potential of BiImp in reducing biases and stereotypes present in word embeddings. Furthermore, task or domain-specific interpretable word embeddings can be obtained by adjusting the corresponding word groups in embedding dimensions according to task or domain. As a result, BiImp offers wide liberty in studying word embeddings without any further effort.

Towards an Integrated Evaluation Framework for XAI: An Experimental Study

Increasing prevalence of opaque black-box AI has highlighted the need for explanations of their behaviours, for example, via explanation artefacts/proxy models. The current paper presents a paradigm for human-grounded experiments to evaluate the relationship between explanation fidelity, human learning performance, understanding and trust in a black-box AI by manipulating the complexity of an explanatory artefact. Decision trees were used in the current experiment as exemplar interpretable surrogate models, providing explanations approximating black-box behaviour, by means of explanation by simplification. Consistent with our hypotheses: 1) explanatory artefacts brought about better learning, while greater decision tree depths led to greater interpretability of the AI's performance and greater trust in the AI; and 2) explanatory artefacts facilitated learning and task performance even after they were withdrawn. Findings are discussed in terms of the interplay between human understanding, trust and AI system performance, highlighting the simplifying assumption of a monotonic relationship between explanation fidelity and interpretability.

Developing an Explainable Machine Learning-Based Thyroid Disease Prediction Model

Healthcare and medicine are key areas where machine learning algorithms are widely used. The medical decision support systems thus created are accurate enough, however, they suffer from the lack of transparency in decision making and shows a black box behavior. However, transparency and trust are significant in the field of health and medicine and hence, a black box system is sub optimal in terms of widespread applicability and reach. Hence, the explainablility of the research make the system reliable and understandable, thereby enhancing its social acceptability. The presented work explores a thyroid disease diagnosis system. SHAP, a popular method based on coalition game theory is used for interpretability of results. The work explains the system behavior both locally and globally and shows how machine leaning can be used to ascertain the causality of the disease and support doctors to suggest the most effective treatment of the disease. The work not only demonstrates the results of machine learning algorithms but also explains related feature importance and model insights.

Glassboxing Deep Learning to Enhance Aircraft Detection from SAR Imagery

Although deep learning has achieved great success in aircraft detection from SAR imagery, its blackbox behavior has been criticized for low comprehensibility and interpretability. Such challenges have impeded the trustworthiness and wide application of deep learning techniques in SAR image analytics. In this paper, we propose an innovative eXplainable Artificial Intelligence (XAI) framework to glassbox deep neural networks (DNN) by using aircraft detection as a case study. This framework is composed of three parts: hybrid global attribution mapping (HGAM) for backbone network selection, path aggregation network (PANet), and class-specific confidence scores mapping (CCSM) for visualization of the detector. HGAM integrates the local and global XAI techniques to evaluate the effectiveness of DNN feature extraction; PANet provides advanced feature fusion to generate multi-scale prediction feature maps; while CCSM relies on visualization methods to examine the detection performance with given DNN and input SAR images. This framework can select the optimal backbone DNN for aircraft detection and map the detection performance for better understanding of the DNN. We verify its effectiveness with experiments using Gaofen-3 imagery. Our XAI framework offers an explainable approach to design, develop, and deploy DNN for SAR image analytics.

VNLP: Visible natural language processing

In general, Natural Language Processing (NLP) algorithms exhibit black-box behavior. Users input text and output are provided with no explanation of how the results are obtained. In order to increase understanding and trust, users value transparent processing which may explain derived results and enable understanding of the underlying routines. Many approaches take an opaque approach by default when designing NLP tools and do not incorporate a means to steer and manipulate the intermediate NLP steps. We present an interactive, customizable, visual framework that enables users to observe and participate in the NLP pipeline processes, explicitly manipulate the parameters of each step, and explore the result visually based on user preferences. The visible NLP (VNLP) pipeline design is then applied to a text similarity application to demonstrate the utility and advantages of a visible and transparent NLP pipeline in supporting users to understand and justify both the process and results. We also report feedback on our framework from a modern languages expert.

Introspective analysis of convolutional neural networks for improving discrimination performance and feature visualisation.

Deep neural networks have been widely explored and utilised as a useful tool for feature extraction in computer vision and machine learning. It is often observed that the last fully connected (FC) layers of convolutional neural network possess higher discrimination power as compared to the convolutional and maxpooling layers whose goal is to preserve local and low-level information of the input image and down sample it to avoid overfitting. Inspired from the functionality of local binary pattern (LBP) operator, this paper proposes to induce discrimination into the mid layers of convolutional neural network by introducing a discriminatively boosted alternative to pooling (DBAP) layer that has shown to serve as a favourable replacement of early maxpooling layer in a convolutional neural network (CNN). A thorough research of the related works show that the proposed change in the neural architecture is novel and has not been proposed before to bring enhanced discrimination and feature visualisation power achieved from the mid layer features. The empirical results reveal that the introduction of DBAP layer in popular neural architectures such as AlexNet and LeNet produces competitive classification results in comparison to their baseline models as well as other ultra-deep models on several benchmark data sets. In addition, better visualisation of intermediate features can allow one to seek understanding and interpretation of black box behaviour of convolutional neural networks, used widely by the research community.