Black-box Machine Learning Models Research Articles

Non-Differentiable Loss Function Optimization and Interaction Effect Discovery in Insurance Pricing Using the Genetic Algorithm

Non-Differentiable Loss Function Optimization and Interaction Effect Discovery in Insurance Pricing Using the Genetic Algorithm

Efficient Paddy Grain Quality Assessment Approach Utilizing Affordable Sensors

Paddy (Oryza sativa) is one of the most consumed food grains in the world. The process from its sowing to consumption via harvesting, processing, storage and management require much effort and expertise. The grain quality of the product is heavily affected by the weather conditions, irrigation frequency, and many other factors. However, quality control is of immense importance, and thus, the evaluation of grain quality is necessary. Since it is necessary and arduous, we try to overcome the limitations and shortcomings of grain quality evaluation using image processing and machine learning (ML) techniques. Most existing methods are designed for rice grain quality assessment, noting that the key characteristics of paddy and rice are different. In addition, they have complex and expensive setups and utilize black-box ML models. To handle these issues, in this paper, we propose a reliable ML-based IoT paddy grain quality assessment system utilizing affordable sensors. It involves a specific data collection procedure followed by image processing with an ML-based model to predict the quality. Different explainable features are used for classifying the grain quality of paddy grain, like the shape, size, moisture, and maturity of the grain. The precision of the system was tested in real-world scenarios. To our knowledge, it is the first automated system to precisely provide an overall quality metric. The main feature of our system is its explainability in terms of utilized features and fuzzy rules, which increases the confidence and trustworthiness of the public toward its use. The grain variety used for experiments majorly belonged to the Indian Subcontinent, but it covered a significant variation in the shape and size of the grain.

Interpretable AI Services for Enhanced Air Quality Forecasting

Most of the Machine Learning (ML) models used these days establish a complex relationship between the in- dependent variables (X) and dependent variable (y). Without understanding the relationship, we risk introducing undesirable features into the predictions. Biased collection of the data, used to build the model, might bolster these undesirable features. The model might soon become unfit for its intended tasks. This project tries to get deeper insights into such black box machine learning models by looking into various ExplainableAI (XAI) tools and provide it as a service to users. These tools when used in conjunction can make complex models easy to understand and operate for the end-user. Specifically, the tools used would help the user of the machine learning model interact with it and monitor how it behaves on changing certain aspects of the data. To facilitate the better understanding of the achieved outcome, this project uses a weather data-set which is used to classify the air quality.

Optimising machine learning prediction of minimum inhibitory concentrations in Klebsiella pneumoniae.

Minimum Inhibitory Concentrations (MICs) are the gold standard for quantitatively measuring antibiotic resistance. However, lab-based MIC determination can be time-consuming and suffers from low reproducibility, and interpretation as sensitive or resistant relies on guidelines which change over time. Genome sequencing and machine learning promise to allow in silico MIC prediction as an alternative approach which overcomes some of these difficulties, albeit the interpretation of MIC is still needed. Nevertheless, precisely how we should handle MIC data when dealing with predictive models remains unclear, since they are measured semi-quantitatively, with varying resolution, and are typically also left- and right-censored within varying ranges. We therefore investigated genome-based prediction of MICs in the pathogen Klebsiella pneumoniae using 4367 genomes with both simulated semi-quantitative traits and real MICs. As we were focused on clinical interpretation, we used interpretable rather than black-box machine learning models, namely, Elastic Net, Random Forests, and linear mixed models. Simulated traits were generated accounting for oligogenic, polygenic, and homoplastic genetic effects with different levels of heritability. Then we assessed how model prediction accuracy was affected when MICs were framed as regression and classification. Our results showed that treating the MICs differently depending on the number of concentration levels of antibiotic available was the most promising learning strategy. Specifically, to optimise both prediction accuracy and inference of the correct causal variants, we recommend considering the MICs as continuous and framing the learning problem as a regression when the number of observed antibiotic concentration levels is large, whereas with a smaller number of concentration levels they should be treated as a categorical variable and the learning problem should be framed as a classification. Our findings also underline how predictive models can be improved when prior biological knowledge is taken into account, due to the varying genetic architecture of each antibiotic resistance trait. Finally, we emphasise that incrementing the population database is pivotal for the future clinical implementation of these models to support routine machine-learning based diagnostics.

I-CEE: Tailoring Explanations of Image Classification Models to User Expertise

Effectively explaining decisions of black-box machine learning models is critical to responsible deployment of AI systems that rely on them. Recognizing their importance, the field of explainable AI (XAI) provides several techniques to generate these explanations. Yet, there is relatively little emphasis on the user (the explainee) in this growing body of work and most XAI techniques generate "one-size-fits-all'' explanations. To bridge this gap and achieve a step closer towards human-centered XAI, we present I-CEE, a framework that provides Image Classification Explanations tailored to User Expertise. Informed by existing work, I-CEE explains the decisions of image classification models by providing the user with an informative subset of training data (i.e., example images), corresponding local explanations, and model decisions. However, unlike prior work, I-CEE models the informativeness of the example images to depend on user expertise, resulting in different examples for different users. We posit that by tailoring the example set to user expertise, I-CEE can better facilitate users' understanding and simulatability of the model. To evaluate our approach, we conduct detailed experiments in both simulation and with human participants (N = 100) on multiple datasets. Experiments with simulated users show that I-CEE improves users' ability to accurately predict the model's decisions (simulatability) compared to baselines, providing promising preliminary results. Experiments with human participants demonstrate that our method significantly improves user simulatability accuracy, highlighting the importance of human-centered XAI.

Uncovering the heterogeneity of a solar flare mechanism with mixture models

The physics of solar flares occurring on the Sun is highly complex and far from fully understood. However, observations show that solar eruptions are associated with the intense kilogauss fields of active regions, where free energies are stored with field-aligned electric currents. With the advent of high-quality data sources such as the Geostationary Operational Environmental Satellites (GOES) and Solar Dynamics Observatory (SDO)/Helioseismic and Magnetic Imager (HMI), recent works on solar flare forecasting have been focusing on data-driven methods. In particular, black box machine learning and deep learning models are increasingly being adopted in which underlying data structures are not modeled explicitly. If the active regions indeed follow the same laws of physics, similar patterns should be shared among them, reflected by the observations. Yet, these black box models currently used in the literature do not explicitly characterize the heterogeneous nature of the solar flare data within and between active regions. In this paper, we propose two finite mixture models designed to capture the heterogeneous patterns of active regions and their associated solar flare events. With extensive numerical studies, we demonstrate the usefulness of our proposed method for both resolving the sample imbalance issue and modeling the heterogeneity for rare energetic solar flare events.

A stroke prediction framework using explainable ensemble learning

The death of brain cells occurs when blood flow to a particular area of the brain is abruptly cut off, resulting in a stroke. Early recognition of stroke symptoms is essential to prevent strokes and promote a healthy lifestyle. FAST tests (looking for abnormalities in the face, arms, and speech) have limitations in reliability and accuracy for diagnosing strokes. This research employs machine learning (ML) techniques to develop and assess multiple ML models to establish a robust stroke risk prediction framework. This research uses a stacking-based ensemble method to select the best three machine learning (ML) models and combine their collective intelligence. An empirical evaluation of a publicly available stroke prediction dataset demonstrates the superior performance of the proposed stacking-based ensemble model, with only one misclassification. The experimental results reveal that the proposed stacking model surpasses other state-of-the-art research, achieving accuracy, precision, F1-score of 99.99%, recall of 100%, receiver operating characteristics (ROC), Mathews correlation coefficient (MCC), and Kappa scores 1.0. Furthermore, Shapley’s Additive Explanations (SHAP) are employed to analyze the predictions of the black-box machine learning (ML) models. The findings highlight that age, BMI, and glucose level are the most significant risk factors for stroke prediction. These findings contribute to the development of more efficient techniques for stroke prediction, potentially saving many lives.

Using slisemap to interpret physical data.

Manifold visualisation techniques are commonly used to visualise high-dimensional datasets in physical sciences. In this paper, we apply a recently introduced manifold visualisation method, slisemap, on datasets from physics and chemistry. slisemap combines manifold visualisation with explainable artificial intelligence. Explainable artificial intelligence investigates the decision processes of black box machine learning models and complex simulators. With slisemap, we find an embedding such that data items with similar local explanations are grouped together. Hence, slisemap gives us an overview of the different behaviours of a black box model, where the patterns in the embedding reflect a target property. In this paper, we show how slisemap can be used and evaluated on physical data and that it is helpful in finding meaningful information on classification and regression models trained on these datasets.

Knowledge-guided machine learning can improve carbon cycle quantification in agroecosystems

Accurate and cost-effective quantification of the carbon cycle for agroecosystems at decision-relevant scales is critical to mitigating climate change and ensuring sustainable food production. However, conventional process-based or data-driven modeling approaches alone have large prediction uncertainties due to the complex biogeochemical processes to model and the lack of observations to constrain many key state and flux variables. Here we propose a Knowledge-Guided Machine Learning (KGML) framework that addresses the above challenges by integrating knowledge embedded in a process-based model, high-resolution remote sensing observations, and machine learning (ML) techniques. Using the U.S. Corn Belt as a testbed, we demonstrate that KGML can outperform conventional process-based and black-box ML models in quantifying carbon cycle dynamics. Our high-resolution approach quantitatively reveals 86% more spatial detail of soil organic carbon changes than conventional coarse-resolution approaches. Moreover, we outline a protocol for improving KGML via various paths, which can be generalized to develop hybrid models to better predict complex earth system dynamics.

Using SHAP Values and Machine Learning to Understand Trends in the Transient Stability Limit

Machine learning (ML) for transient stability assessment has gained traction due to the significant increase in computational requirements as renewables connect to power systems. To achieve a high degree of accuracy; black-box ML models are often required – inhibiting interpretation of predictions and consequently reducing confidence in the use of such methods. This paper proposes the use of SHapley Additive exPlanations (SHAP) – a unifying interpretability framework based on Shapley values from cooperative game theory – to provide insights into ML models that are trained to predict critical clearing time (CCT). We use SHAP to obtain explanations of location-specific ML models trained to predict CCT at each busbar on the network. This can provide unique insights into power system variables influencing the entire stability boundary under increasing system complexity and uncertainty. Subsequently, the covariance between a variable of interest and the corresponding SHAP values from each location-specific ML model – can reveal how a change in that variable impacts the stability boundary throughout the network. Such insights can inform planning and/or operational decisions. The case study provided demonstrates the method using a highly accurate opaque ML algorithm in the IEEE 39-bus test network with Type IV wind generation.

Interact: A visual what-if analysis tool for virtual product design

Virtual prototyping is increasingly used by businesses to streamline operations, cut costs, and enhance daily operations. This often includes a variety of modeling techniques among which, complex, black-box models. The path from model development to utilization in applied contexts is yet long. Domain experts need to be convinced of the validity of the models and to trust their predictions. To be used in the field, model capabilities need to be affordable, that is, allow rapid and interactive scenario building, even for non-experts. Complex relations governed by statistical interactions must be unveiled for users to understand unexpected predictions. We propose Interact, a model-agnostic, visual what-if tool for regression problems, supporting (1) the visualization of statistical interactions between features, (2) the creation of interactive what-if scenarios using predictive models, (3) the evaluation of model quality and building trust, and (4) the externalization of knowledge through model explainability. While the approach applies in various industrial contexts, we validate the application purpose and design with a detailed case study and a qualitative user study with engineers in the tire industry. By unraveling statistical interactions between features, the INTERACT tool proves to be useful to increase the transparency of black-box machine learning models. We also reflect on lessons learned concerning the development of visual what-if tools for virtual product development and beyond.

From Black Boxes to Actionable Insights: A Perspective on Explainable Artificial Intelligence for Scientific Discovery.

The application of Explainable Artificial Intelligence (XAI) in the field of chemistry has garnered growing interest for its potential to justify the prediction of black-box machine learning models and provide actionable insights. We first survey a range of XAI techniques adapted for chemical applications and categorize them based on the technical details of each methodology. We then present a few case studies to illustrate the practical utility of XAI, such as identifying carcinogenic molecules and guiding molecular optimizations, in order to provide chemists with concrete examples of ways to take full advantage of XAI-augmented machine learning for chemistry. Despite the initial success of XAI in chemistry, we still face the challenges of developing more reliable explanations, assuring robustness against adversarial actions, and customizing the explanation for different applications and needs of the diverse scientific community. Finally, we discuss the emerging role of large language models like GPT in generating natural language explanations and discusses the specific challenges associated with them. We advocate that addressing the aforementioned challenges and actively embracing new techniques may contribute to establishing machine learning as an indispensable technique for chemistry in this digital era.

Generation of Molecular Counterfactuals for Explainable Machine Learning Based on Core-Substituent Recombination.

The use of black box machine learning models whose decisions cannot be understood limits the acceptance of predictions in interdisciplinary research and camouflages artificial learning characteristics leading to predictions for other than anticipated reasons. Consequently, there is increasing interest in explainable artificial intelligence to rationalize predictions and uncover potential pitfalls. Among others, relevant approaches include feature attribution methods to identify molecular structures determining predictions and counterfactuals (CFs) or contrastive explanations. CFs are defined as variants of test instances with minimal modifications leading to opposing predictions. In medicinal chemistry, CFs have thus far only been little investigated although they are particularly intuitive from a chemical perspective. We introduce a new methodology for the systematic generation of CFs that is centered on well-defined structural analogues of test compounds. The approach is transparent, computationally straightforward, and shown to provide a wealth of CFs for test sets. The method is made freely available.

Explainable Boosting Machine: A Contemporary Glass-Box Model to Analyze Work Zone-Related Road Traffic Crashes

Examining the factors contributing to work zone crashes and implementing measures to reduce their occurrence can significantly improve road safety. In this research, we utilized the explainable boosting machine (EBM), a modern glass-box machine learning (ML) model, to categorize and predict work zone-related crashes and to interpret the various contributing factors. The issue of data imbalance was also addressed by utilizing work zone crash data from the state of New Jersey, comprising data collected over the course of two years (2017 and 2018) and applying data augmentation strategies such synthetic minority over-sampling technique (SMOTE), borderline-SMOTE, and SVM-SMOTE. The EBM model was trained using augmented data and Bayesian optimization for hyperparameter tuning. The performance of the EBM model was evaluated and compared to black-box ML models such as combined kernel and tree boosting (KTBoost, python 3.7.1 and KTboost package version 0.2.2), light gradient boosting machine (LightGBM version 3.2.1), and extreme gradient boosting (XGBoost version 1.7.6). The EBM model, using borderline-SMOTE-treated data, demonstrated greater efficacy with respect to precision (81.37%), recall (82.53%), geometric mean (75.39%), and Matthews correlation coefficient (0.43). The EBM model also allows for an in-depth evaluation of single and pairwise factor interactions in predicting work zone-related crash severity. It examines both global and local perspectives, and assists in assessing the influence of various factors.

Towards white box modeling of compressive strength of sustainable ternary cement concrete using explainable artificial intelligence (XAI)

Since the production of sustainable ternary cement concrete (TCC) involves a large range of constituents which can affect the compressive strength (CS) of TCC in different ways, the evaluation of CS in a unified manner is necessary. Unlike the black box machine learning (ML) models, the predictions of CS of TCC using local explanations to global understanding with explainable artificial intelligence (XAI) models have been investigated in a systematic approach by means of SHapley Additive exPlanations (SHAP). For this, two conventional ML and three ensemble ML models were used. Hyper-parameter tuning was also carried out. The ensemble ML models had a higher accuracy than the conventional ML models. Taylor diagram, model evaluation parameters and the SHAP values were used to interpret machine learning model predictivity. From insights obtained through local explanations, it can be concluded that predictions made by black box models should be used carefully.

CoSP: co-selection pick for a global explainability of black box machine learning models

CoSP: co-selection pick for a global explainability of black box machine learning models

Explainable Machine Learning for Credit Risk Management When Features are Dependent

ABSTRACT Complex Machine Learning (ML) models used to support decision-making in peer-to-peer (P2P) lending often lack clear, accurate, and interpretable explanations. While the game-theoretic concept of Shapley values and its computationally efficient variant Kernel SHAP may be employed for this aim, similarly to other existing methods, the latter makes the assumption that the features are independent. The assumption of uncorrelated features in credit risk management is fairly restrictive and, thus, prediction explanations coming from correlated features might result in highly misleading Shapley values, even when considering simple models. We therefore propose an evaluation of different dependent-feature estimation methods of Kernel SHAP for classification purposes in credit risk management. We show that dependent-feature estimation of Shapley values can improve the understanding of true prediction explanations, their robustness and is essential for better identifying the most relevant variables to default predictions coming from black-box ML models.

Explainable machine learning for hydrocarbon prospect risking

Hydrocarbon prospect risk assessment is an important process in oil and gas exploration involving the integrated analysis of various geophysical data modalities, including seismic data, well logs, and geologic information, to estimate the likelihood of drilling success for a given drill location. Over the years, geophysicists have attempted to understand the various factors at play influencing the probability of success for hydrocarbon prospects. Toward this end, a large database of prospect drill outcomes and associated attributes has been collected and analyzed via correlation-based techniques to determine the features that contribute the most in deciding the final outcome. Machine learning (ML) has the potential to model complex feature interactions to learn input-output mappings for complicated high-dimensional data sets. However, in many instances, ML models are not interpretable to end users, limiting their utility toward understanding the underlying scientific principles for the problem domain and being deployed to assist in the risk assessment process. In this context, we leverage the concept of explainable ML to interpret various black-box ML models trained on the aforementioned prospect database for risk assessment. Using various case studies on real data, we determine that this model-agnostic explainability analysis for prospect risking can (1) reveal novel scientific insights into the interplay of various features in regard to deciding prospect outcome, (2) assist with performing feature engineering for ML models, (3) detect bias in data sets involving spurious correlations, and (4) build a global picture of a model’s understanding of the data by aggregating local explanations on individual data points.

IHCP: interpretable hepatitis C prediction system based on black-box machine learning models

BackgroundHepatitis C is a prevalent disease that poses a high risk to the human liver. Early diagnosis of hepatitis C is crucial for treatment and prognosis. Therefore, developing an effective medical decision system is essential. In recent years, many computational methods have been proposed to identify hepatitis C patients. Although existing hepatitis prediction models have achieved good results in terms of accuracy, most of them are black-box models and cannot gain the trust of doctors and patients in clinical practice. As a result, this study aims to use various Machine Learning (ML) models to predict whether a patient has hepatitis C, while also using explainable models to elucidate the prediction process of the ML models, thus making the prediction process more transparent.ResultWe conducted a study on the prediction of hepatitis C based on serological testing and provided comprehensive explanations for the prediction process. Throughout the experiment, we modeled the benchmark dataset, and evaluated model performance using fivefold cross-validation and independent testing experiments. After evaluating three types of black-box machine learning models, Random Forest (RF), Support Vector Machine (SVM), and AdaBoost, we adopted Bayesian-optimized RF as the classification algorithm. In terms of model interpretation, in addition to using common SHapley Additive exPlanations (SHAP) to provide global explanations for the model, we also utilized the Local Interpretable Model-Agnostic Explanations with stability (LIME_stabilitly) to provide local explanations for the model.ConclusionBoth the fivefold cross-validation and independent testing show that our proposed method significantly outperforms the state-of-the-art method. IHCP maintains excellent model interpretability while obtaining excellent predictive performance. This helps uncover potential predictive patterns of the model and enables clinicians to better understand the model's decision-making process.

Geospatial XAI: A Review

Explainable Artificial Intelligence (XAI) has the potential to open up black-box machine learning models. XAI can be used to optimize machine learning models, to search for scientific findings, or to improve the understandability of the AI system for the end users. Geospatial XAI refers to AI systems that apply XAI techniques to geospatial data. Geospatial data are associated with geographical locations or areas and can be displayed on maps. This paper provides an overview of the state-of-the-art in the field of geospatial XAI. A structured literature review is used to present and discuss the findings on the main objectives, the implemented machine learning models, and the used XAI techniques. The results show that research has focused either on using XAI in geospatial use cases to improve model quality or on scientific discovery. Geospatial XAI has been used less for improving understandability for end users. The used techniques to communicate the AI analysis results or AI findings to users show that there is still a gap between the used XAI technique and the appropriate visualization method in the case of geospatial data.