Black-box Machine Learning Models Research Articles

The accuracy versus interpretability trade-off in fraud detection model

AbstractLike a hydra, fraudsters adapt and circumvent increasingly sophisticated barriers erected by public or private institutions. Among these institutions, banks must quickly take measures to avoid losses while guaranteeing the satisfaction of law-abiding customers. Facing an expanding flow of operations, effective banking relies on data analytics to support established risk control processes, but also on a better understanding of the underlying fraud mechanism. In addition, fraud being a criminal offence, the evidential aspect of the process must also be considered. These legal, operational, and strategic constraints lead to compromises on the means to be implemented for fraud management. This paper first focuses on the translation of practical questions raised in the banking industry at each step of the fraud management process into performance evaluation required to design a fraud detection model. Secondly, it considers a range of machine learning approaches that address these specificities: the imbalance between fraudulent and nonfraudulent operations, the lack of fully trusted labels, the concept-drift phenomenon, and the unavoidable trade-off between accuracy and interpretability of detection. This state-of-the-art review sheds some light on a technology race between black box machine learning models improved by post-hoc interpretation and intrinsic interpretable models boosted to gain accuracy. Finally, it discusses how concrete and promising hybrid approaches can provide pragmatic, short-term answers to banks and policy makers without swallowing up stakeholders with economical and ethical stakes in this technological race.

Personalized Monitoring Model for Electrocardiogram Signals: Diagnostic Accuracy Study.

BackgroundDue to the COVID-19 pandemic, the demand for remote electrocardiogram (ECG) monitoring has increased drastically in an attempt to prevent the spread of the virus and keep vulnerable individuals with less severe cases out of hospitals. Enabling clinicians to set up remote patient ECG monitoring easily and determining how to classify the ECG signals accurately so relevant alerts are sent in a timely fashion is an urgent problem to be addressed for remote patient monitoring (RPM) to be adopted widely. Hence, a new technique is required to enable routine and widespread use of RPM, as is needed due to COVID-19.ObjectiveThe primary aim of this research is to create a robust and easy-to-use solution for personalized ECG monitoring in real-world settings that is precise, easily configurable, and understandable by clinicians.MethodsIn this paper, we propose a Personalized Monitoring Model (PMM) for ECG data based on motif discovery. Motif discovery finds meaningful or frequently recurring patterns in patient ECG readings. The main strategy is to use motif discovery to extract a small sample of personalized motifs for each individual patient and then use these motifs to predict abnormalities in real-time readings of that patient using an artificial logical network configured by a physician.ResultsOur approach was tested on 30 minutes of ECG readings from 32 patients. The average diagnostic accuracy of the PMM was always above 90% and reached 100% for some parameters, compared to 80% accuracy for the Generalized Monitoring Models (GMM). Regardless of parameter settings, PMM training models were generated within 3-4 minutes, compared to 1 hour (or longer, with increasing amounts of training data) for the GMM.ConclusionsOur proposed PMM almost eliminates many of the training and small sample issues associated with GMMs. It also addresses accuracy and computational cost issues of the GMM, caused by the uniqueness of heartbeats and training issues. In addition, it addresses the fact that doctors and nurses typically do not have data science training and the skills needed to configure, understand, and even trust existing black box machine learning models.

Hourly performance forecast of a dew point cooler using explainable Artificial Intelligence and evolutionary optimisations by 2050

The empirical success of the Artificial Intelligence (AI), has enhanced importance of the transparency in black box Machine Learning (ML) models. This study pioneers in developing an explainable and interpretable Deep Neural Network (DNN) model for a Guideless Irregular Dew Point Cooler (GIDPC). The game theory based SHapley Additive exPlanations (SHAP) method is used to interpret contribution of the operating conditions on performance parameters. Furthermore, in a response to the endeavours in developing more efficient metaheuristic optimisation algorithms for the energy systems, two Evolutionary Optimisation (EO) algorithms including a novel bio-inspired algorithm i.e., Slime Mould Algorithm (SMA), and Particle Swarm Optimization (PSO), are employed to simultaneously maximise the cooling efficiency and minimise the construction cost of the GIDPC. Additionally, performance of the optimised GIDPCs are compared in both statistical and deterministic way. The comparisons are carried out in diverse climates in 2020 and 2050 in which the hourly future weather data are projected using a high-emission scenario defined by Intergovernmental Panel for Climate Change (IPCC). The results revealed that the hourly COP of the optimised systems outperform the base design. Although power consumption of all systems increases from 2020 to 2050, owing to more operating hours as a result of global warming, but power savings of up to 72%, 69.49%, 63.24%, and 69.21% in hot summer continental, Arid, tropical rainforest and Mediterranean hot summer climates respectively, can be achieved when the systems run optimally.

Evaluation of white-box versus black-box machine learning models in estimating ambient black carbon concentration

Air quality prediction with black-box (BB) modelling is gaining widespread interest in research and industry. This type of data-driven models work generally better in terms of accuracy but are limited to capture physical, chemical and meteorological processes and therefore accountability for interpretation. In this paper, we evaluated different white-box (WB) and BB methods that estimate atmospheric black carbon (BC) concentration by a suite of observations from the same measurement site. This study involves data in the period of 1st January 2017–31st December 2018 from two measurement sites, from a street canyon site in Mäkelänkatu and from an urban background site in Kumpula, in Helsinki, Finland. At the street canyon site, WB models performed (R2 = 0.81–0.87) in a similar way as the BB models did (R2 = 0.86–0.87). The overall performance of the BC concentration estimation methods at the urban background site was much worse probably because of a combination of smaller dynamic variability in the BC values and longer data gaps. However, the difference in WB (R2 = 0.44–0.60) and BB models (R2 = 0.41–0.64) was not significant. Furthermore, the WB models are closer to physics-based models, and it is easier to spot the relative importance of the predictor variable and determine if the model output makes sense. This feature outweighs slightly higher performance of some individual BB models, and inherently the WB models are a better choice due to their transparency in the model architecture. Among all the WB models, IAP and LASSO are recommended due to its flexibility and its efficiency, respectively. Our findings also ascertain the importance of temporal properties in statistical modelling. In the future, the developed BC estimation model could serve as a virtual sensor and complement the current air quality monitoring. Main findingsWhite-box models are preferred over black-box models in estimating black carbon because they are closer to physics-based models, and it is easier to spot the relative importance of the predictor variable. The black carbon model could serve as a virtual sensor integrating into air quality network in support with real measurements, so as to complement the current air quality index.

Ada-WHIPS: explaining AdaBoost classification with applications in the health sciences

BackgroundComputer Aided Diagnostics (CAD) can support medical practitioners to make critical decisions about their patients’ disease conditions. Practitioners require access to the chain of reasoning behind CAD to build trust in the CAD advice and to supplement their own expertise. Yet, CAD systems might be based on black box machine learning models and high dimensional data sources such as electronic health records, magnetic resonance imaging scans, cardiotocograms, etc. These foundations make interpretation and explanation of the CAD advice very challenging. This challenge is recognised throughout the machine learning research community. eXplainable Artificial Intelligence (XAI) is emerging as one of the most important research areas of recent years because it addresses the interpretability and trust concerns of critical decision makers, including those in clinical and medical practice.MethodsIn this work, we focus on AdaBoost, a black box model that has been widely adopted in the CAD literature. We address the challenge – to explain AdaBoost classification – with a novel algorithm that extracts simple, logical rules from AdaBoost models. Our algorithm, Adaptive-Weighted High Importance Path Snippets (Ada-WHIPS), makes use of AdaBoost’s adaptive classifier weights. Using a novel formulation, Ada-WHIPS uniquely redistributes the weights among individual decision nodes of the internal decision trees of the AdaBoost model. Then, a simple heuristic search of the weighted nodes finds a single rule that dominated the model’s decision. We compare the explanations generated by our novel approach with the state of the art in an experimental study. We evaluate the derived explanations with simple statistical tests of well-known quality measures, precision and coverage, and a novel measure stability that is better suited to the XAI setting.ResultsExperiments on 9 CAD-related data sets showed that Ada-WHIPS explanations consistently generalise better (mean coverage 15%-68%) than the state of the art while remaining competitive for specificity (mean precision 80%-99%). A very small trade-off in specificity is shown to guard against over-fitting which is a known problem in the state of the art methods.ConclusionsThe experimental results demonstrate the benefits of using our novel algorithm for explaining CAD AdaBoost classifiers widely found in the literature. Our tightly coupled, AdaBoost-specific approach outperforms model-agnostic explanation methods and should be considered by practitioners looking for an XAI solution for this class of models.

Using black-box performance models to detect performance regressions under varying workloads: an empirical study

Performance regressions of large-scale software systems often lead to both financial and reputational losses. In order to detect performance regressions, performance tests are typically conducted in an in-house (non-production) environment using test suites with predefined workloads. Then, performance analysis is performed to check whether a software version has a performance regression against an earlier version. However, the real workloads in the field are constantly changing, making it unrealistic to resemble the field workloads in predefined test suites. More importantly, performance testing is usually very expensive as it requires extensive resources and lasts for an extended period. In this work, we leverage black-box machine learning models to automatically detect performance regressions in the field operations of large-scale software systems. Practitioners can leverage our approaches to complement or replace resource-demanding performance tests that may not even be realistic in a fast-paced environment. Our approaches use black-box models to capture the relationship between the performance of a software system (e.g., CPU usage) under varying workloads and the runtime activities that are recorded in the readily-available logs. Then, our approaches compare the black-box models derived from the current software version with an earlier version to detect performance regressions between these two versions. We performed empirical experiments on two open-source systems and applied our approaches on a large-scale industrial system. Our results show that such black-box models can effectively and timely detect real performance regressions and injected ones under varying workloads that are unseen when training these models. Our approaches have been adopted in practice to detect performance regressions of a large-scale industry system on a daily basis.

Peeking into the Black Box: An Actuarial Case Study for Interpretable Machine Learning

This tutorial gives an overview of tools for explaining and interpreting black box machine learning models like boosted trees or deep neural networks. All our methods are illustrated on a publicly available real car insurance data set on claims frequencies.

Mapping the Risk Terrain for Crime Using Machine Learning

We illustrate how a machine learning algorithm, Random Forests, can provide accurate long-term predictions of crime at micro places relative to other popular techniques. We also show how recent advances in model summaries can help to open the ‘black box’ of Random Forests, considerably improving their interpretability. We generate long-term crime forecasts for robberies in Dallas at 200 by 200 feet grid cells that allow spatially varying associations of crime generators and demographic factors across the study area. We then show how using interpretable model summaries facilitate understanding the model’s inner workings. We find that Random Forests greatly outperform Risk Terrain Models and Kernel Density Estimation in terms of forecasting future crimes using different measures of predictive accuracy, but only slightly outperform using prior counts of crime. We find different factors that predict crime are highly non-linear and vary over space. We show how using black-box machine learning models can provide accurate micro placed based crime predictions, but still be interpreted in a manner that fosters understanding of why a place is predicted to be risky.

Combining machine learning and process engineering physics towards enhanced accuracy and explainability of data-driven models

Machine learning models are often considered as black-box solutions which is one of the main reasons why they are still not widely used in operation of process engineering systems. One approach to overcome this problem is to combine machine learning with first principles models of a process engineering system. In this work, we investigate different methods of combining machine learning with first principles and test them on a case study of multiphase flowrate estimation in a petroleum production system. However, the methods can be applied to any process engineering system. The results show that by adding physics-based models to machine learning, it is possible not only to improve the performance of the purely black-box machine learning models, but also to make them more transparent and interpretable. We also propose a step-by-step procedure for selecting a method for combining physics and machine learning depending on the process engineering system conditions.

Mapping the risk terrain for crime using machine learning

We illustrate how a machine learning algorithm, Random Forests, can provide accurate long-term predictions of crime at micro places relative to other popular techniques. We also show how recent advances in model summaries can help to open the ‘black box’ of Random Forests, considerably improving their interpretability. We generate long-term crime forecasts for robberies in Dallas at 200 by 200 feet grid cells that allow spatially varying associations of crime generators and demographic factors across the study area. We then show how using interpretable model summaries facilitate understanding the model’s inner workings. We find that Random Forests greatly outperform Risk Terrain Models and Kernel Density Estimation in terms of forecasting future crimes using different measures of predictive accuracy, but only slightly outperform using prior counts of crime. We find different factors that predict crime are highly non-linear and vary over space. We show how using black-box machine learning models can provide accurate micro placed based crime predictions, but still be interpreted in a manner that fosters understanding of why a place is predicted to be risky.

An integrated computational intelligence technique based operating parameters optimization scheme for quality improvement oriented process-manufacturing system

The analysis and improvement of product quality for process industry is an increasing concern for academia and industry. As the outputs of a manufacturing system mainly depend on corresponding input conditions, so it is of high significance to develop an optimization scheme to actively and accurately determine operating parameters to obtain desired quality. However, the widely employed single-model modeling mode for whole production process neglects the natural characteristics within process manufacturing system such as multistage manufacturing and hysteresis. Additionally, the popular data-driven modeling techniques in current works, especially black-box machine learning models have been restricted to satisfying the requirements regarding excellent approximation capability and explicit mathematical expression simultaneously. To fill up above research gap, it is meaningful to develop a new data-driven optimization scheme in this work to effectively and accurately determine the optimum operating parameters considering the abovementioned characteristics and requirements. Firstly, two different connecting strategies are discussed to determine the more accurate and feasible quality propagation mode between adjacent stages. Then, two computational intelligence (CI) techniques, i.e., Multi-Gene Genetic Programming (MGGP) and Multi-objective Particle Swarm Optimization (MOPSO) algorithm are exploited to construct correlation model with explicit mathematical expression and derive the optimal operating parameters, respectively. Afterwards, the fuzzy Multi-criteria Decision Making (FMCDM) method is further proposed to select the optimal solution from the obtained Pareto solutions sets. The application of the proposed scheme in a coal preparation process indicates that the proposed scheme is promising and competitive on prediction accuracy and optimization efficiency over baseline methods, and can significantly improve the final product quality comparing with initial parameters setting. Moreover, the feasible quality specification for intermediate product can also be obtained by our proposed scheme which is beneficial for early detection of quality abnormality and timely parameters adjustment.

Model-Based Deep Learning for One-Bit Compressive Sensing

In this work, we consider the problem of one-bit deep compressive sensing from both a system design and a signal recovery perspective. In particular, we develop hybrid model-based deep learning architectures based on the deep unfolding methodology. We further interpret the overall data-acquisition and signal recovery modules as an auto-encoder structure allowing for learning task-specific sensing matrix, quantization thresholds, as well as the latent-parameters of iterative first-order optimization algorithms specifically designed for the problem of one-bit sparse signal recovery. The proposed model-based deep architectures have the ability to adaptively learn the proper quantization thresholds, paving the way for amplitude recovery in one-bit compressive sensing. We further show that the proposed methodology implicitly learns task-specific sensing matrices with very low coherence, which is highly desirable in a compressive sensing setting. Due to the model-based nature of the proposed deep architecture, it enjoys from the interpretability and versatility of model-based techniques as well as benefiting from the expressive power of data-driven methods. Specifically, owing to its model-based nature, it has far fewer parameters and requires far less samples for training as compared to black-box machine learning models. Our results demonstrate a significant improvement compared to state-of-the-art algorithms.

Data-Driven Approach for Predicting and Explaining the Risk of Long-Term Unemployment

Long-term unemployment has significant societal impact and is of particular concerns for policymakers with regard to economic growth and public finances. This paper constructs advanced ensemble machine learning models to predict citizens’ risks of becoming long-term unemployed using data collected from European public authorities for employment service. The proposed model achieves 81.2% accuracy on identifying citizens with high risks of long-term unemployment. This paper also examines how to dissect black-box machine learning models by offering explanations at both a local and global level using SHAP, a state-of-the-art model-agnostic approach to explain factors that contribute to long-term unemployment. Lastly, this paper addresses an under-explored question when applying machine learning in the public domain, that is, the inherent bias in model predictions. The results show that popular models such as gradient boosted trees may produce unfair predictions against senior age groups and immigrants. Overall, this paper sheds light on the recent increasing shift for governments to adopt machine learning models to profile and prioritize employment resources to reduce the detrimental effects of long-term unemployment and improve public welfare.

Query-efficient label-only attacks against black-box machine learning models

Recent studies have shown that machine learning algorithms are susceptible to imperceptible perturbations. These studies focus on the laboratory environment, where the attacker has knowledge about internal information of the victim model or feedbacks such as class probabilities. There is still a gap between theory and physical world, the risk of adversarial attacks under the more extreme and realistic condition needs to be figured out. Here we propose a knowledge restricted black-box attack model where the attacker can only get the final predict label. In the meantime, we model the attacker as a resource-restricted one, such as query-limited. The limitations of knowledge level and resources make previous work unable to be directly applied. For this problem, the current state-of-the-art method is boundary attack, however it requires large a number of queries. In this paper, we make several contributions to investigate the vulnerability of machine learning models in more realistic scenarios. First, we reconstruct the optimization problem, measure the quality of the sample points by L2 distance. Second, we provide a more effective algorithm, using cutting plane method and local optimization. Third, we propose two effective dynamic defense strategy, which is easy to implement. At last, we conduct an experimental evaluation on MNIST, Fashion-MNIST and malware detection datasets. The results show that (1) compared with state-of-the-art method, our cutting plane method reduces the number of queries while ensuring the attack efficiency; (2) Dynamic defense strategy is effective against label-only adversarial attacks, the rate of attack success dropped from nearly 100% to 23%, with a considerable classification accuracy; (3) Improved defense strategy guarantees the effectiveness of defense and improves the stability of the whole model.

A Machine-Learning Methodology Using Domain-Knowledge Constraints for Well-Data Integration and Well-Production Prediction

Summary In this paper, we propose a machine–learning methodology using domain–knowledge constraints for well–data integration, prior/expert–knowledge incorporation, and sweet–spot identification. Such methodology enables the analysis of the effects of the main variables involved in production prediction and the evaluation of what–if scenarios of production prediction within geological zones. This methodology will allow streamlining the process of data integration, analytics and machine learning for better decisions, saving time, and helping geologists and reservoir and completion engineers in the task of sweet–spot identification and completion design. We tested the proposed methodology with production, completions, and petrophysical data from a field within a geological target zone. For instance, using local Kriging, we estimated gamma ray features from gamma ray measurements from vertical and horizontal wells, and we integrated those features into the production– and completion–well data, generating an integrated data set for machine–learning modeling. Besides the usual black–box machine–learning models, we used generalized additive models (GAMs) and shape–constraint additive models (SCAMs) for predictive modeling. Those models permit the incorporation of prior/expert knowledge in terms of interaction terms and mathematical constraints on the shape of the effect of the covariates, such as petrophysical and completion parameters, resulting in greater accuracy and interpretability of the predicted production vs. classical black–box machine–learning modeling. We also defined hypothetical what–if scenarios of oil production, such as by estimating the empirical distributions of production estimates using hypothetical settings for completions within the region of interest.

Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead.

Black box machine learning models are currently being used for high stakes decision-making throughout society, causing problems throughout healthcare, criminal justice, and in other domains. People have hoped that creating methods for explaining these black box models will alleviate some of these problems, but trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practices and can potentially cause catastrophic harm to society. There is a way forward - it is to design models that are inherently interpretable. This manuscript clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare, and computer vision.

Black-Box Models and Sociological Explanations: Predicting High School Grade Point Average Using Neural Networks

The Fragile Families Challenge provided an opportunity to empirically assess the applicability of black-box machine learning models to sociological questions and the extent to which interpretable explanations can be extracted from these models. In this article the author uses neural network models to predict high school grade point average and examines how variations of basic network parameters affect predictive performance. Using a recently proposed technique, the author identifies the most important predictive variables used by the best performing model, finding that they relate to parenting and the child’s cognitive and behavioral development, consistent with prior work. The author concludes by discussing the implications of these findings for the relationship between prediction and explanation in sociological analyses.

Hit and run crashes: Knowledge extraction from bicycle involved crashes using first and frugal tree

Abstract Hit and run crash is a punishable offense. In many cases, it involves higher severity levels of the associated roadway users due to the delay of emergency help. For vulnerable road users like pedestrians and bicyclists, this issue is more gruesome. We present an analysis of the effect of crash, geometric, and environmental characteristics on the bicycle-involved hit and run crashes by using six years of Louisiana crash data with an application of fast and frugal tree (FFT) heuristics algorithm. Over 1000 bicycle-involved hit and run crashes occurred in Louisiana out of around 108,000 hit and run crashes during 2010–2015. The fatal bicycle crashes represent 10% of the total fatal hit and run crashes. Additionally, hit and run bicycle crashes represent 22% of total bicycle crashes in Louisiana. In the preliminary analysis, we provided statistical significance test of the key contributing factors for two major groups (bicycle-involved hit and run crashes, and not bicycle-involved hit and run crashes). We divided the complete dataset into two separate datasets: training data for model development, and testing data for performance evaluation. FFT identifies five major cues or variable threshold attributes that contribute significantly in predicting bicycle-involved crashes. These cues include fatal and injury crashes, right angle/turning/head on collisions, city streets/others, intersection, and residential/mixed localities. The balanced accuracy is around 76% for both training and testing data. The current model shows higher sensitivity than other complex and black box machine learning models (e.g., support vector machine, random forest). Findings of our study will provide valuable insights for hit and run bicycle crash reduction in both planning and operation levels.

Decision-making framework with double-loop learning through interpretable black-box machine learning models

PurposeThe purpose of this paper is to address the problem of weak acceptance of machine learning (ML) models in business. The proposed framework of top-performing ML models coupled with general explanation methods provides additional information to the decision-making process. This builds a foundation for sustainable organizational learning.Design/methodology/approachTo address user acceptance, participatory approach of action design research (ADR) was chosen. The proposed framework is demonstrated on a B2B sales forecasting process in an organizational setting, following cross-industry standard process for data mining (CRISP-DM) methodology.FindingsThe provided ML model explanations efficiently support business decision makers, reduce forecasting error for new sales opportunities, and facilitate discussion about the context of opportunities in the sales team.Research limitations/implicationsThe quality and quantity of available data affect the performance of models and explanations.Practical implicationsThe application in the real-world company demonstrates the utility of the approach and provides evidence that transparent explanations of ML models contribute to individual and organizational learning.Social implicationsAll used methods are available as an open-source software and can improve the acceptance of ML in data-driven decision making.Originality/valueThe proposed framework incorporates existing ML models and general explanation methodology into a decision-making process. To the authors’ knowledge, this is the first attempt to support organizational learning with a framework combining ML explanations, ADR, and data mining methodology based on the CRISP-DM industry standard.

Interpretable Classification Models for Recidivism Prediction

SummaryWe investigate a long-debated question, which is how to create predictive models of recidivism that are sufficiently accurate, transparent and interpretable to use for decision making. This question is complicated as these models are used to support different decisions, from sentencing, to determining release on probation to allocating preventative social services. Each case might have an objective other than classification accuracy, such as a desired true positive rate TPR or false positive rate FPR. Each (TPR, FPR) pair is a point on the receiver operator characteristic (ROC) curve. We use popular machine learning methods to create models along the full ROC curve on a wide range of recidivism prediction problems. We show that many methods (support vector machines, stochastic gradient boosting and ridge regression) produce equally accurate models along the full ROC curve. However, methods that are designed for interpretability (classification and regression trees and C5.0) cannot be tuned to produce models that are accurate and/or interpretable. To handle this shortcoming, we use a recent method called supersparse linear integer models to produce accurate, transparent and interpretable scoring systems along the full ROC curve. These scoring systems can be used for decision making for many different use cases, since they are just as accurate as the most powerful black box machine learning models for many applications, but completely transparent, and highly interpretable.