Interpreting Machine Learning Research Articles

Prioritizing user requirements for digital products using explainable artificial intelligence: A data-driven analysis on video conferencing apps

The advent of Industry 5.0 has brought a wealth of digital information to mobile app stores. With the help of emerging technologies such as machine learning and explainable artificial intelligence (XAI), these large amounts of user-generated data can be efficiently captured and analyzed. In this study, we propose an app store analysis framework and demonstrate the utility of the framework by mining and prioritizing user requirements in three popular video conferencing apps. We used the Sentistrength sentiment analysis tool, structural topic modeling, the Gephi web analysis tool, machine learning, and XAI techniques to conduct an in-depth analysis of user requirements in Microsoft Teams, ZOOM Cloud Meetings, and Google Meet. The findings indicated that Steal data, Audio and video quality, Customer service, Hacker issues, Meeting and account passwords, Mute and unmute, Features, and Office platform were the web conferencing system's key areas for improvement. The study demonstrated the usability of app store analysis frameworks and the great potential of XAI to provide insights about requirements prioritization by interpreting machine learning models. Additionally, it offered valuable suggestions for app developers on using the massive data in app stores to improve their apps.

Using Generative AI to Improve the Performance and Interpretability of Rule-Based Diagnosis of Type 2 Diabetes Mellitus

Introduction: Type 2 diabetes mellitus is a major global health concern, but interpreting machine learning models for diagnosis remains challenging. This study investigates combining association rule mining with advanced natural language processing to improve both diagnostic accuracy and interpretability. This novel approach has not been explored before in using pretrained transformers for diabetes classification on tabular data. Methods: The study used the Pima Indians Diabetes dataset to investigate Type 2 diabetes mellitus. Python and Jupyter Notebook were employed for analysis, with the NiaARM framework for association rule mining. LightGBM and the dalex package were used for performance comparison and feature importance analysis, respectively. SHAP was used for local interpretability. OpenAI GPT version 3.5 was utilized for outcome prediction and interpretation. The source code is available on GitHub. Results: NiaARM generated 350 rules to predict diabetes. LightGBM performed better than the GPT-based model. A comparison of GPT and NiaARM rules showed disparities, prompting a similarity score analysis. LightGBM’s decision making leaned heavily on glucose, age, and BMI, as highlighted in feature importance rankings. Beeswarm plots demonstrated how feature values correlate with their influence on diagnosis outcomes. Discussion: Combining association rule mining with GPT for Type 2 diabetes mellitus classification yields limited effectiveness. Enhancements like preprocessing and hyperparameter tuning are required. Interpretation challenges and GPT’s dependency on provided rules indicate the necessity for prompt engineering and similarity score methods. Variations in feature importance rankings underscore the complexity of T2DM. Concerns regarding GPT’s reliability emphasize the importance of iterative approaches for improving prediction accuracy.

Enhancing winter road maintenance with explainable AI: SHAP analysis for interpreting machine learning models in road friction estimation

Effective winter road maintenance relies on precise road friction estimation. Machine learning (ML) models have shown significant promise in this; however, their inherent complexity makes understanding their inner workings challenging. This paper addresses this issue by conducting a comparative analysis of road friction estimation models using four ML methods, including regression tree, random forest, eXtreme Gradient Boosting (XGBoost), and support vector regression (SVR). We then employ the SHapley Additive exPlanations (SHAP) explainable artificial intelligence (AI) to enhance model interpretability. Our analysis on an Alberta dataset reveals that the XGBoost model performs best with an accuracy of 91.39%. The SHAP analysis illustrates the logical relationships between predictor features and friction within all three tree-based models, but it also uncovers inconsistencies within the SVR model, potentially attributed to insufficient feature interactions. Thus, this paper not only showcase the role of explainable AI in improving the ML interpretability of models for road friction estimation, but also provides practical insights that could improve winter road maintenance decisions.

Big data and machine learning in cost estimation: An automotive case study

This paper presents a case study on the applicability of machine learning and big data technology for product cost estimation, using data on the material cost of passenger cars. The study provides contributions to six research aspects. First, we show which machine learning algorithms are appropriate when dealing with product cost estimation for highly complex products with more than 2000 parts and hundreds of cost drivers. Second, our case study provides a novel approach to increasing the predictive accuracy of cost estimates for subsequent product generations. Third, we show that the accuracy is up to 3.5 times higher when using big data compared to an intermediate size of data. Fourth, machine learning can outperform cost estimates from cost experts during the early stage of new product development, even when dealing with highly complex products. Then, we evaluate the use cases, issues, and benefits of machine learning and big data from the perspective of cost experts. Specifically, the case study shows that machine learning can reliably select the most important cost drivers (fifth aspect) and calculate the average cost of cost drivers over thousands of product configurations (sixth aspect). However, cost experts must be knowledgeable about the product and remain careful when interpreting machine learning outcomes, as they can yield misleading outcomes in some exceptional cases. In conclusion, machine learning and big data empirically proved to be able to generate additional value in many aspects for managing costs during the early phase of new product development.

Novel Correlation for the Solid–Liquid Mass Transfer Coefficient in Stirred Tanks Developed by Interpreting Machine Learning Models Trained on Literature Data

Novel Correlation for the Solid–Liquid Mass Transfer Coefficient in Stirred Tanks Developed by Interpreting Machine Learning Models Trained on Literature Data

CatBoost-Based Automatic Classification Study of River Network

Existing research on automatic river network classification methods has difficulty scientifically quantifying and determining feature threshold settings and evaluating weights when calculating multi-indicator features of the local and overall structures of river reaches. In order to further improve the accuracy of river network classification and evaluate the feature weight, this paper proposes an automatic grading method for river networks based on ensemble learning in CatBoost. First, the graded river network based on expert knowledge is taken as the case; with the support of the existing case results, a total of eight features from the semantic, geometric, and topological aspects of the river network were selected for calculation. Second, the classification model, obtained through learning and training, was used to calculate the classification results of the main stream and tributaries of the river reach to be classified. Furthermore, the main stream river reaches were connected, and the main stream rivers at different levels were hierarchized to achieve river network classification. Finally, the Shapley Additive explanation (SHAP) framework for interpreting machine learning models was introduced to test the influence of feature terms on the classification results from the global and local aspects, so as to improve the interpretability and transparency of the model. Performance evaluation can determine the advantages and disadvantages of the classifier, improve the classification effect and practicability of the classifier, and improve the accuracy and reliability of river network classification. The experiment demonstrates that the proposed method achieves expert-level imitation and has higher accuracy for identifying the main stream and tributaries of river networks. Compared with other classification algorithms, the accuracy was improved by 0.85–5.94%, the precision was improved by 1.82–9.84%, and the F1_Score was improved by 0.8–5.74%. In this paper, CatBoost is used for river network classification for the first time, and SHAP is used to explain the influence of characteristics, which improves the accuracy of river network classification and enhances the interpretability of the classification method. By constructing a reasonable hierarchy, a better grading effect can be achieved, and the intelligence level of automatic grading of river networks can be further improved.

The importance of interpreting machine learning models for blood glucose prediction in diabetes: an analysis using SHAP

Machine learning has become a popular tool for learning models of complex dynamics from biomedical data. In Type 1 Diabetes (T1D) management, these models are increasingly been integrated in decision support systems (DSS) to forecast glucose levels and provide preventive therapeutic suggestions, like corrective insulin boluses (CIB), accordingly. Typically, models are chosen based on their prediction accuracy. However, since patient safety is a concern in this application, the algorithm should also be physiologically sound and its outcome should be explainable. This paper aims to discuss the importance of using tools to interpret the output of black-box models in T1D management by presenting a case-of-study on the selection of the best prediction algorithm to integrate in a DSS for CIB suggestion. By retrospectively “replaying” real patient data, we show that two long-short term memory neural networks (LSTM) (named p-LSTM and np-LSTM) with similar prediction accuracy could lead to different therapeutic decisions. An analysis with SHAP—a tool for explaining black-box models’ output—unambiguously shows that only p-LSTM learnt the physiological relationship between inputs and glucose prediction, and should therefore be preferred. This is verified by showing that, when embedded in the DSS, only p-LSTM can improve patients’ glycemic control.

Are false positives in suicide classification models a risk group? Evidence for "true alarms" in a population-representative longitudinal study of Norwegian adolescents.

False positives in retrospective binary suicide attempt classification models are commonly attributed to sheer classification error. However, when machine learning suicide attempt classification models are trained with a multitude of psycho-socio-environmental factors and achieve high accuracy in suicide risk assessment, false positives may turn out to be at high risk of developing suicidal behavior or attempting suicide in the future. Thus, they may be better viewed as "true alarms," relevant for a suicide prevention program. In this study, using large population-based longitudinal dataset, we examine three hypotheses: (1) false positives, compared to the true negatives, are at higher risk of suicide attempt in future, (2) the suicide attempts risk for the false positives increase as a function of increase in specificity threshold; and (3) as specificity increases, the severity of risk factors between false positives and true positives becomes more similar. Utilizing the Gradient Boosting algorithm, we used a sample of 11,369 Norwegian adolescents, assessed at two timepoints (1992 and 1994), to classify suicide attempters at the first time point. We then assessed the relative risk of suicide attempt at the second time point for false positives in comparison to true negatives, and in relation to the level of specificity. We found that false positives were at significantly higher risk of attempting suicide compared to true negatives. When selecting a higher classification risk threshold by gradually increasing the specificity cutoff from 60% to 97.5%, the relative suicide attempt risk of the false positive group increased, ranging from minimum of 2.96 to 7.22 times. As the risk threshold increased, the severity of various mental health indicators became significantly more comparable between false positives and true positives. We argue that the performance evaluation of machine learning suicide classification models should take the clinical relevance into account, rather than focusing solely on classification error metrics. As shown here, the so-called false positives represent a truly at-risk group that should be included in suicide prevention programs. Hence, these findings should be taken into consideration when interpreting machine learning suicide classification models as well as planning future suicide prevention interventions for adolescents.

Explainable machine learning to inform real-world evidence: A case study on small cell lung cancer survival analysis.

113 Background: In recent years, machine learning algorithms for survival analysis have been developed as an alternative to traditional Cox regression. Despite the good performances, these are viewed as black box models that lack interpretability, which may limit their clinical applications. There has been a growing interest in explaining and interpreting machine learning models, which has led to the emergence of the field of explainable artificial intelligence (XAI). However, few studies have focused on how these methods could be of benefit to survival analysis of real-world healthcare data. Methods: In this study, we explored the use of current XAI techniques to analyze the effects of platinum doublet chemotherapy on small cell lung cancer patients in developed machine learning models.The analyzed data included real-world patients treated at Karolinska University Hospital (n=570) and three phase III randomized clinical trials shared by the Project Data Sphere initiative (n=987). The real-world data covariates included were age, sex, TNM staging, ECOG performance status, lab values, brain metastasis, and concomitant radiotherapy. The aggregated dataset including the clinical trials contained the following variables: age, sex, performance status, brain metastasis, and protocol violations. Eight machine learning models were trained and compared with Cox regression. The performance of the models was evaluated using C-index, and the time variation of Brier Score and C/D AUC.Temporal feature importance and partial dependence were used to explore the overall covariate impact on overall survival (Global XAI). Ceteris-paribus and SurvShap(t) were used to investigate the covariate impact for the single patient prediction (local XAI). The models were firstly trained only on real-world data before aggregation with the clinical trials. Results: Ensemble machine learning provided the best performances. XAI techniques showed the potential to increase explainability of survival predictions in function of time. Global XAI showed the time range of the model reliability, trend inversions regarding treatment decisions and covariate importance along the time. Local XAI allowed to test the impact of covariates between long survival patients and the comparison between real-world and clinical trials. Conclusions: Our results demonstrate the potential of XAI techniques applied to survival machine learning and real-world data, thus providing insights into the mechanisms driving model predictions and demonstrate the utility of this approach in clinical research.

Explainability meets uncertainty quantification: Insights from feature-based model fusion on multimodal time series

Feature importance evaluation is one of the prevalent approaches to interpreting Machine Learning (ML) models. A drawback of using these methods for high-dimensional datasets is that they often lead to high-dimensional explanation output that hinders human analysis. This is especially true for explaining multimodal ML models, where the problem’s complexity is further exacerbated by the inclusion of multiple data modalities and an increase in the overall number of features. This work proposes a novel approach to lower the complexity of feature-based explanations. The proposed approach is based on uncertainty quantification techniques, allowing for a principled way of reducing the number of modalities required to explain the model’s predictions. We evaluated our method in three multimodal datasets comprising physiological time series. Results show that the proposed method can reduce the complexity of the explanations while maintaining a high level of accuracy in the predictions. This study illustrates an innovative example of the intersection between the disciplines of uncertainty quantification and explainable artificial intelligence.

Curriculum for Early Exposure to Clinical Informatics and Data Science for Noninformatics Trainees to Promote Interest and Inclusion in Informatics

Abstract Background Curricula aimed at increasing exposure to informatics and practical data analytics among medical trainees could increase their effectiveness in clinical research, quality improvement, and clinical operations. Objectives The Clinical Informatics Data Science (CI-DS) pathway is a cross-disciplinary curriculum aimed at improving informatics exposure among medical trainees. We describe the development of this novel curriculum, the inaugural cohort, and lessons learned. Methods The CI-DS pathway is framed around upfront informatics didactics followed by a longitudinal, experiential training focused on mentorship, clinical data extraction/machine learning, and health technology governance. The curriculum was evaluated based on pre- and postpathway surveys completed by learners and logs of the elective activities selected by learners. Results The CI-DS pathway attracted 19 learners across 12 medical subspecialties, from medical students to fellows. Baseline surveys showed limited exposure to informatics across learners. The top three longitudinal activities completed were participating in electronic health record (EHR) governance meetings, data science supplemental courses, and designated mentorship meetings. Comparison of baseline with postpathway surveys demonstrated significant improvements in learner self-reported confidence in appraising an EHR modification ticket, accessing UCSF's deidentified data, exploring a database with basic structured query language (SQL), extracting data using SQL, and interpreting machine learning models. Conclusion An early exposure curriculum in clinical informatics with training in data extraction and governance can successfully recruit a diverse array of learners and improve confidence in practical informatics skills. We reflect on the strengths and weaknesses of this curriculum, and summarize the lessons learned to guide others in creating similar curricula for noninformatics clinicians.

Towards interpreting machine learning models for predicting soil moisture droughts

Determination of the dominant factors which affect soil moisture (SM) predictions for drought analysis is an essential step to assess the reliability of the prediction results. However, artificial intelligence (AI) based drought modelling only provides prediction results without the physical interpretation of the models. Here, we propose an explainable AI (XAI) framework to reveal the modelling of SM drought events. Random forest based site-specific SM prediction models were developed using the data from 30 sites, covering 8 vegetation types. The unity of multiply XAI tools was applied to interpret the site-models both globally (generally) and locally. Globally, the models were interpreted using two methods: permutation importance and accumulated local effect (ALE). On the other hand, for each drought event, the models were interpreted locally via Shapley additive explanations (SHAP), local interpretable model-agnostic explanation (LIME) and individual conditional expectation (ICE) methods. Globally, the dominant features for SM predictions were identified as soil temperature, atmospheric aridity, time variables and latent heat flux. But through local interpretations of the drought events, SM showed a greater reliance on soil temperature, atmospheric aridity and latent heat flux at grass sites, with higher correlation to the time-dependent parameters at the sites located in forests. The temporal variation of the feature which effects the drought events was also demonstrated. The interpretation could shed light on how predictions are made and could promote the application of AI techniques in drought prediction, which may be useful for irrigation and water resource management.

Improving the generalizability of protein-ligand binding predictions with AI-Bind

Identifying novel drug-target interactions is a critical and rate-limiting step in drug discovery. While deep learning models have been proposed to accelerate the identification process, here we show that state-of-the-art models fail to generalize to novel (i.e., never-before-seen) structures. We unveil the mechanisms responsible for this shortcoming, demonstrating how models rely on shortcuts that leverage the topology of the protein-ligand bipartite network, rather than learning the node features. Here we introduce AI-Bind, a pipeline that combines network-based sampling strategies with unsupervised pre-training to improve binding predictions for novel proteins and ligands. We validate AI-Bind predictions via docking simulations and comparison with recent experimental evidence, and step up the process of interpreting machine learning prediction of protein-ligand binding by identifying potential active binding sites on the amino acid sequence. AI-Bind is a high-throughput approach to identify drug-target combinations with the potential of becoming a powerful tool in drug discovery.

Neural network interpretation techniques for analysis of histological images of breast abnormalities

Background. Neural networks are actively used in digital pathology to analyze histological images and support medical decision-making. A common approach is to solve the classification problem, where only class labels are the only model responses. However, one should understand which areas of the image have the most significant impact on the model's response. Machine learning interpretation techniques help solve this problem.
 Aim. To study the consistency of different methods of neural network interpretation when classifying histological images of the breast and to obtain an expert assessment of the results of the evaluated methods.
 Materials and methods. We performed a preliminary analysis and pre-processing of the existing data set used to train pre-selected neural network models. The existing methods of visualizing the areas of attention of trained models on easy-to-understand data were applied, followed by verification of their correct use. The same neural network models were trained on histological data, and the selected interpretation methods were used to systematize histological images, followed by the evaluation of the results consistency and an expert assessment of the results.
 Results. In this paper, several methods of interpreting machine learning are studied using two different neural network architectures and a set of histological images of breast abnormalities. Results of ResNet18 and ViT-B-16 models training on a set of histological images on the test sample: accuracy metric 0.89 and 0.89, ROC_AUC metric 0.99 and 0.96, respectively. The results were also evaluated by an expert using the Label Studio tool. For each pair of images, the expert was asked to select the most appropriate answer ("Yes" or "No") to the question: "The highlighted areas generally correspond to the Malignant class." The "Yes" response rate for the ResNet_Malignant category was 0.56; for ViT_Malignant, it was 1.0.
 Conclusion. Interpretability experiments were conducted with two different architectures: the ResNet18 convolutional network and the ViT-B-16 attention-enhanced network. The results of the trained models were visualized using the GradCAM and Attention Rollout methods, respectively. First, experiments were conducted on a simple-to-interpret dataset to ensure they were used correctly. The methods are then applied to the set of histological images. In easy-to-understand images (cat images), the convolutional network is more consistent with human perception; on the contrary, in histological images of breast cancer, ViT-B-16 provided results much more similar to the expert's perception.

Analysis of interpreted machine learning methods for predicting the execution of government contracts in the field of electric power

The power industry plays a key role in ensuring the energy security of the state. Sustainable and reliable functioning of the energy system requires the fulfillment of contracts with strictly observed deadlines and quality of work. This article describes an algorithm for selecting methods for interpreting machine learning models, analyzes gradient boosting-based machine learning methods recommended for solving prediction tasks in the field of power engineering, and presents methods for interpreting the results. The authors have achieved good results in training models and determined objective assessments of the contribution of each feature to solving the prediction tasks of contract fulfillment. This research is significant in the context of ensuring the efficiency and transparency of public procurement and can be beneficial for specialists and government bodies responsible for monitoring contract fulfillment in the field of power engineering.

Analyzing Effective Factors of Online Learning Performance by Interpreting Machine Learning Models

Analyzing Effective Factors of Online Learning Performance by Interpreting Machine Learning Models

Making the black-box brighter: Interpreting machine learning algorithm for forecasting drilling accidents

We present an approach for interpreting a black-box alarming system for forecasting accidents and anomalies during the drilling of oil and gas wells. The interpretation methodology aims to explain the local behavior of the accident predictive model to drilling engineers. The explanatory model uses Shapley additive explanations analysis of features, obtained through Bag-of-features representation of telemetry logs used during the drilling accident forecasting phase. Validation shows that the explanatory model has 15% precision at 70% recall, and overcomes the metric values of a random baseline and multi-head attention neural network. These results justify that the developed explanatory model is better aligned with explanations of drilling engineers, than the state-of-the-art method. The joint performance of explanatory and Bag-of-features models allows drilling engineers to understand the logic behind the system decisions at the particular moment, pay attention to highlighted telemetry regions, and correspondingly, increase the trust level in the accident forecasting alarms.

Distilling seed-assisted zeolite synthesis conditions by machine learning

Traditional zeolite synthesis requires usage of the organic structure-directing agent (OSDA) which is capital-intensive and non-eco-friendly. Seed-assisted zeolite synthesis can reduce or eliminate OSDAs, making it a green and economical approach. Seed-assisted synthesis conditions reside in a high-dimensional chemical space where a brute-force search is unfeasible. Here we utilize archived experiment records to build machine learning models for predicting crystallization tendency. Effects of seed physicochemical properties, gel compositions as well as crystallization conditions are calibrated. In particular, seed framework density, NaOH concentration in the gel and crystallization time have synergistic influences on the zeolite transformation. The machine learning model is interpreted with domain knowledge to obtain chemical hypothesis. The hypothesis is verified by follow-up experiments, distilling zeolite transitions that had been overlooked previously. • Applied and evaluated machine learning techniques to predict crystallization tendency via seed-assisted zeolite synthesis. • Effects of seed properties,gel compositions and crystallization conditions were calibrated by machine learning models. • Emphases were made on interpreting machine learning models. • Chemical hypothesis was obtained from machine learning models to guide follow-up experiments. • Follow up experiments verified the hypothesis and revealed IWV-MWW zeolite transition that was previously overlooked.

Manipulating SHAP via Adversarial Data Perturbations (Student Abstract)

We introduce a model-agnostic algorithm for manipulating SHapley Additive exPlanations (SHAP) with perturbation of tabular data. It is evaluated on predictive tasks from healthcare and financial domains to illustrate how crucial is the context of data distribution in interpreting machine learning models. Our method supports checking the stability of the explanations used by various stakeholders apparent in the domain of responsible AI; moreover, the result highlights the explanations' vulnerability that can be exploited by an adversary.

Interpretable and Explainable Machine Learning for Materials Science and Chemistry

ConspectusMachine learning has become a common and powerful tool in materials research. As more data become available, with the use of high-performance computing and high-throughput experimentation, machine learning has proven potential to accelerate scientific research and technology development. Though the uptake of data-driven approaches for materials science is at an exciting, early stage, to realize the true potential of machine learning models for successful scientific discovery, they must have qualities beyond purely predictive power. The predictions and inner workings of models should provide a certain degree of explainability by human experts, permitting the identification of potential model issues or limitations, building trust in model predictions, and unveiling unexpected correlations that may lead to scientific insights. In this work, we summarize applications of interpretability and explainability techniques for materials science and chemistry and discuss how these techniques can improve the outcome of scientific studies. We start by defining the fundamental concepts of interpretability and explainability in machine learning and making them less abstract by providing examples in the field. We show how interpretability in scientific machine learning has additional constraints compared to general applications. Building upon formal definitions in machine learning, we formulate the basic trade-offs among the explainability, completeness, and scientific validity of model explanations in scientific problems. In the context of these trade-offs, we discuss how interpretable models can be constructed, what insights they provide, and what drawbacks they have. We present numerous examples of the application of interpretable machine learning in a variety of experimental and simulation studies, encompassing first-principles calculations, physicochemical characterization, materials development, and integration into complex systems. We discuss the varied impacts and uses of interpretabiltiy in these cases according to the nature and constraints of the scientific study of interest. We discuss various challenges for interpretable machine learning in materials science and, more broadly, in scientific settings. In particular, we emphasize the risks of inferring causation or reaching generalization by purely interpreting machine learning models and the need for uncertainty estimates for model explanations. Finally, we showcase a number of exciting developments in other fields that could benefit interpretability in material science problems. Adding interpretability to a machine learning model often requires no more technical know-how than building the model itself. By providing concrete examples of studies (many with associated open source code and data), we hope that this Account will encourage all practitioners of machine learning in materials science to look deeper into their models.