Shapley Value Research Articles

A large-scale riverbank erosion risk assessment model integrating multi-source data and explainable artificial intelligence (XAI)

The impact of riverbank erosion poses serious threat to the environment, socio-economics and human safety. Due to the extremely complex mechanisms of erosion, assessing the risk of riverbank erosion is challenging. To address this, we propose an interpretable intelligent model framework to accurately assess large-scale riverbank erosion risk. Firstly, we constructed a multi-source dataset that encompasses 29 riverbank erosion influencing factors. Subsequently, by employing an adaptive feature weighting method, a comprehensive water level factor was synthesized, unifying data dimensions. The Relief algorithm was used to identify influential features for riverbank erosion, and an adaptive feature weighting SMOTE (AFW-SMOTE) algorithm was developed to balance the riverbank erosion dataset. Additionally, an ELM and BiGRU autoencoder was designed to effectively capture and learn key information from static and dynamic features. Finally, the outputs of the two autoencoders were integrated using the XGBoost algorithm to produce riverbank erosion risk assessment results, and the risks were visualized. This model not only performs excellently across multiple evaluation metrics but also significantly surpasses 22 other machine learning models. By integrating the Shapley value method, it enhances the model’s interpretability. This provides policymakers and relevant environmental management agencies with a powerful tool to scientifically assess and manage the risk of riverbank erosion.

A Rank Graduation Box for SAFE AI

The growth of Artificial Intelligence applications requires to develop risk management models that can balance opportunities with risks.We contribute to the development of Artificial Intelligence risk models proposing a Rank Graduation Box (RGB), a set of integrated statistical metrics that can measure the “Sustainability”, “Accuracy”, “Fairness” and “Explainability” of any Artificial Intelligence application.Our metrics are consistent with each other, as they are all derived from a common underlying statistical methodology: the Lorenz curve.The validity of the metrics is assessed by means of their practical application to both simulated and real data. The results from the comparison of alternative machine learning models to simulated data are aligned with the generating models, in general indicating linear regression models as the most accurate, regression tree models as the most fair, Random Forest models as the most robust; and leading to model explanations similar to the true ones from the generating model.The outcomes from the application of Random Forest models to real data show that the proposed RGB metrics are more interpretable and more consistent with the expectations, with respect to standard metrics such as AUC, RMSE and Shapley values. The evidence also shows that the RGB metrics are very general and can be applied to any machine learning method, regardless of the underlying data and model.

Strategies of Mixed Coalitions in Group Lending and Equitable Distribution of Joint Profit

The objective of this paper is to highlight the usefulness of cooperative game theory as a conceptual framework for approaching problems related to the formation of mixed coalitions within group lending and the fair distribution of joint profit. The focus was on a particular technique, the Shapley value, originally proposed as an a priori solution to a cooperative game. Furthermore, the central idea of this approach is indifference to the desires of others: individualism. We succeed in establishing the threshold level for determining the value obtained for a fair distribution of joint resources. The only element that could limit the applicability of this value to any sharing problem comes from the assumption that coalitions are formed in a random way, which is not always respected in reality

Can Psychedelic Use Benefit Meditation Practice? Examining Individual, Psychedelic, and Meditation-Related Factors.

Meditation practice and psychedelic use have attracted increasing attention in the public sphere and scientific research. Both methods induce non-ordinary states of consciousness that may have significant therapeutic benefits. Thus, there is growing scientific interest in potential synergies between psychedelic use and meditation practice with some research suggesting that psychedelics may benefit meditation practice. The present study examined individual, psychedelic-related, and meditation-related factors to determine under what conditions meditators perceive psychedelic use as beneficial for their meditation practice. Participants (N = 863) who had reported psychedelic use and a regular meditation practice (at least 3 times per week during the last 12 months) were included in the study. To accommodate a large number of variables, machine learning (i.e., elastic net, random forest) was used to analyze the data. Most participants (n = 634, 73.5%) found psychedelic use to have a positive influence on their quality of meditation. Twenty-eight variables showed significant zero-order associations with perceived benefits even following a correction. Elastic net had the best performance (R2 = .266) and was used to identify the most important features. Across 53 variables, the model found that greater use of psychedelics, intention setting during psychedelic use, agreeableness, and exposure to N,N-Dimethyltryptamine (N,N-DMT) were most likely to be associated with the perception that psychedelics benefit meditation practice. The results were consistent across several different approaches used to identify the most important variables (i.e., Shapley values, feature ablation). Results suggest that most meditators found psychedelic use to have a positive influence on their meditation practice, with: 1) regularity of psychedelic use, 2) the setting of intentions for psychedelic use, 3) having an agreeable personality, and 4) reported use of N,N-DMT being the most likely predictors of perceiving psychedelic use as beneficial. Longitudinal designs and randomized trials manipulating psychedelic use are needed to establish causality.

Navigating ambiguity: A novel neutrosophic cubic shapley normalized weighted Bonferroni Mean aggregation operator with application in the investment environment

The Neutrosophic Cubic Shapley Normalized Bonferroni (NC-SNWBM) method represents a cutting-edge approach to decision making theory, combining three distinct mathematical frameworks the neutrosophic cubic sets (NCS), Shapley values, and the Bonferroni aggregation operator. This innovative method addresses the challenges posed by uncertainty, vagueness, and imprecision in decision-making (DM) processes, offering a comprehensive and versatile tool for handling complex and dynamic scenarios. Neutrosophic cubic sets offers a strong platform to handle ambiguous and vague data due to three components Membership Grade (MG), Non-Membership Grade (NMG) and Indeterminancy Grade (IG) in data. By adding Shapley Fuzzy Measures (SFM), which come from cooperative game theory, distribute values among cooperative agents equally and to account for each agent's contributions to all potential coalitions. The Bonferroni aggregation operator—a statistical aggregative tool that regulates the likelihood of many types in error in statistical tests and the interdependence of the input arguments by allowing different values to parameters involved. These values are further improved by normalization in the framework of the NC-SNWBM approach in order to consider the various degrees of impact that agents exert in various circumstances. This operator is smoothly combined with normalized Shapley values and neutrosophic cubic sets in the NC-SNWBM approach to enable the aggregation of data with different levels of imprecision and uncertainty from various sources using NCS. The MG, NMG and IG connected to NCS are important elements of the NC-SNWBM approach. To evaluate each element's contribution to the overall value distribution SFM are used, and the Bonferroni aggregation operator maintains a careful balance between conservatism and significance. Together, these components provide a thorough framework that successfully tackles the problems caused by ambiguity, imprecision, and uncertainty in scenarios involving decision-making. The NC-SNWBM operator is applied to a numerical problem as an application in investment environment and sensitive and comparative analysis are conducted. The recommendation based on sensitive and comparative analysis proposed.

Concentration division for adsorption coefficient prediction using machine learning with Abraham descriptors: Data-splitting approach comparison and critical factors identification

Machine learning (ML) including Abraham descriptors from polyparameter linear free energy relationships (pp-LFERs) has been a popular method for the adsorption coefficient (Kd) prediction. However, Abraham descriptors from pp-LFERs are concentration-dependent and the significance of these descriptors can change over different adsorbate concentrations. Ignoring concentration effects on the adsorption process and Kd prediction will hinder the understanding of interactions among solutes, solvents and adsorbents at different equilibrium concentration (Ce) ranges. Therefore, our study first systematically investigated the concentration effects on micropollutant adsorption to carbon-based adsorbents using ML with Abraham descriptors. Concentration-selection approach, as a new data-splitting approach, divided the whole dataset according to the different Ce ranges. This concentration-selection approach performed better than the data-splitting approach used in previous studies. After the ML models were built in different Ce subsets, Shapley values were calculated to quantify input descriptor contributions. The results indicated specific surface area (BET) was the only critical factor when Ce was in the highest range. The importance of Abraham descriptors increased gradually when Ce decreased. Total pore volume (Vt) was a far less important feature than BET for Kd prediction. Critical factors identified at different Ce ranges for Kd prediction provide a guidance for novel carbon-based adsorbent design.

Thermodynamics and explainable machine learning assist in interpreting biodegradability of dissolved organic matter in sludge anaerobic digestion with thermal hydrolysis

Dissolved organic matter (DOM) is essential in biological treatment, yet its specific roles remain incompletely understood. This study introduces a machine learning (ML) framework to interpret DOM biodegradability in the anaerobic digestion (AD) of sludge, incorporating a thermodynamic indicator (λ). Ensemble models such as Xgboost and LightGBM achieved high accuracy (training: 0.90–0.98; testing: 0.75–0.85). The explainability of the ML models revealed that the features λ, measured m/z, nitrogen to carbon ratio (N/C), hydrogen to carbon ratio (H/C), and nominal oxidation state of carbon (NOSC) were significant formula features determining biodegradability. Shapley values further indicated that the biodegradable DOM were mostly formulas with λ lower than 0.03, measured m/z value higher than 600 Da, and N/C ratios higher than 0.2. This study suggests that a strategy based on ML and its explainability, considering formula features, particularly thermodynamic indicators, provides a novel approach for understanding and estimating the biodegradation of DOM.

Enablers of new business density: a comparison between developed and developing countries using deep learning and explainable AI

PurposeNew business density (NBD) is the ratio of the number of newly registered liability corporations to the working-age population per year. NBD is critical to assessing a country's business environment. The present work endeavors to discover and gauge the contribution of 28 potential socio-economic enablers of NBD for 2006–2021 across developed and developing economies separately and to make a comparative assessment between those two regions.Design/methodology/approachUsing World Bank data, the study first performs exploratory data analysis (EDA). Then, it deploys a deep learning (DL)-based regression framework by utilizing a deep neural network (DNN) to perform predictive modeling of NBD for developed and developing nations. Subsequently, we use two explainable artificial intelligence (XAI) techniques, Shapley values and a partial dependence plot, to unveil the influence patterns of chosen enablers. Finally, the results from the DL method are validated with the explainable boosting machine (EBM) method.FindingsThis research analyzes the role of 28 potential socio-economic enablers of NBD in developed and developing countries. This research finds that the NBD in developed countries is predominantly governed by the contribution of manufacturing and service sectors to GDP. In contrast, the propensity for research and development and ease of doing business control the NBD of developing nations. The research findings also indicate four common enablers – business disclosure, ease of doing business, employment in industry and startup procedures for developed and developing countries.Practical implicationsNBD is directly linked to any nation's economic affairs. Therefore, assessing the NBD enablers is of paramount significance for channelizing capital for new business formation. It will guide investment firms and entrepreneurs in discovering the factors that significantly impact the NBD dynamics across different regions of the globe. Entrepreneurs fraught with inevitable market uncertainties while developing a new idea into a successful new business can momentously benefit from the awareness of crucial NBD enablers, which can serve as a basis for business risk assessment.Originality/valueDL-based regression framework simultaneously caters to successful predictive modeling and model explanation for practical insights about NBD at the global level. It overcomes the limitations in the present literature that assume the NBD is country- and industry-specific, and factors of the NBD cannot be generalized globally. With DL-based regression and XAI methods, we prove our research hypothesis that NBD can be effectively assessed and compared with the help of global macro-level indicators. This research justifies the robustness of the findings by using the socio-economic data from the renowned data repository of the World Bank and by implementing the DL modeling with validation through the EBM method.

Visualization of explainable artificial intelligence for GeoAI

Shapley additive explanations are a widely used technique for explaining machine learning models. They can be applied to basically any type of model and provide both global and local explanations. While there are different plots available to visualize Shapley values, there is a lack of suitable visualization for geospatial use cases, resulting in the loss of the geospatial context in traditional plots. This study presents a concept for visualizing Shapley values in geospatial use cases and demonstrate its feasibility through an exemplary use case—predicting bike activity in a rental bike system. The visualizations show that visualizing Shapley values on geographic maps can provide valuable insights that are not visible in traditional plots for Shapley additive explanations. Geovisualizations are recommended for explaining machine learning models in geospatial applications or for extracting knowledge about real-world applications. Suitable visualizations for the considered use case are a proportional symbol map and a mapping of computed Voronoi values to the street network.

An inter-provincial coordinate model under Renewable Portfolio Standards policy based on tradable green certificate options trading

The current territorial management model under Renewable Portfolio Standards in China cannot fully develop renewable energy due to the mismatch between high energy-consumption regions and renewable energy-rich regions. Therefore, we proposed an inter-provincial coordinate model based on tradable green certificate options trading (CMTOT), which includes four components: (1) A dual-objective optimization model is established to minimize the RPS target realization cost and maximize the pollutant emission reduction based on Blacke-Scholes option pricing model. (2) The identification of buyer and seller provinces of tradable green certificate (TGC) options based on the above model. (3) An optimization model in buyers' or sellers' cooperation alliance is established. (4) The cost allocation models by Shapley value and Nucleolus are established to ensure the fairness in different scenarios. A case study of Tianjin, Shandong, Shanxi and Inner Mongolia shows that the CMTOT can effectively reduce the RPS target realization cost by 16.79 % and increase the pollutant emission reduction by 4.32 %. A sensitivity analysis confirms the robustness of the CMTOT. The results show that the CMTOT can both fully develop renewable energy and effectively reduce pollutant emission. Therefore, we proposed some policy recommendations to support the implementation of the CMTOT.

Resilience-based explainable reinforcement learning in chemical process safety

For future applications of artificial intelligence, namely reinforcement learning (RL), we develop a resilience-based explainable RL agent to make decisions about the activation of mitigation systems. The applied reinforcement learning algorithm is Deep Q-learning and the reward function is resilience. We investigate two explainable reinforcement learning methods, which are the decision tree, as a policy-explaining method, and the Shapley value as a state-explaining method.The policy can be visualized in the agent’s state space using a decision tree for better understanding. We compare the agent’s decision boundary with the runaway boundaries defined by runaway criteria, namely the divergence criterion and modified dynamic condition. Shapley value explains the contribution of the state variables on the behavior of the agent over time. The results show that the decisions of the artificial agent in a resilience-based mitigation system can be explained and can be presented in a transparent way.

Improving the explainability of autoencoder factors for commodities through forecast-based Shapley values

Autoencoders are dimension reduction models in the field of machine learning which can be thought of as a neural network counterpart of principal components analysis (PCA). Due to their flexibility and good performance, autoencoders have been recently used for estimating nonlinear factor models in finance. The main weakness of autoencoders is that the results are less explainable than those obtained with the PCA. In this paper, we propose the adoption of the Shapley value to improve the explainability of autoencoders in the context of nonlinear factor models. In particular, we measure the relevance of nonlinear latent factors using a forecast-based Shapley value approach that measures each latent factor’s contributions in determining the out-of-sample accuracy in factor-augmented models. Considering the interesting empirical instance of the commodity market, we identify the most relevant latent factors for each commodity based on their out-of-sample forecasting ability.

Exploring the mechanism of path-creating strategy for latecomers: a combined approach of econometrics and causal machine learning

In this paper, the traditional econometrics method and causal machine-learning method are combined to study the mechanism of a path-creating strategy of latecomers to influence latecomers’ catch-up performance. A total of 283 high-tech manufacturing enterprises listed on the Shanghai and Shenzhen stock exchanges from 2007 to 2019 were selected for the study. OLS linear regression model verifies that path-creating has a positive impact on latecomers’ technological catch-up performance, technological capability plays an intermediary role between path-creating and technology catch-up performance, technological innovation appropriability positively moderates the effect of path-creating on technological capability, and technological innovation cumulativeness negatively moderates the effect of path-creating on technological catch-up but positively moderate the effect of technological capability on catch-up performance. Through machine learning, on the one hand, a conclusion basically consistent with the linear regression model is obtained, but on the other hand, a more heterogeneous situation is presented. Through analyses of the individual treatment effect of a path-creating strategy of latecomers, the Shapley value graph shows the complex influence of different features on the treatment effect of the enterprise using the path-creating strategy. Through the decision tree, some more complex patterns are found. In addition, the decision tree model based on causal analysis can also assist enterprises in making strategic decisions.

DES-AS: Dynamic ensemble selection based on algorithm Shapley

Dynamic ensemble selection (DES) effectively improves aggregation performance by dynamically finding the most appropriate subset of classifiers for each query sample. The key component of this strategy is to define an appropriate criterion for evaluating the competence of the candidate classifiers. In this study, we present a new group-based criterion, synergy competence, to comprehensively measure the magnitude of the intricate synergy effect among classifiers. In addition, we introduce the algorithm Shapley, a variant of the Shapley value (SV) in the game theory, to measure the synergy competence of each classifier, which helps in the construction of a synergy-based DES system, called DES-AS. We argue that candidate classifiers with positive algorithm Shapley should be selected as they contribute to the ensemble. After selecting a subset of classifiers, the normalized algorithm Shapley is employed to calculate the voting weights for a cooperative final classification decision. The experimental analyses conducted on forty real-world datasets not only indicate that the synergy competence estimated by the algorithm Shapley is more applicable than classic group-based metrics, but also show that DES-AS is substantially more effective and robust than state-of-the-art models. The statistical test results also indicate that the estimated synergy competence reflects the real competence of each classifier better than the baseline models.

Channel Selection for Seizure Detection Based on Explainable AI With Shapley Values

Channel Selection for Seizure Detection Based on Explainable AI With Shapley Values

Predicting Long COVID in the National COVID Cohort Collaborative Using Super Learner: Cohort Study.

Postacute sequelae of COVID-19 (PASC), also known as long COVID, is a broad grouping of a range of long-term symptoms following acute COVID-19. These symptoms can occur across a range of biological systems, leading to challenges in determining risk factors for PASC and the causal etiology of this disorder. An understanding of characteristics that are predictive of future PASC is valuable, as this can inform the identification of high-risk individuals and future preventative efforts. However, current knowledge regarding PASC risk factors is limited. Using a sample of 55,257 patients (at a ratio of 1 patient with PASC to 4 matched controls) from the National COVID Cohort Collaborative, as part of the National Institutes of Health Long COVID Computational Challenge, we sought to predict individual risk of PASC diagnosis from a curated set of clinically informed covariates. The National COVID Cohort Collaborative includes electronic health records for more than 22 million patients from 84 sites across the United States. We predicted individual PASC status, given covariate information, using Super Learner (an ensemble machine learning algorithm also known as stacking) to learn the optimal combination of gradient boosting and random forest algorithms to maximize the area under the receiver operator curve. We evaluated variable importance (Shapley values) based on 3 levels: individual features, temporal windows, and clinical domains. We externally validated these findings using a holdout set of randomly selected study sites. We were able to predict individual PASC diagnoses accurately (area under the curve 0.874). The individual features of the length of observation period, number of health care interactions during acute COVID-19, and viral lower respiratory infection were the most predictive of subsequent PASC diagnosis. Temporally, we found that baseline characteristics were the most predictive of future PASC diagnosis, compared with characteristics immediately before, during, or after acute COVID-19. We found that the clinical domains of health care use, demographics or anthropometry, and respiratory factors were the most predictive of PASC diagnosis. The methods outlined here provide an open-source, applied example of using Super Learner to predict PASC status using electronic health record data, which can be replicated across a variety of settings. Across individual predictors and clinical domains, we consistently found that factors related to health care use were the strongest predictors of PASC diagnosis. This indicates that any observational studies using PASC diagnosis as a primary outcome must rigorously account for heterogeneous health care use. Our temporal findings support the hypothesis that clinicians may be able to accurately assess the risk of PASC in patients before acute COVID-19 diagnosis, which could improve early interventions and preventive care. Our findings also highlight the importance of respiratory characteristics in PASC risk assessment. RR2-10.1101/2023.07.27.23293272.

Age Group Discrimination via Free Handwriting Indicators.

Ageing is a physiological phenomenon associated with cognitive and functional decline which, in the long term, could hamper the execution of daily life activities and threaten both social and independent life. The onset of chronic diseases can intensify this process, increasing the risk of hospitalisation and admission to long term care. This represents a significant burden on public health and reduces the quality of life of those affected. Early detection of unhealthy decline is therefore key, but the similarity to normal ageing hinders its prompt screening. This study presents a first step towards the early screening of unhealthy ageing, based on an innovative instrumented ink pen to ecologically assess handwriting performance in different age groups: 40-59 (Group 1), 60-69 (Group 2) and 70+ (Group 3) years old. Raw handwriting data were collected from 60 healthy subjects and used to extract fourteen indicators related to gesture and tremor. The indicators were then used to discriminate between subjects of different age groups in three binary classification tasks, using a selection of machine learning algorithms. This approach produced remarkable results, particularly in the task of greatest interest, identifying subjects at the very beginning of the ageing process (Group 2) from elderly subjects (Group 3), achieving an accuracy of 97.5%, an F1 score of 97.44% and a ROC-AUC of 95%. Explainability of the model, facilitated by the analysis of the Shapley values of the learned indicators, revealed age-dependent sensitivity of handwriting and tremor-related indicators. The proposed method represents a promising solution for the early detection of abnormal signs of ageing, and is designed for the remote, non-invasive, unsupervised home monitoring, to improve the care of older adults.

Manifold-based Shapley explanations for high dimensional correlated features

Explainable artificial intelligence (XAI) holds significant importance in enhancing the reliability and transparency of network decision-making. SHapley Additive exPlanations (SHAP) is a game-theoretic approach for network interpretation, attributing confidence to inputs features to measure their importance. However, SHAP often relies on a flawed assumption that the model’s features are independent, leading to incorrect results when dealing with correlated features. In this paper, we introduce a novel manifold-based Shapley explanation method, termed Latent SHAP. Latent SHAP transforms high-dimensional data into low-dimensional manifolds to capture correlations among features. We compute Shapley values on the data manifold and devise three distinct gradient-based mapping methods to transfer them back to the high-dimensional space. Our primary objectives include: (1) correcting misinterpretations by SHAP in certain samples; (2) addressing the challenge of feature correlations in high-dimensional data interpretation; and (3) reducing algorithmic complexity through Manifold SHAP for application in complex network interpretations. Code is available at https://github.com/Teriri1999/Latent-SHAP.

Cooperative operation for multiple virtual power plants considering energy-carbon trading: A Nash bargaining model

With the development of the energy and carbon markets, it has become a trend for multiple virtual power plants (MVPP) that aggregate distributed resources from different regions to participate in market trading through cooperative operation. In order to study the energy-carbon interactions between VPPs and the supply-demand interactions for MVPP and users, this paper introduces a load aggregator (LA) and establishes a cooperative alliance consisting of MVPP and LA. Firstly, a carbon emission allowance allocation and transaction mechanism for MVPP is proposed based on the modified Shapley value method, which considers the impact of energy interactions within the alliance on the carbon emissions. Then, a cooperation model for the MVPP-LA alliance is established based on Nash bargaining theory, and it is decomposed into two subproblems of the maximal alliance benefits and the optimal benefit distribution. In terms of benefit distribution, this paper modifies the bargaining mechanism based on the contribution of each VPP to the alliance benefit. A dynamic penalty factor-based alternating direction method of multipliers (DP-ADMM) is used to solve the two subproblems in a distributed manner. Finally, the validity of the proposed trading mechanism, allocation method and solution algorithm are verified through several cases.

A novel framework for enhancing transparency in credit scoring: Leveraging Shapley values for interpretable credit scorecards.

Credit scorecards are essential tools for banks to assess the creditworthiness of loan applicants. While advanced machine learning models like XGBoost and random forest often outperform traditional logistic regression in predictive accuracy, their lack of interpretability hinders their adoption in practice. This study bridges the gap between research and practice by developing a novel framework for constructing interpretable credit scorecards using Shapley values. We apply this framework to two credit datasets, discretizing numerical variables and utilizing one-hot encoding to facilitate model development. Shapley values are then employed to derive credit scores for each predictor variable group in XGBoost, random forest, LightGBM, and CatBoost models. Our results demonstrate that this approach yields credit scorecards with interpretability comparable to logistic regression while maintaining superior predictive accuracy. This framework offers a practical and effective solution for credit practitioners seeking to leverage the power of advanced models without sacrificing transparency and regulatory compliance.