Overestimation Bias Research Articles

Enhancing intrusion detection in next-generation networks based on a multi-agent game-theoretic framework

<span lang="EN-US">With cyber threats becoming increasingly sophisticated, existing intrusion detection systems (IDS) in next generation networks (NGNs) are subjected to more false-positives and struggles to offer robust security feature, highlighting a critical need for more adaptive and reliable threat detection mechanisms. This research introduces a novel IDS that leverages a dueling deep Q-network (DQN) a reinforcement learning algorithm within game-theoretic framework simulating a multi-agent adversarial learning scenario to address these challenges. By employing a customized OpenAI Gym environment for realistic threat simulation and advanced dueling DQN mechanisms for reduced overestimation bias, the proposed scheme significantly enhances the adaptability and accuracy of intrusion detection. Comparative analysis against current state-of-the-art methods reveals that the proposed system achieves superior performance, with accuracy and F1-score improvements to 95.02% and 94.68%, respectively. These results highlight the potential scope of the proposed adaptive IDS to provide a robust defense against the dynamic threat landscape in NGNs.</span>

A Meta-Learning Approach to Mitigating the Estimation Bias of Q-Learning

It is a longstanding problem that Q-learning suffers from the overestimation bias. This issue originates from the fact that Q-learning uses the expectation of maximum Q-value to approximate the maximum expected Q-value. A number of algorithms, such as Double Q-learning, were proposed to address this problem by reducing the estimation of maximum Q-value, but this may lead to an underestimation bias. Note that this underestimation bias may have a larger performance penalty than the overestimation bias. Different from previous algorithms, this article studies this issue from a fresh perspective, i.e., meta-learning view, which leads to our Meta-Debias Q-learning. The main idea is to extract the maximum expected Q-value with meta-learning over multiple tasks to remove the estimation bias of maximum Q-value and help the agent choose the optimal action more accurately. However, there are two challenges: (1) How to automatically select suitable training tasks? (2) How to positively transfer the meta-knowledge from selected tasks to remove the estimation bias of maximum Q-value? To address the two challenges mentioned above, we quantify the similarity between the training tasks and the test task. This similarity enables us to select appropriate “partial” training tasks and helps the agent extract the maximum expected Q-value to remove the estimation bias. Extensive experiment results show that our Meta-Debias Q-learning outperforms SOTA baselines drastically in three evaluation indicators, i.e., maximum Q-value, policy, and reward. More specifically, our Meta-Debias Q-learning only underestimates \(1.2*10^{-3}\) than the maximum expected Q-value in the multi-armed bandit environment and only differs \(5.04\%-5\%=0.04\%\) than the optimal policy in the two states MDP environment. In addition, we compare the uniform weight and our similarity weight. Experiment results reveal fundamental insights into why our proposed algorithm outperforms in the maximum Q-value, policy, and reward.

Adaptive Multi-layer Attention Double Dueling Deep Q-Network for Muti-agent Reinforcement Learning

Abstract. In multi-agent fields, traditional muti-agent DQN methods often suffer from overestimation bias and overestimation of unimportant actions, especially when state-action Q-value differences are slight. To deal with such issue, we present an adaptive Multi-layer Attention Double Dueling Deep Q-Network (MAD-D3QN) model, aiming to improve decision-making accuracy in complex multi-agent environments. The proposed model utilizes two attention layers that dynamically calculate state value and action advantage weights, facilitating more precise Q-value estimation and reducing the common overestimation bias. Related experiments carried out in StarWar II scenarios show that the MAD-D3QN model obviously outperforms traditional methods (IQL,DQN), achieving higher decision efficiency and robustness. Our findings demonstrates that the MAD-D3QN framework not only promotes the state-of-the-art in multi-agent reinforcement learning but also provides potential applications in real-world cooperative tasks. Future research will delve into the integration of advanced multi-agent communication structures to further enhance model adaptability.

Assessment of coronary microvascular function with absolute coronary fl ow measurements

Abstract Background Reliable assessment of microvascular function is essential, both in research setting and clinical practice. New techniques, such as continuous coronary thermodilution, are very promising and could revolutionize the field. However, there is a need for proper validation before widespread use is possible. Purpose The goals of this study were: 1. to validate the potential of saline infusion to generate maximum hyperemia in vivo. 2. to validate absolute blood flow (ABF) measured with continuous coronary thermodilution at higher (40-50 ml/min) infusion speeds 3. To assess the safety of higher infusion speeds. Methods Fourteen closed-chest sheep underwent ABF measurements with increasing saline infusion speeds with and without adenosine. Another seven open-chest sheep underwent these measurements with epicardial doppler flow probes on both the left anterior descending artery and left circumflex coronary artery. Maximal hyperemia was defined as maximal flow after 45 seconds of proximal occlusion. The principles of laboratory animal care and all national regulations were followed. Results 20 ml/min of saline infusion induced lower hyperemic coronary flow (140 vs 191 ml/min, p=0.0165) and lower coronary flow reserve (CFR) (1.82 vs 3.21, p= <0.0001) than coronary occlusion. In contrast, 30 ml/min resulted in flow (196 vs 192 ml/min, p=0.8292) and CFR (2.77 vs 3.21, p=0.1107) that were not different from post-occlusion. There was a significant flow overestimation bias (Bland-Altman: Bias ± SD: -73.09 ± 30.52, 95% Limits of agreement: -132.9 to -13.27) with 40-50ml/min of saline. Occasionally, higher saline infusion rates resulted in ischemic ECG-changes and ventricular fibrillation (9.5% at 50 ml/min). Conclusions Continuous saline infusion of 30ml/min, but not 20 ml/min, induced maximal hyperemia. ABF measured with saline infusion speeds of 40-50 ml/min was not accurate nor safe, and should therefore be avoided.

Advanced hybrid neural network techniques for minimizing gas turbine emissions

Purpose Emissions have significant environmental impacts. Hence, minimizing emissions is essential. This study aims to use a hybrid neural network model to predict carbon monoxide (CO) and nitrogen oxide (NOx) emissions from gas turbines (GTs) to enhance emission prediction for GTs in predictive emissions monitoring systems (PEMS). Design/methodology/approach The hybrid model architecture combines convolutional neural networks (CNN) and bidirectional long-short-term memory (Bi-LSTM) networks called CNN-BiLSTM with modified extrinsic attention regression. Over five years, data from a GT power plant was uploaded to Google Colab, split into training and testing sets (80:20), and evaluated using test matrices. The model’s performance was benchmarked against state-of-the-art emissions prediction methodologies. Findings The model showed promising results for GT CO and NOx emissions. CO predictions had a slight underestimation bias of −0.01, with root mean-squared error (RMSE) of 0.064, mean absolute error (MAE) of 0.04 and R2 of 0.82. NOx predictions had an RMSE of 0.051, MAE of 0.036, R2 of 0.887 and a slight overestimation bias of +0.01. Research limitations/implications While the model demonstrates relative accuracy in CO emission predictions, there is potential for further improvement in future research. Practical implications Implementing the model in real-time PEMS and establishing a continuous feedback loop will ensure accuracy in real-world applications, enhance GT functioning and reduce emissions, fuel consumption and running costs. Social implications Accurate GT emissions predictions support stricter emission standards, promote sustainable development goals and ensure a healthier societal environment. Originality/value This paper presents a novel approach that integrates CNN and Bi-LSTM networks. It considers both spatial and temporal data to mitigate previous prediction shortcomings.

Investigating the Limitations of Multi‐Model Ensembling of Climate Model Outputs in Capturing Climate Extremes

ABSTRACTIn the context of climate change, the widespread practice of directly employing Multi‐Model Ensembles (MMEs) for projecting future climate extremes, without prior evaluation of MME performance in historical periods, remains underexplored. This research addresses this gap through a comprehensive analysis of ensemble means derived from CMIP6‐based models, including both simple and weighted averages of precipitation (SEMP and WEMP) and temperature (SEMT and WEMT) time series, as well as simple (SEME) and weighted (WEME) averages of extremes based on model‐by‐model analysis. The study evaluates the efficacy of MMEs in capturing mean annual values of ETCCDI indices over India for the period 1951–2014, utilising the IMD gridded data set as a reference. The results reveal that SEME and WEME consistently align closely with IMD data across various precipitation indices. At the same time, SEMP and WEMP consistently display underestimation biases ranging from 20% to 80% across all precipitation indices, except for CWD, where there is an overestimation bias. Moreover, SEMP and WEMP consistently underestimate CDD and overestimate CWD, indicating a systematic bias in these ensemble means, while WEME and SEME demonstrate satisfactory performance. SEMT and WEMT exhibit notable underestimation in temperature indices. In summary, adopting SEME and SEMT leads to a more robust assessment of precipitation and temperature extremes, respectively. These findings highlight the limitations of traditional MME methodologies in reproducing observed extreme precipitation events across various climatic zones in India, offering essential insights for refining climate models and improving the reliability of climate projections specific to the Indian subcontinent.

Body image at the trunk: An investigation into externally referenced width perception and picture mapping.

Body image is a conscious representation of the body, encompassing how our body feels to us. Body image can be measured in a variety of ways, including metric and depictive measures. This study sought to assess body image at the trunk by investigating, and comparing, a metric and depictive measure. Sixty-nine healthy participants estimated their thorax, waist, and hip width by externally referencing mechanical calipers. Participants were also asked to select the true image of their trunk from a random display of nine images containing the true image and incrementally shrunken or enlarged images. Participants demonstrated evidence of thorax and waist width overestimation in the width perception task, with no evidence for hip misestimation. For the picture mapping task, the majority of participants were inaccurate. In participants who were inaccurate, approximately equal proportions underestimated and overestimated their trunk width. The two tasks were found to be independent of each other. Distortions, or inaccuracies, were apparent in a metric measure, and inaccuracies also present in a depictive measure, of body image at the trunk for healthy participants. An overestimation bias was apparent in the metric, but not depictive, task. No relationship was found between tasks..

Evaluating the impact of multiple uncertainty shocks on China's airline stocks volatility: A novel joint quantile perspective

This study proposes a new joint quantile impulse response function (jQIRF) and applies that function to investigate the impact of multiple uncertainty shocks on the volatility of China's airline industry. The jQIRF not only allows one to examine the joint impact of multiple factors on the target variable; the impact of these factors on specific quantiles or states of the target variable can also be examined. The empirical results show that, compared to traditional IRF, the proposed jQIRF successfully reveals the positive impact of multiple uncertainties on the volatility of the airline industry and obtains a narrower confidence interval for IRF. The jQIRF also successfully corrects the overestimation bias caused by simple aggregation in generalized IRF. In addition, empirical results at different quantiles show the existence of a “leverage effect” in the impact of uncertainty on airline volatility. This means that, in more volatile market environments, the positive joint impact of uncertainties is stronger. However, research that has focused on individual airline stocks indicates that the airlines appear to be capable of implementing measures to stabilize stock volatility, thereby mitigating the negative impact of uncertainties on the airline industry. Overall, the proposed jQIRF and empirical conclusions in this paper help to more accurately assess the impact of multiple factors on the airline industry from a joint perspective. This ability is beneficial for both policymakers and investors.

Research on Supply Chain Optimization and Management Based on Deep Reinforcement Learning

This research introduces a groundbreaking approach to supply chain optimization and management, termed as Deep Reinforcement Learning based Supply Chain Optimization and Management (DRL-SCOM). At the core of this approach is the utilization of advancements in Deep Reinforcement Learning (DRL), specifically through the integration of Randomized Ensembled Double Q-learning (REDQ) and Trust Region Policy Optimization (TRPO). DRL-SCOM is designed to effectively tackle the inherent complexities and dynamic challenges that are characteristic of supply chain management. One of the key strengths of DRL-SCOM lies in its use of REDQ, which plays a crucial role in mitigating the overestimation bias commonly associated with traditional Q-learning methods. This results in more accurate value estimation and policy improvement, a critical factor in the effective management of supply chains. Additionally, the integration of TRPO into the framework brings the advantage of safe and stable policy updates. Such stability is vital for maintaining the robustness required in the fluctuating environment of supply chain operations. The combination of REDQ and TRPO in DRL-SCOM creates a powerful synergy. REDQ’s ensembled learning approach, when fused with TRPO’s trust-region method, enables the framework to efficiently navigate the complex and high-dimensional decision space typical of supply chains. This allows for real-time optimization of decisions while staying within operational constraints. The DRL-SCOM methodology shows significant potential in addressing various aspects of supply chain management, from demand forecasting and inventory management to logistics, adeptly handling the nonlinearities and uncertainties that are prevalent in these areas. Thus, the DRL-SCOM framework emerges as an innovative solution, pushing the frontiers of traditional supply chain management. It paves the way for a more agile, responsive, and intelligent system, equipped to adapt to changing market demands and operational challenges. This approach represents a significant stride towards transforming supply chain management into a more advanced, data-driven, and adaptive field.

The relationship between interpersonal distance preference and estimation accuracy in autism.

People naturally seek an interpersonal distance that feels comfortable, striking a balance between not being too close or too far from others until reaching a state of equilibrium. Previous studies on interpersonal distance preferences among autistic individuals have yielded inconsistent results. Some show a preference for greater distance, while others indicate a preference for shorter distances, or reveal higher variance in preferences among autistic individuals. In a related vein, previous studies have also investigated the way autistics accurately judge distance, and these studies have received inconsistent results, with some showing superior spatial abilities and others indicating biases in distance estimations. However, the link between distance estimation and preference has never been examined. To address this gap, our study measured interpersonal distance preferences and estimations and tested the correlation between the two factors. The results indicate greater variance among autistic people in both the preference of distance and the ability to estimate distance accurately, suggesting that inconsistencies in previous studies originate from greater individual differences among autistics. Furthermore, only among autistic individuals were interpersonal distance preference and estimation bias associated in a manner that violated equilibrium. Underestimation bias (judging others as closer than they are) was linked to a preference for closer proximity, while overestimation bias (judging others as further away) was associated with a preference for maintaining a greater distance. This connection suggests that biases in the estimation of interpersonal distance contribute to extreme preferences (being too close or too far away). Taken together, the findings suggest that biases in the estimation of interpersonal distance are associated with socially inappropriate distance preferences among autistics.

Actor-Critic With Synthesis Loss for Solving Approximation Biases.

Approximation biases of value functions are considered a key problem in reinforcement learning (RL). In particular, existing RL algorithms are hindered by overestimation and underestimation biases, i.e., value mismatching between RL's actual returns and action-value approximations limits the performance of RL algorithms. In this article, we first develop a new synthesis loss function for RL's action-value estimation integrating a regularization term and a modified "clipped double Q-learning" structure for solving overestimation and underestimation biases. To minimize the differences between action-value estimations and actual returns in RL, we develop a new discrepancy function to determine the type and magnitude of approximation biases. Then, two coefficients embedded in the synthesis loss are automatically tuned by minimizing the discrepancy function during training to minimize approximation biases. We further design a new actor-critic (AC) algorithm, named AC with synthesis loss (ACSL), by integrating the synthesis loss function and an error-controlled mechanism. Experimental results on continuous control tasks illustrate that the proposed ACSL algorithm outperforms other cutting-edge RL methods in many tasks and that the proposed synthesis loss function is easily implemented into other algorithms and significantly reduces approximation biases while improving performance. The proposed method can successfully handle many complex continuous control tasks and can greatly outperform other state-of-the-art algorithms on most tasks.

Integrating physics-based WRF atmospheric variables and machine learning algorithms to predict snowfall accumulation in Northeast United States

Accurate snowfall prediction is crucial for enhancing preparedness and resilience in the Northeast United States during winter weather events. This study introduces a novel approach that integrates Machine Learning (ML) models with atmospheric variables from the Weather Research and Forecasting (WRF) model to improve snowfall forecasts in the region. The significance lies in bridging the gap between physics-based Numerical Weather Prediction (NWP) models and the versatility of ML models, offering a promising advancement in winter storm predictions. Atmospheric variables related to 32 winter storms simulated by WRF, were used to feed Random Forest (RF) and Extreme Gradient Boosting (XGBoost) algorithms to predict 24 h accumulations of snowfall using the National Snowfall Analysis (NSA) product as reference. The comprehensive results revealed that the integrated approach provided more accurate snowfall prediction than the WRF and Air Force Weather Agency (AFWA) diagnostic, but faced challenges in the precision estimated by under-dispersion of the results. The WRF/XGBoost reduced the RMSE by 10.34 %, whereas WRF/RF reduced RMSE by 9.72 %, compared to WRF/AFWA. Similarly, WRF/XGBoost and WRF/RF increased the correlation coefficient by 12 % and 11 %. The most important variables in both ML algorithms were liquid water equivalent precipitation (LWE), snow ratio, wet-bulb temperature, temperature, and humidity at different heights, pressure tendency and wind speed, validating their ability to discern crucial factors influencing snowfall. Specific cases highlight the integrated approach’s effectiveness in addressing challenges faced by traditional diagnostics, including LWE overestimation and warm temperature bias. However, limitations emerge in capturing storms characterized by anomalous atmospheric conditions and high snowfall values, most likely attributed to underrepresentation in the training data. By highlighting strengths and acknowledging limitations, this integrated approach holds the potential for improved snowfall prediction for future storms in the region compared to the traditional approach.

EETS: An energy-efficient task scheduler in cloud computing based on improved DQN algorithm

The huge energy consumption of data centers in cloud computing leads to increased operating costs and high carbon emissions to the environment. Deep Reinforcement Learning (DRL) technology combines of deep learning and reinforcement learning, which has an obvious advantage in solving complex task scheduling problems. Deep Q Network(DQN)-based task scheduling has been employed for objective optimization. However, training the DQN algorithm may result in value overestimation, which can negatively impact the learning effectiveness. The replay buffer technique, while increasing sample utilization, does not distinguish between sample importance, resulting in limited utilization of valuable samples. This study proposes an enhanced task scheduling algorithm based on the DQN framework, which utilizes a more optimized Dueling-network architecture as well as Double DQN strategy to alleviate the overestimation bias and address the shortcomings of DQN. It also incorporates a prioritized experience replay technique to achieve importance sampling of experience data, which overcomes the problem of low utilization due to uniform sampling from replay memory. Based on these improved techniques, we developed an energy-efficient task scheduling algorithm called EETS (Energy-Efficient Task Scheduling). This algorithm automatically learns the optimal scheduling policy from historical data while interacting with the environment. Experimental results demonstrate that EETS exhibits faster convergence rates and higher rewards compared to both DQN and DDQN. In scheduling performance, EETS outperforms other baseline algorithms in key metrics, including energy consumption, average task response time, and average machine working time. Particularly, it has a significant advantage when handling large batches of tasks.

Addressing maximization bias in reinforcement learning with two-sample testing

Value-based reinforcement-learning algorithms have shown strong results in games, robotics, and other real-world applications. Overestimation bias is a known threat to those algorithms and can sometimes lead to dramatic performance decreases or even complete algorithmic failure. We frame the bias problem statistically and consider it an instance of estimating the maximum expected value (MEV) of a set of random variables. We propose the T-Estimator (TE) based on two-sample testing for the mean, that flexibly interpolates between over- and underestimation by adjusting the significance level of the underlying hypothesis tests. We also introduce a generalization, termed K-Estimator (KE), that obeys the same bias and variance bounds as the TE and relies on a nearly arbitrary kernel function. We introduce modifications of Q-Learning and the Bootstrapped Deep Q-Network (BDQN) using the TE and the KE, and prove convergence in the tabular setting. Furthermore, we propose an adaptive variant of the TE-based BDQN that dynamically adjusts the significance level to minimize the absolute estimation bias. All proposed estimators and algorithms are thoroughly tested and validated on diverse tasks and environments, illustrating the bias control and performance potential of the TE and KE.

The importance of spatial scale and vegetation complexity in woody species diversity and its relationship with remotely sensed variables

Plant species diversity is key to ecosystem functioning, but in recent decades anthropogenic activities have prompted an alarming decline in this community trait. Thus, developing strategies to understand diversity dynamics based on affordable and efficient remote sensing monitoring is essential, as well as examining the relevance of spatial scale and vegetation structural complexity to these dynamics. Here, we used two mathematical approaches to assess the relationship between tropical woody species diversity and spectral diversity in a human-modified landscape in two vegetation types differing in their degree of complexity. Vegetation complexity was measured through the fraction of species that concentrate different proportions of the cumulative importance value index. Species diversity was assessed using Hill numbers at three spatial scales, and metrics of spectral heterogeneity, vegetation indices, as well as raw data from Landsat 9 and Sentinel-2 sensors were calculated and analysed through general linear models (GLM) and Random Forest. Vegetation complexity emerged as an important variable in modelling species from remote sensing metrics, indicating the need to model species diversity by vegetation type rather than region. Hill numbers showed different relationships with remotely sensed metrics, in consistency with the scale-dependency of ecological processes on species diversity. Contrary to multiple previous reports, in our study, GLMs produced the best fits between Hill numbers of all orders and remotely sensed metrics. If we are to meet the need of conducting efficient and speedy woody species diversity monitoring globally, we propose modelling this diversity from remotely-sensed variables as an attractive strategy, so long as the intrinsic properties of each vegetation type are acknowledged to avoid under- or overestimation biases.

FireSync EMS: A Novel Mobile Application for Burn Surface Area Calculation.

The percent total body surface area burned is a critical determinant of the required level of care, initial management, and prognosis in burn patients. The current gold standard for estimating this measurement, the Lund-Browder chart, requires familiarity with its construction and may not be practical for use by first responders in the field. In this study, we present a novel burn surface area calculator mobile application developed for first responders and validate its accuracy. Infant, pediatric, and adult manikins were fabricated with eight simulated burns of varying sizes and distributions. 42 pre-clinical medical students and firefighters were tasked with estimating the total body surface area of each burn using both the FireSync-EMS app and Lund-Browder chart. Univariate analysis and mixed-effects linear regression modeling were performed to compare the accuracy of both methods in relation to user experience, manikin size, and burn size. FireSync-EMS significantly reduced overestimation bias (0.11%, SD 2.33 versus 0.91%, SD 4.12, p = 0.002), particularly for burns on child-size manikins (p < 0.001) and burns involving <10% (p = 0.005) and >20% (p = 0.030) total body surface area. Multivariable modeling revealed that the Lund-Browder chart was an independent determinant of the magnitude of estimation error, with a 1.19 times multiplicative effect relative to FireSyncEMS (p < 0.001). Participants overwhelmingly found FireSync-EMS easier, more intuitive, faster, and preferable (p < 0.001 for all). FireSync-EMS may be an easier, faster, and more accurate alternative to the Lund-Browder chart for estimation of the total body surface area burned.

Challenges of estimating treatment effects after a positive interim analysis

BackgroundThis research investigates why a beneficial treatment effect reported at the first interim analysis (IA) may diminish at a subsequent analysis (SA). We examined three challenges in interpreting treatment effects from randomized clinical trials (RCTs) after the first positive IA: overestimation bias; non-proportional hazards; and heterogeneity in recruitment. We investigate how a penalized estimation method can address overestimation bias, and discuss additional factors to consider when interpreting positive IA results. MethodsWe identified oncology RCTs reporting positive results at the initial IA and a SA for event-free (EFS) and overall survival (OS). We modeled: (1) the hazard ratio at IA (HRIA) versus its timing as measured by the information fraction (IF; i.e., events at IA versus total events sought); and (2), the ratio of HRIA to HRSA (rHR) versus the IF. This was repeated for HRIA adjusted for overestimation bias. Examples of the other two challenges were sought. ResultsAmongst 71 RCTs, HRIA were positively associated with the IF (slope: EFS 0.83, 95 % CI 0.44–1.22; OS 0.25, 95 % CI 0.10–0.41). HRIA tended to exaggerate HRSA, and more so the lower the IF (slope rHR versus IF: EFS 0.10, 95 % CI − 0.22 to 0.42; OS 0.26, 95 % CI 0.07–0.46). Adjusted HRIA did not exaggerate HRSA (slope rHR versus IF: EFS − 0.14, 95 % CI − 0.67 to 0.39; OS 0.02, 95 % CI − 0.26 to 0.30). Examples of two other challenges are shown. ConclusionOverestimation bias, non-proportional hazards, and heterogeneity in recruitment and other important treatments should be considered when communicating estimates of treatment effects from positive IAs.

Musculoskeletal spine modeling in large patient cohorts: how morphological individualization affects lumbar load estimation.

Introduction: Achieving an adequate level of detail is a crucial part of any modeling process. Thus, oversimplification of complex systems can lead to overestimation, underestimation, and general bias of effects, while elaborate models run the risk of losing validity due to the uncontrolled interaction of multiple influencing factors and error propagation. Methods: We used a validated pipeline for the automated generation of multi-body models of the trunk to create 279 models based on CT data from 93 patients to investigate how different degrees of individualization affect the observed effects of different morphological characteristics on lumbar loads. Specifically, individual parameters related to spinal morphology (thoracic kyphosis (TK), lumbar lordosis (LL), and torso height (TH)), as well as torso weight (TW) and distribution, were fully or partly considered in the respective models according to their degree of individualization, and the effect strengths of these parameters on spinal loading were compared between semi- and highly individualized models. T-distributed stochastic neighbor embedding (T-SNE) analysis was performed for overarching pattern recognition and multiple regression analyses to evaluate changes in occurring effects and significance. Results: We were able to identify significant effects (p < 0.05) of various morphological parameters on lumbar loads in models with different degrees of individualization. Torso weight and lumbar lordosis showed the strongest effects on compression (β ≈ 0.9) and anterior-posterior shear forces (β ≈ 0.7), respectively. We could further show that the effect strength of individual parameters tended to decrease if more individual characteristics were included in the models. Discussion: The induced variability due to model individualization could only partly be explained by simple morphological parameters. Our study shows that model simplification can lead to an emphasis on individual effects, which needs to be critically assessed with regard to in vivo complexity. At the same time, we demonstrated that individualized models representing a population-based cohort are still able to identify relevant influences on spinal loading while considering a variety of influencing factors and their interactions.

Sample-efficient reinforcement learning with knowledge-embedded hybrid model for optimal control of mining industry

The froth flotation process is a cost-effective and widely employed method for mineral separation. A core challenge is to maximize mineral recovery while maintaining a specified minimum concentrate grade. This requires precise control of aeration and slurry level and overcoming disturbances. The rapid advancements in reinforcement learning present a promising approach for controlling froth flotation. However, reinforcement learning suffers from sample inefficiency, which significantly hampers its application to real-world problems. In this research, we proposed an innovative sample-efficient model-based reinforcement learning algorithm to enhance flotation performance by directly leveraging the dynamics of the slurry phase. Our proposed algorithm involves the construction of a hybrid model, comprising a physical model and a data residual model, which effectively learns the underlying dynamics of the flotation process during the interaction between the reinforcement learning agent and process. The physical model integrates domain knowledge into the hybrid model to expedite the training process and enhance the interpretability and generalizability of the hybrid model. To capture uncertainty and address unmodeled aspects of the physical model, we employ an ensemble data residual model. Actor and critic are augmented with fuzzy representation modules based on fuzzy logic inference to improve the learning capacity of the networks and the multi-head critic is applied to reduce overestimation biases and enhance training stability. Case studies demonstrate that, compared to the baseline algorithm, our proposed algorithm requires merely 86.8% of the samples employed by conventional model-based reinforcement learning algorithm and improves the long-term return by at least 12.0%, underscoring the role of domain knowledge in facilitating efficient reinforcement learning.

An efficient and lightweight off-policy actor–critic reinforcement learning framework

In the framework of current off-policy actor–critic methods, the state–action pairs in an experience replay buffer (called historical behaviors) cannot be used to improve the policy, and the target network and the clipped double Q-learning techniques need to be used to evaluate the policy. The framework limits the policy learning capability in complex environments, and needs to maintain four critic networks. As a result, we propose an efficient and lightweight off-policy actor–critic (EL-AC) framework. In the policy improvement, an efficient off-policy likelihood ratio policy gradient algorithm with historical behaviors reusing (PG-HBR) is proposed, which promotes the agent to learn an approximately optimal policy by using the historical behaviors. Moreover, a theoretically interpretable universal critic network is designed. It can approximate the action-value and the state-value functions simultaneously, so as to obtain the advantage function in PG-HBR. In the policy evaluation, we develop the algorithms of low-pass filtering for target state-values and adaptive controlling algorithm for overestimation bias, which can evaluate the policy efficiently and accurately using only one universal critic network. Extensive evaluation results indicate that EL-AC outperforms the state-of-the-art off-policy actor–critic methods in terms of approximately optimal policy learning and neural network storage space occupation, and it is more suitable for policy learning in complex environments.