Underestimation Bias Research Articles

A Meta-Learning Approach to Mitigating the Estimation Bias of Q-Learning

It is a longstanding problem that Q-learning suffers from the overestimation bias. This issue originates from the fact that Q-learning uses the expectation of maximum Q-value to approximate the maximum expected Q-value. A number of algorithms, such as Double Q-learning, were proposed to address this problem by reducing the estimation of maximum Q-value, but this may lead to an underestimation bias. Note that this underestimation bias may have a larger performance penalty than the overestimation bias. Different from previous algorithms, this article studies this issue from a fresh perspective, i.e., meta-learning view, which leads to our Meta-Debias Q-learning. The main idea is to extract the maximum expected Q-value with meta-learning over multiple tasks to remove the estimation bias of maximum Q-value and help the agent choose the optimal action more accurately. However, there are two challenges: (1) How to automatically select suitable training tasks? (2) How to positively transfer the meta-knowledge from selected tasks to remove the estimation bias of maximum Q-value? To address the two challenges mentioned above, we quantify the similarity between the training tasks and the test task. This similarity enables us to select appropriate “partial” training tasks and helps the agent extract the maximum expected Q-value to remove the estimation bias. Extensive experiment results show that our Meta-Debias Q-learning outperforms SOTA baselines drastically in three evaluation indicators, i.e., maximum Q-value, policy, and reward. More specifically, our Meta-Debias Q-learning only underestimates \(1.2*10^{-3}\) than the maximum expected Q-value in the multi-armed bandit environment and only differs \(5.04\%-5\%=0.04\%\) than the optimal policy in the two states MDP environment. In addition, we compare the uniform weight and our similarity weight. Experiment results reveal fundamental insights into why our proposed algorithm outperforms in the maximum Q-value, policy, and reward.

Small sample adjustment for inference without assuming orthogonality in a mixed model for repeated measures analysis

ABSTRACT The mixed model for repeated measures (MMRM) analysis is sometimes used as a primary statistical analysis for a longitudinal randomized clinical trial. When the MMRM analysis is implemented in ordinary statistical software, the standard error of the treatment effect is estimated by assuming orthogonality between the fixed effects and covariance parameters, based on the characteristics of the normal distribution. However, orthogonality does not hold unless the normality assumption of the error distribution holds, and/or the missing data are derived from the missing completely at random structure. Therefore, assuming orthogonality in the MMRM analysis is not preferable. However, without the assumption of orthogonality, the small-sample bias in the standard error of the treatment effect is significant. Nonetheless, there is no method to improve small-sample performance. Furthermore, there is no software that can easily implement inferences on treatment effects without assuming orthogonality. Hence, we propose two small-sample adjustment methods inflating standard errors that are reasonable in ideal situations and achieve empirical conservatism even in general situations. We also provide an R package to implement these inference processes. The simulation results show that one of the proposed small-sample adjustment methods performs particularly well in terms of underestimation bias of standard errors; consequently, the proposed method is recommended. When using the MMRM analysis, our proposed method is recommended if the sample size is not large and between-group heteroscedasticity is expected.

Advanced hybrid neural network techniques for minimizing gas turbine emissions

Purpose Emissions have significant environmental impacts. Hence, minimizing emissions is essential. This study aims to use a hybrid neural network model to predict carbon monoxide (CO) and nitrogen oxide (NOx) emissions from gas turbines (GTs) to enhance emission prediction for GTs in predictive emissions monitoring systems (PEMS). Design/methodology/approach The hybrid model architecture combines convolutional neural networks (CNN) and bidirectional long-short-term memory (Bi-LSTM) networks called CNN-BiLSTM with modified extrinsic attention regression. Over five years, data from a GT power plant was uploaded to Google Colab, split into training and testing sets (80:20), and evaluated using test matrices. The model’s performance was benchmarked against state-of-the-art emissions prediction methodologies. Findings The model showed promising results for GT CO and NOx emissions. CO predictions had a slight underestimation bias of −0.01, with root mean-squared error (RMSE) of 0.064, mean absolute error (MAE) of 0.04 and R2 of 0.82. NOx predictions had an RMSE of 0.051, MAE of 0.036, R2 of 0.887 and a slight overestimation bias of +0.01. Research limitations/implications While the model demonstrates relative accuracy in CO emission predictions, there is potential for further improvement in future research. Practical implications Implementing the model in real-time PEMS and establishing a continuous feedback loop will ensure accuracy in real-world applications, enhance GT functioning and reduce emissions, fuel consumption and running costs. Social implications Accurate GT emissions predictions support stricter emission standards, promote sustainable development goals and ensure a healthier societal environment. Originality/value This paper presents a novel approach that integrates CNN and Bi-LSTM networks. It considers both spatial and temporal data to mitigate previous prediction shortcomings.

Application of Sentinel-2 Level-2A images for monitoring water surface in reservoirs in the semiarid region of Pernambuco — Brazil

Remote sensing techniques offer effective and efficient alternatives for observing the spatiotemporal dynamics of surface water in reservoirs. This paper aimed to analyze the applicability of Sentinel-2 Level-2A satellite images from 2016 to 2024 for mapping and monitoring the extent of water surfaces in reservoirs in the Sertão region of Pernambuco state. An automatic, unsupervised, and non-parametric algorithm was employed, combining water indices with reflectance bands of optical images to identify water pixels. The results were compared with two datasets: in situ monitoring and MapBiomas. Issues with optical images affected by clouds over the reservoir and errors in classifying water pixels were noted. Generally, the algorithm tended to underestimate the extent of the water surface due to difficulty detecting water pixels at the edges of the reservoirs. To mitigate this issue, an artificial neural network (ANN) was applied to correct the underestimation bias. The bias correction improved the performance of the metrics when the size and representativeness of the calibration sample were sufficient for training and building the ANN model.

Developing a brief older adults' physical activity questionnaire.

This study aimed to develop and evaluate the Brief Older Adults' Physical Activity Questionnaire (BOPAQ), which was designed to quickly assess moderate-to-vigorous physical activity (MVPA) in community-dwelling older adults. We used a cross-sectional study design involving 165 older participants. The BOPAQ calculated weekly MVPA duration based on two questions regarding the number of days per week engaged in MVPA and the daily duration of activity. Validity was assessed by correlating the MVPA durations derived from the BOPAQ with those obtained from the ActiGraph and International Physical Activity Questionnaire short form. Reliability was evaluated using the intraclass correlation coefficient, and measurement errors were analyzed using Bland-Altman plots. The BOPAQ reasonably correlated with accelerometer-based MVPA (rho = 0.297) and showed good test-retest reliability (intraclass correlation coefficient of 0.78, 95% CI 0.64-0.87). In contrast, the correlation between the International Physical Activity Questionnaire short form and accelerometer-based MVPA was poor (rho = 0.139). The cut-off value for the BOPAQ was set to identify participants engaging in <150 min of objectively measured physical activity per week, corresponding to the 150-min threshold. However, the area under the curve in the receiver operating characteristic analyses was not significantly high (0.601, 95% CI 0.514-0.688). The Bland-Altman plots showed an underestimation bias of 51.72 min/week (95% CI 1.61-101.84) and showed heteroscedasticity. Despite some measurement errors, the BOPAQ is an available tool for assessing MVPA in community-dwelling older adults. Geriatr Gerontol Int 2024; ••: ••-••.

Quantitation of mitral regurgitation using positron emission tomography

BackgroundCardiac positron emission tomography (PET) offers non-invasive assessment of perfusion and left ventricular (LV) function from a single dynamic scan. However, no prior assessment of mitral regurgitation severity by PET has been presented. Application of indicator dilution techniques and gated image analyses to PET data enables calculation of forward stroke volume and total LV stroke volume. We aimed to evaluate a combination of these methods for measurement of regurgitant volume (RegVol) and fraction (RegF) using dynamic 15O-water and 11C-acetate PET in comparison to cardiovascular magnetic resonance (CMR).ResultsTwenty-one patients with severe primary mitral valve regurgitation underwent same-day dynamic PET examinations (15O-water and 11C-acetate) and CMR. PET data were reconstructed into dynamic series with short time frames during the first pass, gated 15O-water blood pool images, and gated 11C-acetate myocardial uptake images. PET-based RegVol and RegF correlated strongly with CMR (RegVol: 15O-water r = 0.94, 11C-acetate r = 0.91 and RegF: 15O-water r = 0.88, 11C-acetate r = 0.84, p < 0.001). A systematic underestimation (bias) was found for PET (RegVol: 15O-water − 11 ± 13 mL, p = 0.002, 11C-acetate − 28 ± 16 mL, p < 0.001 and RegF: 15O-water − 4 ± 6%, p = 0.01, 11C-acetate − 10 ± 7%, p < 0.001). PET measurements in patients were compared to healthy volunteers (n = 18). Mean RegVol and RegF was significantly lower in healthy volunteers compared to patients for both tracers. The accuracy of diagnosing moderately elevated regurgitant volume (> 30mL) was 95% for 15O-water and 92% for 11C-acetate.ConclusionsLV regurgitation severity quantified using cardiac PET correlated with CMR and showed high accuracy for discriminating patients from healthy volunteers.

The relationship between interpersonal distance preference and estimation accuracy in autism.

People naturally seek an interpersonal distance that feels comfortable, striking a balance between not being too close or too far from others until reaching a state of equilibrium. Previous studies on interpersonal distance preferences among autistic individuals have yielded inconsistent results. Some show a preference for greater distance, while others indicate a preference for shorter distances, or reveal higher variance in preferences among autistic individuals. In a related vein, previous studies have also investigated the way autistics accurately judge distance, and these studies have received inconsistent results, with some showing superior spatial abilities and others indicating biases in distance estimations. However, the link between distance estimation and preference has never been examined. To address this gap, our study measured interpersonal distance preferences and estimations and tested the correlation between the two factors. The results indicate greater variance among autistic people in both the preference of distance and the ability to estimate distance accurately, suggesting that inconsistencies in previous studies originate from greater individual differences among autistics. Furthermore, only among autistic individuals were interpersonal distance preference and estimation bias associated in a manner that violated equilibrium. Underestimation bias (judging others as closer than they are) was linked to a preference for closer proximity, while overestimation bias (judging others as further away) was associated with a preference for maintaining a greater distance. This connection suggests that biases in the estimation of interpersonal distance contribute to extreme preferences (being too close or too far away). Taken together, the findings suggest that biases in the estimation of interpersonal distance are associated with socially inappropriate distance preferences among autistics.

Mapping every adult baobab (Adansonia digitata L.) across the Sahel and relationships to rural livelihoods.

The baobab tree (Adansonia digitata L.) is an integral part of rural livelihoods throughout the African continent. However, the combined effects of climate change and increasing global demand for baobab products are currently exerting pressure on the sustainable utilization of these resources. Here we use sub-metre-resolution satellite imagery to identify the presence of nearly 2.8 million (underestimation bias 27.1%) baobab trees in the Sahel, a dryland region of 2.4 million km2. This achievement is considered an essential step towards an improved management and monitoring system of valuable woody species. Using Senegal as a case country, we find that 94% of rural buildings have at least one baobab tree in their immediate surroundings and that the abundance of baobabs is associated with a higher likelihood of people consuming a highly nutritious food group: dark green leafy vegetables. The generated database showcases the feasibility of mapping the location of single tree species at a sub-continental scale, providing vital information in times when deforestation and climate change cause the extinction of numerous tree species.

Musculoskeletal spine modeling in large patient cohorts: how morphological individualization affects lumbar load estimation.

Introduction: Achieving an adequate level of detail is a crucial part of any modeling process. Thus, oversimplification of complex systems can lead to overestimation, underestimation, and general bias of effects, while elaborate models run the risk of losing validity due to the uncontrolled interaction of multiple influencing factors and error propagation. Methods: We used a validated pipeline for the automated generation of multi-body models of the trunk to create 279 models based on CT data from 93 patients to investigate how different degrees of individualization affect the observed effects of different morphological characteristics on lumbar loads. Specifically, individual parameters related to spinal morphology (thoracic kyphosis (TK), lumbar lordosis (LL), and torso height (TH)), as well as torso weight (TW) and distribution, were fully or partly considered in the respective models according to their degree of individualization, and the effect strengths of these parameters on spinal loading were compared between semi- and highly individualized models. T-distributed stochastic neighbor embedding (T-SNE) analysis was performed for overarching pattern recognition and multiple regression analyses to evaluate changes in occurring effects and significance. Results: We were able to identify significant effects (p < 0.05) of various morphological parameters on lumbar loads in models with different degrees of individualization. Torso weight and lumbar lordosis showed the strongest effects on compression (β ≈ 0.9) and anterior-posterior shear forces (β ≈ 0.7), respectively. We could further show that the effect strength of individual parameters tended to decrease if more individual characteristics were included in the models. Discussion: The induced variability due to model individualization could only partly be explained by simple morphological parameters. Our study shows that model simplification can lead to an emphasis on individual effects, which needs to be critically assessed with regard to in vivo complexity. At the same time, we demonstrated that individualized models representing a population-based cohort are still able to identify relevant influences on spinal loading while considering a variety of influencing factors and their interactions.

Incidence of Carotid Blowout Syndrome in Patients with Head and Neck Cancer after Radiation Therapy: A Cohort Study.

Carotid blowout syndrome (CBS) is a rare yet life-threatening complication that occurs after radiation therapy (RT). This study aimed to determine the incidence of CBS in patients with head and neck cancer (HNC) undergoing contemporary RT and to explore potential discrepancies in the risk of CBS between nasopharyngeal cancer (NPC) and non-NPC patients. A total of 1084 patients with HNC who underwent RT between 2013 and 2023 were included in the study. All patients were under regular follow-ups at the radio-oncology department, and underwent annual contrast-enhanced computed tomography and/or magnetic resonance imaging for cancer recurrence surveillance. Experienced neuroradiologists and vascular neurologists reviewed the recruited patients' images. Patients were further referred to the neurology department for radiation vasculopathy evaluation. The primary outcome of this study was CBS. Patients were categorized into NPC and non-NPC groups and survival analysis was employed to compare the CBS risk between the two groups. A review of the literature on CBS incidence was also conducted. Among the enrolled patients, the incidence of CBS in the HNC, NPC, and non-NPC groups was 0.8%, 0.9%, and 0.7%, respectively. Kaplan-Meier analysis revealed no significant difference between the NPC and non-NPC groups (p = 0.34). Combining the findings for our cohort with those of previous studies revealed that the cumulative incidence of CBS in patients with HNC is 5% (95% CI = 3-7%) after both surgery and RT, 4% (95% CI = 2-6%) after surgery alone, and 5% (95% CI = 3-7%) after RT alone. Our findings indicate a low incidence of CBS in patients with HNC undergoing contemporary RT. Patients with NPC may have a CBS risk close to that of non-NPC patients. However, the low incidence of CBS could be a potentially cause of selection bias and underestimation bias.

A rapid monitoring protocol to estimate unharvested corn biomass in waterfowl impoundments

AbstractConservation planners use bioenergetic models to develop habitat objectives that satisfy energetic demands of waterfowl during nonbreeding periods. In turn, natural resource managers should estimate yield and availability of natural and cultivated waterfowl forage to monitor contributions to objectives and support adaptive resource management. Because bioenergetic models are particularly sensitive to unharvested flooded croplands, we developed a rapid methodology to estimate biomass of unharvested flooded corn (Zea mays) and tested our methodology in impounded corn fields planted and flooded in western Tennessee during autumn and winter of 2019–2021. We evaluated accuracy of our rapid assessment method and conducted simulations to assess variance‐bias trade‐offs relative to sample size. Rapid assessments lasted 20 min ± 10 minutes per field. Our rapid assessment method underestimated number of kernels per ear by 2.6% ± 0.5%. After adjusting for underestimation bias, corn biomass across all surveys was 5,500 kg/ha ± 250 kg/ha, which is similar yield to previous literature from waterfowl impoundment fields. Sampling 15 ears per field allowed field biomass to be estimated within acceptable accuracy (i.e., variance [SE] of mean percent error &lt;1.8%). We recommend our rapid yield assessment method be incorporated into habitat monitoring protocols to efficiently and precisely estimate corn biomass for improved conservation planning initiatives. Additionally, our rapid assessment method allows managers to monitor field yield rapidly, enabling estimates of corn depletion and availability throughout the nonbreeding season for waterfowl to support management decisions.

Action Candidate Driven Clipped Double Q-Learning for Discrete and Continuous Action Tasks.

Double Q-learning is a popular reinforcement learning algorithm in Markov decision process (MDP) problems. Clipped double Q-learning, as an effective variant of double Q-learning, employs the clipped double estimator to approximate the maximum expected action value. Due to the underestimation bias of the clipped double estimator, the performance of clipped Double Q-learning may be degraded in some stochastic environments. In this article, in order to reduce the underestimation bias, we propose an action candidate-based clipped double estimator (AC-CDE) for Double Q-learning. Specifically, we first select a set of elite action candidates with high action values from one set of estimators. Then, among these candidates, we choose the highest valued action from the other set of estimators. Finally, we use the maximum value in the second set of estimators to clip the action value of the chosen action in the first set of estimators and the clipped value is used for approximating the maximum expected action value. Theoretically, the underestimation bias in our clipped Double Q-learning decays monotonically as the number of action candidates decreases. Moreover, the number of action candidates controls the tradeoff between the overestimation and underestimation biases. In addition, we also extend our clipped Double Q-learning to continuous action tasks via approximating the elite continuous action candidates. We empirically verify that our algorithm can more accurately estimate the maximum expected action value on some toy environments and yield good performance on several benchmark problems. Code is available at https://github.com/Jiang-HB/ac_CDQ.

Parameter-Free Reduction of the Estimation Bias in Deep Reinforcement Learning for Deterministic Policy Gradients

Approximation of the value functions in value-based deep reinforcement learning induces overestimation bias, resulting in suboptimal policies. We show that when the reinforcement signals received by the agents have a high variance, deep actor-critic approaches that overcome the overestimation bias lead to a substantial underestimation bias. We first address the detrimental issues in the existing approaches that aim to overcome such underestimation error. Then, through extensive statistical analysis, we introduce a novel, parameter-free Deep Q-learning variant to reduce this underestimation bias in deterministic policy gradients. By sampling the weights of a linear combination of two approximate critics from a highly shrunk estimation bias interval, our Q-value update rule is not affected by the variance of the rewards received by the agents throughout learning. We test the performance of the introduced improvement on a set of MuJoCo and Box2D continuous control tasks and demonstrate that it outperforms the existing approaches and improves the baseline actor-critic algorithm in most of the environments tested.

Controlling underestimation bias in reinforcement learning via minmax operation

Obtaining the accurate value estimation and reducing the estimation bias are the key issues in reinforcement learning. However, current methods that address the overestimation problem tend to introduce underestimation, which face a challenge of precise decision-making in many fields. To address this issue, we conduct a theoretical analysis of the underestimation bias and propose the minmax operation, which allow for flexible control of the estimation bias. Specifically, we select the maximum value of each action from multiple parallel state-action networks to create a new state-action value sequence. Then, a minimum value is selected to obtain more accurate value estimations. Moreover, based on the minmax operation, we propose two novel algorithms by combining Deep Q-Network (DQN) and Double DQN (DDQN), named minmax-DQN and minmax-DDQN. Meanwhile, we conduct theoretical analyses of the estimation bias and variance caused by our proposed minmax operation, which show that this operation significantly improves both underestimation and overestimation biases and leads to the unbiased estimation. Furthermore, the variance is also reduced, which is helpful to improve the network training stability. Finally, we conduct numerous comparative experiments in various environments, which empirically demonstrate the superiority of our method.

Avaliação visual do índice de área foliar em campos de café (Coffea arabica L.)

ABSTRACT The application of leaf area index (LAI) in coffee crop management depends on the availability of methodologies for proper estimation. The objective of this study was to develop a methodology for the visual assessment of LAI in coffee fields and to establish a protocol for training, evaluation, and feedback for evaluators. Four rounds of LAI measurements were conducted using visual estimates, two instruments (LAI 2200-C and AccuPAR LP-80), and defoliation of coffee hedgerows in Poás, Costa Rica. In each round, five workers visually estimated the LAI values on two occasions separated by 15 days, and feedback reinforcement was provided to each worker at the end of each round. Visual assessments showed high repeatability and reproducibility and the estimates were adjusted to the linear regression model in most cases. Evaluators improved their capacity to visually assess the LAI throughout the rounds, as the value of R2increased consistently for most workers, with values as high as 0.87. Instrumentation evaluation of LAI produced R2values of 0.5-0.6, with significant underestimation bias. The performance of the different methods is discussed in the context of widely spaced hedgerows. The proposed visual methodology constitutes a statistically sound, rapid, simple, and reliable method for determining the LAI of coffee fields to aid in decision-making for crop management.

Survivorship of discarded cuckoo ray in bottom trawl fisheries in the northern Bay of Biscay, Southern Celtic and Irish Seas

Estimating the survivorship of discarded fish is often crucial for stock assessment and resultant management of fisheries resources. In order to assess the survivorship of cuckoo ray (Leucoraja naevus) caught in commercial bottom trawl fisheries, experiments involving French and Irish fisheries were conducted. These experiments were particularly challenging considering the fisheries are offshore. Data from these experiments were analysed jointly in this study. Results from monitoring of individuals held in captivity in onshore facilities and observations of vitality status were combined to produce estimates of long-term survival rates across fishing trips in all four seasons. These rates varied greatly between fishing trips and ranged between 3.6% and 26%. Based on indications that the observed mortality may be at least partly attributable to the conditions of captivity, we propose an approach to reduce the underestimation bias when estimating discard survival. This approach produced higher bounds for the estimated discard survival rates. The estimated survival rates are lower than those of other skate species in the North-East Atlantic, suggesting the cuckoo ray is less resilient to trawl-and-release, although previous studies were conducted in shallower water and with shorter haul duration.

Explorer-Actor-Critic: Better actors for deep reinforcement learning

Actor-critic deep reinforcement learning methods have demonstrated significant performance in many challenging decision-making and control tasks, but also suffer from high sample complexity and overestimation bias. Current researches focus on using underestimation to balance overestimation and reducing bias through ensemble learning, but introducing underestimation bias and excessive network costs. In this paper, we first analyze the effect of action selection policy on estimation bias. Then, we propose the Explorer-Actor-Critic (EAC) method that gives a more conservative objective for the actor to reduce overestimation, introduces a learnable explorer to improve exploration ability, and uses an action mixing mechanism to mitigate experience distribution bias. Furthermore, we apply the EAC method to TD3 and SAC and verify its effectiveness through extensive comparison and ablation experiments. Our algorithm not only outperforms state-of-the-art algorithms, but also is compatible with other actor-critic methods.

Scattered tree death contributes to substantial forest loss in California

In recent years, large-scale tree mortality events linked to global change have occurred around the world. Current forest monitoring methods are crucial for identifying mortality hotspots, but systematic assessments of isolated or scattered dead trees over large areas are needed to reduce uncertainty on the actual extent of tree mortality. Here, we mapped individual dead trees in California using sub-meter resolution aerial photographs from 2020 and deep learning-based dead tree detection. We identified 91.4 million dead trees over 27.8 million hectares of vegetated areas (16.7-24.7% underestimation bias when compared to field data). Among these, a total of 19.5 million dead trees appeared isolated, and 60% of all dead trees occurred in small groups ( ≤ 3 dead trees within a 30 × 30 m grid), which is largely undetected by other state-level monitoring methods. The widespread mortality of individual trees impacts the carbon budget and sequestration capacity of California forests and can be considered a threat to forest health and a fuel source for future wildfires.

Quantitative metrics of the LV trabeculated layer by cardiac CT and cardiac MRI in patients with suspected noncompaction cardiomyopathy.

To compare cardiac computed tomography (CCT) and cardiac magnetic resonance (CMR) for the quantitative assessment of the left ventricular (LV) trabeculated layer in patients with suspected noncompaction cardiomyopathy (NCCM). Subjects with LV excessive trabeculation who underwent both CMR and CCT imaging as part of the prospective international multicenter NONCOMPACT clinical study were included. For each subject, short-axis CCT and CMR slices were matched. Four quantitative metrics were estimated: 1D noncompacted-to-compacted ratio (NCC), trabecular-to-myocardial area ratio (TMA), trabecular-to-endocardial cavity area ratio (TCA), and trabecular-to-myocardial volume ratio (TMV). In 20 subjects, end-diastolic and mid-diastolic CCT images were compared for the quantification of the trabeculated layer. Relationships between the metrics were investigated using linear regression models and Bland-Altman analyses. Forty-eight subjects (49.9 ± 12.8years; 28 female) were included in this study. NCC was moderately correlated (r = 0.62), TMA and TMV were strongly correlated (r = 0.78 and 0.78), and TCA had excellent correlation (r = 0.92) between CMR and CCT, with an underestimation bias from CCT of 0.3 units, and 5.1, 4.8, and 5.4 percent-points for the 4 metrics, respectively. TMA, TCA, and TMV had excellent correlations (r = 0.93, 0.96, 0.94) and low biases (- 3.8, 0.8, - 3.8 percent-points) between the end-diastolic and mid-diastolic CCT images. TMA, TCA, and TMV metrics of the LV trabeculated layer in patients with suspected NCCM demonstrated high concordance between CCT and CMR images. TMA and TCA were highly reproducible and demonstrated minimal differences between mid-diastolic and end-diastolic CCT images. The results indicate similarity of CCT to CMR for quantifying the LV trabeculated layer, and the small differences in quantification between end-diastole and mid-diastole demonstrate the potential for quantifying the LV trabeculated layer from clinically performed coronary CT angiograms. • Data on cardiac CT for quantifying the left ventricular trabeculated layer are limited. • Cardiac CT yielded highly reproducible metrics of the left ventricular trabeculated layer that correlated well with metrics defined by cardiac MR. • Cardiac CT appears to be equivalent to cardiac MR for the quantification of the left ventricular trabeculated layer.

Q-learning with heterogeneous update strategy

A variety of algorithms has been proposed to mitigate the overestimation bias of Q-learning. These algorithms reduce the estimation of maximum Q-value, i.e., homogeneous update. As a result, some of these algorithms such as Double Q-learning suffer from the underestimation bias. Different from these algorithms, this paper proposes a heterogeneous update idea. It aims to enlarge the normalized gap between Q-value corresponding to the optimal action and that corresponding to the other actions. Based on heterogeneous update, we design HetUp Q-learning. More specifically, HetUp Q-learning increases the normalized gap by overestimating Q-value corresponding to the optimal action and underestimating Q-value corresponding to the other actions. However, one limitation is that our HetUp Q-learning takes the optimal action as input to decide whether a state-action pair should be overestimated or underestimated. To address this challenge, we apply a softmax strategy to estimate the optimal action and obtain HetUpSoft Q-learning. We also extend HetUpSoft Q-learning to HetUpSoft DQN for high-dimensional environments. Extensive experiment results show that our proposed methods outperform SOTA baselines drastically in different settings. In particular, HetUpSoft DQN improves the average score per episode over SOTA baselines by at least 55.49% and 32.26% in the Pixelcopter and Breakout environments, respectively.