Probabilistic Learning Research Articles

Confidence regulates feedback processing during human probabilistic learning.

Uncertainty presents a key challenge when learning how best to act to attain a desired outcome. People can report uncertainty in the form of confidence judgments, but how such judgments contribute to learning and subsequent decisions remains unclear. In a series of three experiments employing an operant learning task, we tested the hypothesis that confidence plays a central role in learning by regulating resource allocation to the seeking and processing of feedback. We predicted that, as participants' confidence in their task knowledge grew, they would discount feedback when it was provided and be correspondingly less willing to pay for it when it was costly. Consistent with these predictions, we found that higher confidence was associated with reduced electrophysiological markers of feedback processing and decreased updating of beliefs following feedback receipt. Bayesian modeling suggests that this decrease in processing was due to a drop in the expected informative value of novel information when participants were highly confident. Thus, when choosing whether to pay a fee to receive further feedback, participants' subjective confidence, rather than the objective accuracy of their decisions, guided their choices. Overall, our results suggest that confidence regulates learning and subsequent decision making. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Pharmacological modulation of dopamine receptors reveals distinct brain-wide networks associated with learning and motivation in non-human primates.

D1 and D2 receptors are heavily implicated in cognitive and motivational processes, as well as in a number of psychiatric disorders. Despite this, little is known about how selective manipulation of these different receptors impacts cognition through changing activity across brain-wide intrinsic networks. Here, we examined the acute behavioral and brain-wide effects of D1 and D2 receptor-selective antagonists, SCH-23390 and haloperidol, in macaques performing a probabilistic learning task. SCH administration diminished, and haloperidol improved, animals' task performance. Mirroring these effects on behavior, SCH reduced, and haloperidol increased, the resting-state functional connectivity across brain-wide networks, most notably in the cortico-striatal areas. Thus, our results highlight the opposing effects of D1 and D2 receptor modulation on the brain and behavior.

Supplemental Material for Confidence Regulates Feedback Processing During Human Probabilistic Learning

Supplemental Material for Confidence Regulates Feedback Processing During Human Probabilistic Learning

Long‐Term Foehn Reconstruction Combining Unsupervised and Supervised Learning

ABSTRACTFoehn winds, characterised by abrupt temperature increases and wind speed changes, significantly impact regions on the leeward side of mountain ranges, e.g., by spreading wildfires. Understanding how foehn occurrences change under climate change is crucial. As foehn is a meteorological phenomenon, its prevalence has to be inferred from meteorological measurements employing suitable classification schemes. Hence, this approach is typically limited to specific periods for which the necessary data are available. We present a novel approach for reconstructing historical foehn occurrences using a combination of unsupervised and supervised probabilistic statistical learning methods. We utilise in situ measurements (available for recent decades) to train an unsupervised learner (finite mixture model) for automatic foehn classification. These labelled data are then linked to reanalysis data (covering longer periods) using a supervised learner (lasso or boosting). This allows us to reconstruct past foehn probabilities based solely on reanalysis data. Applying this method to ERA5 reanalysis data for six stations across Switzerland and Austria achieves accurate hourly reconstructions of north and south foehn occurrence, respectively, dating back to 1940. This paves the way for investigating how seasonal foehn patterns have evolved over the past 83 years, providing valuable insights into climate change impacts on these critical wind events.

Probabilistic Machine Learning: New Frontiers for Modeling Consumers and their Choices

Making sense of massive, individual-level data is challenging: marketing researchers and analysts need flexible models that can accommodate rich patterns of heterogeneity and dynamics, work with and link diverse data types, and scale to modern data sizes. Practitioners also need tools that can quantify uncertainty in models and predictions of consumer behavior to inform optimal decision-making. In this paper, we demonstrate the promise of probabilistic machine learning (PML), which refers to the pairing of probabilistic modeling and machine learning methods, in pushing the frontier of combining flexibility, scalability, interpretability, and uncertainty quantification for building better models of consumers and their choices. Specifically, we overview both PML models and inference methods, and highlight their utility for addressing four common classes of marketing problems: (1) uncovering heterogeneity, (2) flexibly modeling nonlinearities and dynamics, (3) handling high-dimensional and unstructured data, and (4) addressing missingness, often via data fusion. We also discuss promising directions in enriching marketing models, reflecting recent developments in representation learning, causal inference, experimentation and decision-making, and theory-based behavioral modeling.

A minute-resolution downscaling algorithm for high-resolution global irradiance time series

Numerous studies have demonstrated the significant impact of the resolution of solar irradiation data on the outcomes of hourly production models. Accurate integration of photovoltaic (PV) systems sometimes demands a high-resolution global horizontal irradiance (GHI) time series to capture the rapid fluctuations in PV power output induced by swift irradiance changes. Most of the available databases provide data at hourly resolution, leading to a lack of accuracy in PV simulations. Those existing hourly averages of global horizontal irradiance in open sources fail to represent this volatility adequately, especially when PV systems are coupled with fast ramp rate technologies. In the present work, an easy-to-use algorithm is implemented to synthesize high-resolution GHI time series from hourly averaged and clear sky irradiance datasets. By employing Linear interpolation, a technique that helps to achieve the desired time resolution and afterward computing critical factors, the algorithm identifies periods characterized by short-term weather phenomena, thus creating a high-resolution time series that accurately represents these dynamics. Avoiding the probabilistic components and machine learning techniques conserves computational power and reduces calculation time, but this comes at the cost of reduced fidelity in reproducing the results. Improving accuracy in PV simulations is not always directly related to reproducing real phenomena, but enhancing the amount of information contained in the data is sufficient. This study’s approach enhances user-friendliness and facilitates seamless integration into existing energy modeling frameworks, aiming for representation with sub-hourly time steps. The algorithm’s effectiveness is demonstrated by applying the model to hourly averaged data to revert them to a one-minute time step, and finally comparing the synthetically produced one-minute GHI data to the original measured data. The comparative analysis between synthesized and measured data demonstrated a strong agreement, with normalized mean bias error (MBE) values ranging between 1.8% and 9.6% and normalized root mean square error (NRMSE) values between 2.7% and 16.1%, depending on weather conditions. Additionally, the coefficient of determination (R2) consistently remained above 0.64. Successful algorithm validation makes our algorithm suitable for use in meteorological datasets and locations, with similar climatic characteristics.

Optogenetic activation of mesencephalic projections to the nucleus accumbens shell impairs probabilistic reversal learning by disrupting learning from negative reinforcement.

Cognitive flexibility, the capacity to adapt behaviour to changes in the environment, is impaired in a range of brain disorders, including schizophrenia and Parkinson's disease. Putative neural substrates of cognitive flexibility include mesencephalic pathways to the ventral striatum (VS) and dorsomedial striatum (DMS), hypothesized to encode learning signals needed to maximize rewarded outcomes during decision-making. However, it is unclear whether mesencephalic projections to the ventral and dorsal striatum are distinct in their contribution to flexible reward-related learning. Here, rats acquired a two-choice spatial probabilistic reversal learning (PRL) task, reinforced on an 80%|20% basis (correct|incorrect responses) that assessed the flexibility of behaviour to repeated reversals of response-outcome contingencies. We report that optogenetic stimulation of projections from the ventral tegmental area (VTA) to the nucleus accumbens shell (NAcS) in the VS significantly impaired reversal learning when optical stimulation was temporally aligned with negative feedback (i.e., reward omission). VTA → NAcS stimulation during other phases of the behavioural task was without significant effect. Optogenetic stimulation of projection neurons from the substantia nigra (SN) to the DMS, aligned either with reward receipt or omission or prior to making a choice, had no significant effect on reversal learning. These findings are consistent with the notion that increased activity in the VTA → NAcS pathway disrupts behavioural flexibility by impairing learning from negative reinforcement.

A Novel Method of Bridge Deflection Prediction Using Probabilistic Deep Learning and Measured Data.

The deflection control of the main girder in suspension bridges, as flexible structures, is critically important during their operation. To predict the vertical deflection of existing suspension bridge girders under the combined effects of stochastic traffic loads and environmental temperature, this paper proposes an integrated deflection interval prediction method based on a Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), a probability density estimation layer, and bridge monitoring data. A time-series training dataset consisting of environmental temperature, vehicle load, and deflection monitoring data was built based on bridge health monitoring data. The CNN-LSTM combined layer is used to capture both local features and long-term dependencies in the time series. A Gaussian distribution (GD) is adopted as the probability density function, and its parameters are estimated using the maximum likelihood method, which outputs the optimal deflection prediction and probability intervals. Furthermore, this paper proposes a method for identifying abnormal deflections of the main girder in existing suspension bridges and establishes warning thresholds. This study indicates that, for short time scales, the CNN-LSTM-GD model achieves a 47.22% improvement in Root Mean Squared Error (RMSE) and a 12.37% increase in the coefficient of determination (R2) compared to the LSTM model. When compared to the CNN-LSTM model, it shows an improvement of 28.30% in RMSE and 6.55% in R2. For long time scales, the CNN-LSTM-GD model shows a 54.40% improvement in RMSE and a 10.22% increase in R2 compared to the LSTM model. Compared to the CNN-LSTM model, it improves RMSE by 38.43% and R2 by 5.31%. This model is instrumental in more accurately identifying abnormal deflections and determining deflection thresholds, making it applicable to bridge deflection early-warning systems.

Transitive inference as probabilistic preference learning.

Transitive inference (TI) is a cognitive task that assesses an organism's ability to infer novel relations between items based on previously acquired knowledge. TI is known for exhibiting various behavioral and neural signatures, such as the serial position effect (SPE), symbolic distance effect (SDE), and the brain's capacity to maintain and merge separate ranking models. We propose a novel framework that casts TI as a probabilistic preference learning task, using one-parameter Mallows models. We present a series of simulations that highlight the effectiveness of our novel approach. We show that the Mallows ranking model natively reproduces SDE and SPE. Furthermore, extending the model using Bayesian selection showcases its capacity to generate and merge ranking hypotheses as pairs with connecting symbols. Finally, we employ neural networks to replicate Mallows models, demonstrating how this framework aligns with observed prefrontal neural activity during TI. Our innovative approach sheds new light on the nature of TI, emphasizing the potential of probabilistic preference learning for unraveling its underlying neural mechanisms.

Chronic Ethanol Exposure Produces Sex-Dependent Impairments in Value Computations in the Striatum.

Value-based decision-making relies on the striatum, where neural plasticity can be altered by chronic ethanol (EtOH) exposure, but the effects of such plasticity on striatal neural dynamics during decision-making remain unclear. This study investigated the long-term impacts of EtOH on reward-driven decision-making and striatal neurocomputations in male and female rats using a dynamic probabilistic reversal learning task. Following a prolonged withdrawal period, EtOH-exposed male rats exhibited deficits in adaptability and exploratory behavior, with a preference for value updating based on rewards rather than omissions. These behavioral changes were linked to altered neural encoding in the dorsomedial striatum (DMS), where EtOH increased outcome-related signals and decreased choice-related signals. In contrast, female rats showed minimal behavioral changes with distinct EtOH-evoked alterations of neural signals, revealing significant sex differences in the impact of chronic EtOH. Our findings underscore the profound impact of chronic EtOH exposure on adaptive decision-making, revealing enduring changes in neurocomputational processes in the striatum underlying cognitive deficits that differ by sex.

Practical Probabilistic Model-Based Reinforcement Learning by Integrating Dropout Uncertainty and Trajectory Sampling.

This article addresses the prediction stability, prediction accuracy, and control capability of the current probabilistic model-based reinforcement learning (MBRL) built on neural networks. A novel approach to dropout-based probabilistic ensembles with trajectory sampling (DPETS) is proposed, where the system uncertainty is stably predicted by combining the Monte Carlo dropout (MC Dropout) and trajectory sampling in one framework. Its loss function is designed to correct the fitting error of neural networks for more accurate prediction of probabilistic models. The state propagation in its policy is extended to filter the aleatoric uncertainty for superior control capability. Evaluated by several Mujoco benchmark control tasks under additional disturbances and one practical robot arm manipulation task, DPETS outperforms related MBRL approaches in both average return and convergence velocity while achieving superior performance than well-known model-free baselines with significant sample efficiency. The open-source code of DPETS is available at https://github.com/mrjun123/DPETS.

Improving the classification of rock samples with diffusion probabilistic machine learning

Machine learning is a powerful data-driven technology for regression and classification tasks in numerous branches of science and technology. With probabilistic machine learning, we can approximate the probability density function of a given digital data set, unlocking the sampling of new and realistic data. We implement probabilistic machine learning as a statistical-based data augmentation tool to effectively improve the performance of a classifier on scarce data sets. Limited data are the norm in most practical situations, as is the case with core samples retrieved from unconventional oil and gas reservoirs. Obtaining and properly labeling rock sample images is a manual and time-consuming task. This leads to scarce data sets that make the lithologic classification of rock samples challenging. We illustrate the technology on images taken from well cores from the Argentinean Neuquén and Austral basins, where by introducing generated samples in the training stage of a machine-learning classifier, we achieve a 5% higher accuracy with respect to the baseline obtained using the same number of samples from the original data set. We reach these results by proposing a lightweight workflow that can be run on very limited hardware. Our workflow is an alternative to classical data augmentation strategies for dealing with incomplete information, such as geometric transformations.

Mechanical properties and probabilistic models of wood and engineered wood products: A review of green construction materials

Wood and engineered wood products are green construction materials with excellent mechanical properties, widely utilized in various buildings and bridges. This paper reviews their short-term and long-term mechanical properties and probabilistic models. Short-term mechanical properties analysis reveals strong tensile (up to 165 MPa) and bending strengths (up to 264 MPa), contrasted by lower compressive strength (up to 103 MPa) and minimal shear/rolling strength (up to 14 MPa). These properties vary based on species type, lay-up angle, and dimensions, and are mutually influential. It also demonstrated that innovative techniques such as compression and cross-laminated lay-ups enhance the strength and sustainability of wood products, utilizing smaller logs effectively and reducing carbon emissions. For long-term mechanical properties, empirical models typically used for creep strength assessment fail to account for load duration variability, a gap partly addressed by damage accumulation models which require further development. The considerable time for long-term tests underlines the need for developing accelerated load tests to more effectively assess durability. As for probabilistic models, numerical methods and machine learning provide advanced probabilistic models by analyzing extensive data, offering significant improvements over traditional experimental methods.The global contribution of the work is mainly to enhance the understanding of wood and engineered wood products for properly design and extensive application of timber structures in civil engineering and promoting sustainable development.

Declarative probabilistic logic programming in discrete-continuous domains

Over the past three decades, the logic programming paradigm has been successfully expanded to support probabilistic modeling, inference and learning. The resulting paradigm of probabilistic logic programming (PLP) and its programming languages owes much of its success to a declarative semantics, the so-called distribution semantics. However, the distribution semantics is limited to discrete random variables only. While PLP has been extended in various ways for supporting hybrid, that is, mixed discrete and continuous random variables, we are still lacking a declarative semantics for hybrid PLP that not only generalizes the distribution semantics and the modeling language but also the standard inference algorithm that is based on knowledge compilation. We contribute the measure semantics together with the hybrid PLP language DC-ProbLog (where DC stands for distributional clauses) and its inference engine infinitesimal algebraic likelihood weighting (IALW). These have the original distribution semantics, standard PLP languages such as ProbLog, and standard inference engines for PLP based on knowledge compilation as special cases. Thus, we generalize the state of the art of PLP towards hybrid PLP in three different aspects: semantics, language and inference. Furthermore, IALW is the first inference algorithm for hybrid probabilistic programming based on knowledge compilation.

Expectation to rewards modulates learning emotional words: Evidence from a hierarchical Bayesian model.

In language acquisition, individuals learn the emotional value of words through external feedback. Previous studies have used emotional words as experimental materials to explore the cognitive mechanisms underlying emotional language processing, but have failed to recognize that languages are acquired in changing environments. To this end, this study aims to combine reinforcement learning with emotional word learning, using a probabilistic reversal learning task to explore how individuals acquire the valence of emotional words in a dynamically changing environment. Computational modeling on both behavioral and event-related potential (ERP) data revealed that individuals' expectations to rewards modulated the learning speed and temporal processing of emotional words, demonstrating a clear negative bias. Specifically, as the expected value increased, individuals responded faster and exhibited higher amplitudes for negative emotional words. These findings shed light on the neural mechanisms of emotional word learning in a volatile environment, highlighting the crucial role of expectations in this process and a preference for learning negative information.

Probabilistic machine learning to improve generalisation of data-driven turbulence modelling

A probabilistic machine learning model is introduced to augment the k−ωSST turbulence model in order to improve the modelling of separated flows and the generalisability of learnt corrections. Increasingly, machine learning methods have been used to leverage experimental and high-fidelity simulation data, improving the accuracy of the Reynolds Averaged Navier–Stokes (RANS) turbulence models widely used in industry. A significant challenge for such methods is their ability to generalise to unseen geometries and flow conditions. Furthermore, heterogeneous datasets containing a mix of experimental and simulation data must be efficiently handled. In this work, field inversion and an ensemble of Gaussian Process Emulators (GPEs) is employed to address both of these challenges. The ensemble model is applied to a range of benchmark test cases, demonstrating improved turbulence modelling for cases involving separated flows with adverse pressure gradients, where RANS simulations are understood to be unreliable. Perhaps more significantly, the simulation reverted to the uncorrected model in regions of the flow exhibiting physics outside of the training data.

Ventrolateral prefrontal cortex in macaques guides decisions in different learning contexts.

Flexibly adjusting our behavioral strategies based on the environmental context is critical to maximize rewards. Ventrolateral prefrontal cortex (vlPFC) has been implicated in both learning and decision-making for probabilistic rewards, although how context influences these processes remains unclear. We collected functional neuroimaging data while rhesus macaques performed a probabilistic learning task in two contexts: one with novel and another with familiar visual stimuli. We found that activity in vlPFC encoded rewards irrespective of the context but encoded behavioral strategies that depend on reward outcome (win-stay/lose-shift) preferentially in novel contexts. Functional connectivity between vlPFC and anterior cingulate cortex varied with behavioral strategy in novel learning blocks. By contrast, connectivity between vlPFC and mediodorsal thalamus was highest when subjects repeated a prior choice. Furthermore, pharmacological D2-receptor blockade altered behavioral strategies during learning and resting-state vlPFC activity. Taken together, our results suggest that multiple vlPFC-linked circuits contribute to adaptive decision-making in different contexts.

Associations of Reversal Learning Performance With Personality Disorder Profile and Drug Abuse History in a Sample of Prison Inmates.

Prison inmate samples present a high prevalence of impulsivity- and compulsivity-related behavioral problems. The Probabilistic Reversal Learning Task (PRLT) is a useful tool to assess decision-making, and we explore its associations with inmates' personality disorder (antisocial personality disorder, APD; obsessive-compulsive personality disorder, OCPD; or both) and history of drug abuse. Mixed-effects methods were used to model acquisition and reacquisition curves across PRLT, in a sample of 275 prison inmates diagnosed with OCPD, APD, or both. Two aspects were assessed: general discrimination learning and decision-making inflexibility. Participants with a mixed personality disorder profile showed a clear pattern of decisional inflexibility. A history of drug abuse was associated with a general poorer performance but not with decision-making inflexibility. Inability to adapt to changing contingencies, and thus to adverse consequences of previously rewarded choices, was not linked to compulsivity, as hypothesized to be present in OCPD and substance use disorders, but to the mixed APD/OCPD profile.

Neural dynamics of reversal learning in the prefrontal cortex and recurrent neural networks.

In probabilistic reversal learning, the choice option yielding reward at higher probability switches at a random trial. To perform optimally in this task, one has to accumulate evidence across trials to infer the probability that a reversal has occurred. In this study, we investigated how this reversal probability is represented in cortical neurons by analyzing the neural activity in prefrontal cortex of monkeys and recurrent neural networks trained on the task. We found that neural trajectories encoding reversal probability had substantial dynamics associated with intervening behaviors necessary to perform the task. Furthermore, the neural trajectories were translated systematically in response to whether outcomes were rewarded, and their position in the neural subspace captured information about reward outcomes. These findings suggested that separable dynamic trajectories, instead of fixed points on a line attractor, provided a better description of neural representation of reversal probability. Near the behavioral reversal, in particular, the trajectories shifted monotonically across trials with stable ordering, representing varying estimates of reversal probability around the reversal point. Perturbing the neural trajectory of trained networks biased when the reversal trial occurred, showing the role of reversal probability activity in decision-making. In sum, our study shows that cortical neurons encode reversal probability in a family of dynamic neural trajectories that accommodate flexible behavior while maintaining separability to represent distinct probabilistic values.

B - 93 The Role of ex-Gaussian Parameters and Apathy in Reward Sensitivity Using a Modified Probabilistic Learning Task

Abstract Objective The current study examined the associations of apathy with attentional processes, indicated by ex-Gaussian (Ex-G) parameters of sensorimotor response speed and reward learning. Methods One hundred college-aged participants (M age = 20.32; SD = 3.69; 69 female) completed self-report measures of depression, apathy, and attentional control. Laboratory tasks included a choice reaction time (RT) task and a modified probabilistic learning task to assess sensorimotor processing and reward sensitivity. In the probabilistic learning task, participants selected the symbol associated with a high probability of gaining a monetary reward. Choice RT data were fit with Ex-G models, from which the parameter τ (bootstrapped sampling, 1000 iterations) was derived and used as the index of attentional lapses. Results A non-linear relationship was found between apathy and reward learning. Hierarchical regression was used to examine linear and higher-order relationships between apathy and behavioral indices (Ex-G τ and reward sensitivity). Results show that apathy explained none of the variance in reward learning (β = −0.005, p = 0.792, [bootstrapCI = −0.037, 0.026]). Incorporating apathy2 into the model led to significant R2 change (0.086), F(2, 89) = 0.4.142, p = 0.019, suggesting that as apathy increases, reward learning increases (β =0.006, p = 0.002, bootstrapCI = 0.002, 0.010]). Moreover, when controlling for depression, τ significantly contributes to the model, F(3, 89) = 4.142, p = 0.019, R2 = 0.17. Interactive effects revealed that at higher levels of apathy, higher Ex-G τ significantly decreases reward sensitivity (β = −0.003, p = 0.037, 95% bootstrapCI = [−0.006, −002]). Conclusions Higher levels of apathy and attentional lapses jointly reduced reward sensitivity, highlighting the role of task disengagement in apathy.