Hierarchical Generative Model Research Articles

Structure-Based Drug Design with a Deep Hierarchical Generative Model.

Recently, the remarkable growth of available crystal structure data and libraries of commercially available or readily synthesizable molecules have unlocked previously inaccessible regions of chemical space for drug development. Paired with improvements in virtual ligand screening methods, these expanded libraries are having a notable impact on early drug design efforts. Yet screening-based methods still face scalability limits, due to computational constraints and the sheer scale of drug-like space. Machine learning approaches are overcoming these limitations by learning the fundamental intra- and intermolecular relationships in drug-target systems from existing data. Here, we introduce DrugHIVE, a deep hierarchical variational autoencoder that outperforms state-of-the-art autoregressive and diffusion-based methods in both speed and performance on common generative benchmarks. DrugHIVE's hierarchical design enables improved control over molecular generation. Its capabilities include dramatically increasing virtual screening efficiency and accelerating a wide range of common drug design tasks, including de novo generation, molecular optimization, scaffold hopping, linker design, and high-throughput pattern replacement. Our highly scalable method can even be applied to receptors with high-confidence AlphaFold-predicted structures, extending the ability to generate high-quality drug-like molecules to a majority of the unsolved human proteome.

Learning With Constraint Learning: New Perspective, Solution Strategy and Various Applications.

The complexity of learning problems, such as Generative Adversarial Network (GAN) and its variants, multi-task and meta-learning, hyper-parameter learning, and a variety of real-world vision applications, demands a deeper understanding of their underlying coupling mechanisms. Existing approaches often address these problems in isolation, lacking a unified perspective that can reveal commonalities and enable effective solutions. Therefore, in this work, we proposed a new framework, named Learning with Constraint Learning (LwCL), that can holistically examine challenges and provide a unified methodology to tackle all the above-mentioned complex learning and vision problems. Specifically, LwCL is designed as a general hierarchical optimization model that captures the essence of these diverse learning and vision problems. Furthermore, we develop a gradient-response based fast solution strategy to overcome optimization challenges of the LwCL framework. Our proposed framework efficiently addresses a wide range of applications in learning and vision, encompassing three categories and nine different problem types. Extensive experiments on synthetic tasks and real-world applications verify the effectiveness of our approach. The LwCL framework offers a comprehensive solution for tackling complex machine learning and computer vision problems, bridging the gap between theory and practice.

Federated Variational Inference: Towards Improved Personalization and Generalization

Conventional federated learning algorithms train a single global model by leveraging all participating clients’ data. However, due to heterogeneity in client generative distributions and predictive models, these approaches may not appropriately approximate the predictive process, converge to an optimal state, or generalize to new clients. We study personalization and generalization in stateless cross-device federated learning setups assuming heterogeneity in client data distributions and predictive models. We first propose a hierarchical generative model and formalize it using Bayesian Inference. We then approximate this process using Variational Inference to train our model efficiently. We call this algorithm Federated Variational Inference (FedVI). We use PAC-Bayes analysis to provide generalization bounds for FedVI. We evaluate our model on FEMNIST and CIFAR-100 image classification and show that FedVI beats the state-of-the-art on both tasks.

Crafting explainable artificial intelligence through active inference: A model for transparent introspection and decision-making

This paper explores the feasibility of constructing interpretable artificial intelligence (AI) systems rooted in active inference and the free energy principle. Initially, we offer a concise introduction to active inference, emphasizing its relevance to modeling decision-making, introspection, and the generation of both overt and covert actions. Subsequently, we delve into how active inference can serve as a foundation for designing explainable AI systems. Specifically, it enables us to capture essential aspects of "introspective" processes and generate intelligible models of decision-making mechanisms. We propose an architectural framework for explainable AI systems employing active inference. Central to this framework is an explicit hierarchical generative model that enables the AI system to monitor and elucidate the factors influencing its decisions. Importantly, this model's structure is designed to be understandable and verifiable by human users. We elucidate how this architecture can amalgamate diverse data sources to make informed decisions in a transparent manner, mirroring aspects of human consciousness and introspection. Finally, we examine the implications of our findings for future AI research and discuss potential ethical considerations associated with developing AI systems with (apparent) introspective capabilities.

Crafting explainable artificial intelligence through active inference: A model for transparent introspection and decision-making

This paper explores the feasibility of constructing interpretable artificial intelligence (AI) systems rooted in active inference and the free energy principle. Initially, we offer a concise introduction to active inference, emphasizing its relevance to modeling decision-making, introspection, and the generation of both overt and covert actions. Subsequently, we delve into how active inference can serve as a foundation for designing explainable AI systems. Specifically, it enables us to capture essential aspects of "introspective" processes and generate intelligible models of decision-making mechanisms. We propose an architectural framework for explainable AI systems employing active inference. Central to this framework is an explicit hierarchical generative model that enables the AI system to monitor and elucidate the factors influencing its decisions. Importantly, this model's structure is designed to be understandable and verifiable by human users. We elucidate how this architecture can amalgamate diverse data sources to make informed decisions in a transparent manner, mirroring aspects of human consciousness and introspection. Finally, we examine the implications of our findings for future AI research and discuss potential ethical considerations associated with developing AI systems with (apparent) introspective capabilities.

Linking fast and slow: The case for generative models.

A pervasive challenge in neuroscience is testing whether neuronal connectivity changes over time due to specific causes, such as stimuli, events, or clinical interventions. Recent hardware innovations and falling data storage costs enable longer, more naturalistic neuronal recordings. The implicit opportunity for understanding the self-organised brain calls for new analysis methods that link temporal scales: from the order of milliseconds over which neuronal dynamics evolve, to the order of minutes, days, or even years over which experimental observations unfold. This review article demonstrates how hierarchical generative models and Bayesian inference help to characterise neuronal activity across different time scales. Crucially, these methods go beyond describing statistical associations among observations and enable inference about underlying mechanisms. We offer an overview of fundamental concepts in state-space modeling and suggest a taxonomy for these methods. Additionally, we introduce key mathematical principles that underscore a separation of temporal scales, such as the slaving principle, and review Bayesian methods that are being used to test hypotheses about the brain with multiscale data. We hope that this review will serve as a useful primer for experimental and computational neuroscientists on the state of the art and current directions of travel in the complex systems modelling literature.

Community-science reveals delayed fall migration of waterfowl and spatiotemporal effects of a changing climate.

Climate change has well-documented, yet variable, influences on the annual movements of migratory birds. The effects of climate change on fall migration remains understudied compared with spring but appears to be less consistent among species, regions and years. Changes in the pattern and timing of waterfowl migration in particular may result in cascading effects on ecosystem function, and socio-economic and cultural outcomes. We investigated changes in the migration of 15 waterfowl species along a major flyway corridor of continental importance in northeastern North America using 43 years of community-science data. We built spatially- and temporally explicit hierarchical generative additive models for each species and demonstrated that climate, specifically the interaction between minimum temperature and precipitation, significantly influences migration phenology for most species. Certain species' migratory movements responded to specific temperature thresholds (climate migrants) and others reacted more to the interaction of temperature and precipitation (extreme event migrants). There are already significant changes in the fall migration phenology of common waterfowl species with high ecological and economic importance, which may simply increase in the context of a changing climate. If not addressed, climate change could induce mismatches in management, regulations and population surveys which would negatively impact the hunting industry. Our findings highlight the importance of considering species-specific spatiotemporal scales of effect on climate on migration and our methods can be widely adapted to quantify and forecast climate-driven changes in wildlife migration.

Hierarchical generative modelling for autonomous robots

Humans generate intricate whole-body motions by planning, executing and combining individual limb movements. We investigated this fundamental aspect of motor control and approached the problem of autonomous task completion by hierarchical generative modelling with multi-level planning, emulating the deep temporal architecture of human motor control. We explored the temporal depth of nested timescales, where successive levels of a forward or generative model unfold, for example, object delivery requires both global planning and local coordination of limb movements. This separation of temporal scales suggests the advantage of hierarchically organizing the global planning and local control of individual limbs. We validated our proposed formulation extensively through physics simulation. Using a hierarchical generative model, we showcase that an embodied artificial intelligence system, a humanoid robot, can autonomously complete a complex task requiring a holistic use of locomotion, manipulation and grasping: the robot adeptly retrieves and transports a box, opens and walks through a door, kicks a football and exhibits robust performance even in the presence of body damage and ground irregularities. Our findings demonstrated the efficacy and feasibility of human-inspired motor control for an embodied artificial intelligence robot, highlighting the viability of the formulized hierarchical architecture for achieving autonomous completion of challenging goal-directed tasks.

Active Vision in Binocular Depth Estimation: A Top-Down Perspective.

Depth estimation is an ill-posed problem; objects of different shapes or dimensions, even if at different distances, may project to the same image on the retina. Our brain uses several cues for depth estimation, including monocular cues such as motion parallax and binocular cues such as diplopia. However, it remains unclear how the computations required for depth estimation are implemented in biologically plausible ways. State-of-the-art approaches to depth estimation based on deep neural networks implicitly describe the brain as a hierarchical feature detector. Instead, in this paper we propose an alternative approach that casts depth estimation as a problem of active inference. We show that depth can be inferred by inverting a hierarchical generative model that simultaneously predicts the eyes' projections from a 2D belief over an object. Model inversion consists of a series of biologically plausible homogeneous transformations based on Predictive Coding principles. Under the plausible assumption of a nonuniform fovea resolution, depth estimation favors an active vision strategy that fixates the object with the eyes, rendering the depth belief more accurate. This strategy is not realized by first fixating on a target and then estimating the depth; instead, it combines the two processes through action-perception cycles, with a similar mechanism of the saccades during object recognition. The proposed approach requires only local (top-down and bottom-up) message passing, which can be implemented in biologically plausible neural circuits.

Profile Likelihood for Hierarchical Models Using Data Doubling.

In scientific problems, an appropriate statistical model often involves a large number of canonical parameters. Often times, the quantities of scientific interest are real-valued functions of these canonical parameters. Statistical inference for a specified function of the canonical parameters can be carried out via the Bayesian approach by simply using the posterior distribution of the specified function of the parameter of interest. Frequentist inference is usually based on the profile likelihood for the parameter of interest. When the likelihood function is analytical, computing the profile likelihood is simply a constrained optimization problem with many numerical algorithms available. However, for hierarchical models, computing the likelihood function and hence the profile likelihood function is difficult because of the high-dimensional integration involved. We describe a simple computational method to compute profile likelihood for any specified function of the parameters of a general hierarchical model using data doubling. We provide a mathematical proof for the validity of the method under regularity conditions that assure that the distribution of the maximum likelihood estimator of the canonical parameters is non-singular, multivariate, and Gaussian.

Fluctuations in alpha and beta power provide neural states favourable for contextually relevant anticipatory processes.

Cued sensory input occasionally fails to immediately ensue its respective trigger. Given that our environments are rich in sensory cues, we often end up processing other contextually relevant information in the meantime. The experimental design of the present study allowed us to investigate how such temporal delays and visual interferences may impact anticipatory processes. Thirty-four participants were trained to remember an individualised set of eight paired-up faces. These paired-up faces were presented pseudorandomly in sequences of unpaired face images. To keep participants engaged throughout the electroencephalography study, they were instructed to classify each face image, according to its sex, as fast as possible without compromising accuracy. We observed dissimilar modulations in alpha and beta power between the 6-s timeframe encompassing the onsets of predictive and expected images (temporal delay block) and the 6-s timeframe encompassing the predictive, interference and expected images (visual interference block). Furthermore, an expectation-facilitated reduction of the face-sensitive N170 component was only observed if an anticipated face image directly followed its corresponding predictive counterpart. This effect was no longer evident when the expected face was preceded by a distracting face image. Regardless of the block type, behavioural measures confirmed that anticipated faces were classified significantly faster and with fewer erroneous responses than faces not foretold by a predictive face. Collectively, these results demonstrate that whilst the brain continuously adjusts internal hierarchical generative models to account for temporal delays in stimulus onset and visual interferences, the higher levels, and subsequent predictions, fundamental for expectation-facilitated behaviours remain intact.

Behavioral and neural evidence that the visual system performs approximate inference in a hierarchical generative model

Behavioral and neural evidence that the visual system performs approximate inference in a hierarchical generative model

Human-Instructed Deep Hierarchical Generative Learning for Automated Urban Planning

The essential task of urban planning is to generate the optimal land-use configuration of a target area. However, traditional urban planning is time-consuming and labor-intensive. Deep generative learning gives us hope that we can automate this planning process and come up with the ideal urban plans. While remarkable achievements have been obtained, they have exhibited limitations in lacking awareness of: 1) the hierarchical dependencies between functional zones and spatial grids; 2) the peer dependencies among functional zones; and 3) human regulations to ensure the usability of generated configurations. To address these limitations, we develop a novel human-instructed deep hierarchical generative model. We rethink the urban planning generative task from a unique functionality perspective, where we summarize planning requirements into different functionality projections for better urban plan generation. To this end, we develop a three-stage generation process from a target area to zones to grids. The first stage is to label the grids of a target area with latent functionalities to discover functional zones. The second stage is to perceive the planning requirements to form urban functionality projections. We propose a novel module: functionalizer to project the embedding of human instructions and geospatial contexts to the zone-level plan to obtain such projections. Each projection includes the information of land-use portfolios and the structural dependencies across spatial grids in terms of a specific urban function. The third stage is to leverage multi-attentions to model the zone-zone peer dependencies of the functionality projections to generate grid-level land-use configurations. Finally, we present extensive experiments to demonstrate the effectiveness of our framework.

Thought and language: association of groupmindedness with young English-speaking children’s production of pronouns

Shared intentionality theory posits that at age 3, children expand their conception of plural agency to include 3- or more-person groups. We sought to determine whether this conceptual shift is detectable in children’s pronoun use. We report the results of a series of Bayesian hierarchical generative models fitted to 479 English-speaking children’s first-person plural, first-person singular, second-person, third-person plural, and third-person singular pronouns. As a proportion of pronouns, children used more first-person plural pronouns, only, after 3;0 compared to before. Additionally, children used more 1pp. pronouns when their mothers used more 1pp. pronouns. As a proportion of total utterances, all pronoun classes were used more often as children aged. These findings suggest that a shift in children’s social conceptualizations at age 3 is reflected in their use of 1pp. pronouns.

Multiresolution equivariant graph variational autoencoder

In this paper, we propose Multiresolution Equivariant Graph Variational Autoencoders (MGVAE), the first hierarchical generative model to learn and generate graphs in a multiresolution and equivariant manner. At each resolution level, MGVAE employs higher order message passing to encode the graph while learning to partition it into mutually exclusive clusters and coarsening into a lower resolution that eventually creates a hierarchy of latent distributions. MGVAE then constructs a hierarchical generative model to variationally decode into a hierarchy of coarsened graphs. Importantly, our proposed framework is end-to-end permutation equivariant with respect to node ordering. MGVAE achieves competitive results with several generative tasks including general graph generation, molecular generation, unsupervised molecular representation learning to predict molecular properties, link prediction on citation graphs, and graph-based image generation. Our implementation is available at https://github.com/HyTruongSon/MGVAE.

Converging theories on dreaming: Between Freud, predictive processing, and psychedelic research.

Dreams are still an enigma of human cognition, studied extensively in psychoanalysis and neuroscience. According to the Freudian dream theory and Solms' modifications of the unconscious derived from it, the fundamental task of meeting our emotional needs is guided by the principle of homeostasis. Our innate value system generates conscious feelings of pleasure and unpleasure, resulting in the behavior of approaching or withdrawing from the world of objects. Based on these experiences, a hierarchical generative model of predictions (priors) about the world is constantly created and modified, with the aim to optimize the meeting of our needs by reducing prediction error, as described in the predictive processing model of cognition. Growing evidence from neuroimaging supports this theory. The same hierarchical functioning of the brain is in place during sleep and dreaming, with some important modifications like a lack of sensual and motor perception and action. Another characteristic of dreaming is the predominance of primary process thinking, an associative, non-rational cognitive style, which can be found in similar altered states of consciousness like the effect of psychedelics. Mental events that do not successfully fulfill an emotional need will cause a prediction error, leading to conscious attention and adaptation of the priors that incorrectly predicted the event. However, this is not the case for repressed priors (RPs), which are defined by the inability to become reconsolidated or removed, despite ongoing error signal production. We hypothesize that Solms' RPs correspond with the conflictual complexes, as described by Moser in his dream formation theory. Thus, in dreams and dream-like states, these unconscious RPs might become accessible in symbolic and non-declarative forms that the subject is able to feel and make sense of. Finally, we present the similarities between dreaming and the psychedelic state. Insights from psychedelic research could be used to inform dream research and related therapeutic interventions, and vice versa. We propose further empirical research questions and methods and finally present our ongoing trial "Biological Functions of Dreaming" to test the hypothesis that dreaming predicts intact sleep architecture and memory consolidation, via a lesion model with stroke patients who lost the ability to dream.

Brain Function: Free Energy, Predictive Processing and Active Inference

"A potential new theory of brain function based on Bayesian inference could be that the brain is a predictive processing system that generates internal models of the world to make predictions about future sensory inputs. According to this theory, the brain generates internal models based on prior beliefs and past experiences, which are used to make predictions about future sensory inputs. In summary, the free energy principle focuses on minimizing the difference between the predicted and actual sensory inputs using a hierarchical generative model, while the predictive processing theory focuses on generating and updating internal models to make predictions about future sensory inputs."

Estimation of survey efficiency and biomass for commercially important species from industry-based paired gear experiments

Fishery-independent surveys provide valuable information about trends in population abundance for management of commercially important fish stocks. A critical component of the relationship of the catches of the survey to the size of a fish stock is the catch efficiency of the survey gear. Using a general hierarchical model we estimated the relative efficiency of a chain sweep to the rockhopper sweep used by the Northeast Fisheries Science Center bottom trawl survey from paired-gear experimental tows carried out between 2015 and 2017 using a twin-trawl vessel. For 10 commercially important species, we fitted and compared a set of models with alternative assumptions about variation of relative efficiency between paired gear tows, size and diel effects on the relative efficiency, and extra-binomial variation of observations within paired gear tows. These analyses provided evidence of changes in relative efficiency with size for all species and diel effects were important for all but one species. We then used the bottom trawl survey data from surveys between 2009 and 2019 with the relative catch efficiency estimates from the best performing models to estimate annual and seasonal chain sweep-based swept area biomass for 17 managed stocks. We estimated uncertainty in all results using bootstrap procedures for each data component. We also assessed the effect of calibration on uncertainty and correlation of the annual biomass estimates.

Combining Continuous and Categorical Data Modeling in Developmental Age Estimation Using Hierarchical Bayes

Residual correlations (correlations that persist after accounting for the effect of chronological age) between variables can have a significant impact on final age estimates. Such correlations can result in overly narrow age intervals and high error rates when not accounted for. Modeling correlations can be mathematically problematic across mixed data types. Hierarchical modeling can incorporate continuous and categorical traits into a single model that accounts for correlated variables while reducing computationally expensive calculations. This paper demonstrates a Bayesian hierarchical modeling approach in which trait variables were grouped by data type or bodily system and used to produce separate age estimates with any appropriate model. These age estimates were combined into a single estimate using a multivariate normal model via nested cross-validation. The data used included nine diaphyseal length measurements and 29 epiphyseal fusion and ossification sites from 179 individuals in the publicly available U.S. Subadult Virtual Anthropology Database. Diaphyseal ages were modeled with linear regression and epiphyseal ages with random forest regression. Age estimates from the hierarchical model had reduced bias relative to diaphyseal or epiphyseal maximum likelihood estimates alone. Combined-indicator age intervals from 95% highest density regions (HDRs) were on average 15% narrower than those from diaphyseal 95% HDRs while success rates were 2% lower (91% vs. 93%). Functional example code is provided. A general hierarchical modeling approach may be applicable to other areas of skeletal analysis that employ correlated variables of mixed data types including adult age estimation and ancestry estimation.

Robustness of the performance of the optimized hierarchical two-parameter logistic IRT model for small-sample item calibration

Hierarchical Bayesian modeling is beneficial when complex models with many parameters of the same type, such as item response theory (IRT) models, are to be estimated with sparse data. Recently, Koenig et al. (Applied Psychological Measurement, 44, 311–326, 2020) illustrated in an optimized hierarchical Bayesian two-parameter logistic model (OH2PL) how to avoid bias due to unintended shrinkage or degeneracies of the posterior, and how to benefit from this approach in small samples. The generalizability of their findings, however, is limited because they investigated only a single specification of the hyperprior structure. Consequently, in a comprehensive simulation study, we investigated the robustness of the performance of the novel OH2PL in several specifications of their hyperpriors under a broad range of data conditions. We show that the novel OH2PL in the half-Cauchy or Exponential configuration yields unbiased (in terms of bias) model parameter estimates in small samples of N = 50. Moreover, it outperforms (especially in terms of the RMSE of the item discrimination parameters) marginal maximum likelihood (MML) estimation and its nonhierarchical counterpart. This further corroborates the possibility that hierarchical Bayesian IRT models behave differently than general hierarchical Bayesian models. We discuss these results regarding the applicability of complex IRT models in small-scale situations typical in psychological research, and illustrate the extended applicability of the 2PL IRT model with an empirical example.