Mixture Of Normal Distributions Research Articles

Efficient training of energy-based models using Jarzynski equality**This article is an updated version of: Carbone D, Hua M, Coste S and Vanden-Eijnden E 2023 Efficient training of energy-based models using Jarzynski equality Advances in Neural Information Processing Systems vol 36, ed A Oh, T Naumann, A Globerson, K Saenko, M Hardt and S Levine (Curran Associates, Inc.) pp 52583–614.

Abstract Energy-based models (EBMs) are generative models inspired by statistical physics with a wide range of applications in unsupervised learning. Their performance is well measured by the cross-entropy (CE) of the model distribution relative to the data distribution. Using the CE as the objective for training is, however, challenging because the computation of its gradient with respect to the model parameters requires sampling of the model distribution. Here, we show how the results for nonequilibrium thermodynamics based on the Jarzynski equality together with tools from sequential Monte Carlo sampling can be used to perform this computation efficiently and avoid the uncontrolled approximations made using the standard contrastive divergence algorithm. Specifically, we introduce a modification of the unadjusted Langevin algorithm (ULA), in which each walker acquires a weight that enables the estimation of the gradient of the CE at any step during gradient descent, thereby bypassing sampling biases induced by slow mixing of the ULA. We illustrate these results with numerical experiments on Gaussian mixture distributions as well as the MNIST and CIFAR-10 datasets. We show that the proposed approach outperforms methods based on the contrastive divergence algorithm in all the considered situations.

A comparative study of different deep learning methods for time-series probabilistic residential load power forecasting

The widespread adoption of nonlinear power electronic devices in residential settings has significantly increased the stochasticity and uncertainty of power systems. The original load power data, characterized by numerous irregular, random, and probabilistic components, adversely impacts the predictive performance of deep learning techniques, particularly neural networks. To address this challenge, this paper proposes a time-series probabilistic load power prediction technique based on the mature neural network point prediction technique, i.e., decomposing the load power data into deterministic and stochastic components. The deterministic component is predicted using deep learning neural network technology, the stochastic component is fitted with Gaussian mixture distribution model and the parameters are fitted using great expectation algorithm, after which the stochastic component prediction data is obtained using the stochastic component generation method. Using a mature neural network point prediction technique, the study evaluates six different deep learning methods to forecast residential load power. By comparing the prediction errors of these methods, the optimal model is identified, leading to a substantial improvement in prediction accuracy.

Visual object detection model integrating deep active learning

The perception of surrounding objects by vehicle autonomous driving is an important means to ensure traffic safety. Object detection models based on deep learning are widely used, but they require a large amount of labeled data for training. This paper proposes an active visual object detection model that uses Gaussian mixture distribution to estimate the uncertainty of unlabeled images, reducing the dependence of model training on labeled data. First, a mixed density network is used as the detection head, and the image features extracted by the deep neural network are used as input to estimate the probability distribution of the classification and positioning of the target prediction box. Secondly, the classification score value of the target prediction box is mapped to the probability space, and the classification uncertainty of the target is calculated using the edge uncertainty; the positioning uncertainty of the target is measured using the prediction box positioning variance. Finally, the most unstable samples are selected for labeling. Compared with other typical active learning sampling strategies on the VOC dataset, the proposed method achieves the best performance, and the 54%data annotation amount can reach the performance of YOLOX supervised learning 98.8%, saving nearly 45% of the data annotation amount.

Enhancing DHA supplementation adherence: A Bayesian approach with finite mixture models and irregular interim schedules in adaptive trial designs.

Docosahexaenoic acid (DHA) supplementation has proven beneficial in reducing preterm births. However, the challenge lies in addressing nonadherence to prescribed supplementation regimens-a hurdle that significantly impacts clinical trial outcomes. Conventional methods of adherence estimation, such as pill counts and questionnaires, usually fall short when estimating adherence within a specific dosage group. Thus, we propose a Bayesian finite mixture model to estimate adherence among women with low baseline red blood cell phospholipid DHA levels (<6%) receiving higher DHA doses. In our model, adherence is defined as the proportion of participants classified into one of the two distinct components in a normal mixture distribution. Subsequently, based on the estimands from the adherence model, we introduce a novel Bayesian adaptive trial design. Unlike conventional adaptive trials that employ regularly spaced interim schedules, the novelty of our proposed trial design lies in its adaptability to adherence percentages across the treatment arm through irregular interims. The irregular interims in the proposed trial are based on the effect size estimation informed by the finite mixture model. In summary, this study presents innovative methods for leveraging the capabilities of Bayesian finite mixture models in adherence analysis and the design of adaptive clinical trials.

Tight and Efficient Gradient Bounds for Parameterized Quantum Circuits

The training of a parameterized model largely depends on the landscape of the underlying loss function. In particular, vanishing gradients are a central bottleneck in the scalability of variational quantum algorithms (VQAs), and are known to arise in various ways. However, a caveat of most existing gradient bound results is the requirement of t-design circuit assumptions that are typically not satisfied in practice. In this work, we loosen these assumptions altogether and derive tight upper and lower bounds on loss and gradient concentration for a large class of parameterized quantum circuits and arbitrary observables, which are significantly stronger than prior work. Moreover, we show that these bounds, as well as the variance of the loss itself, can be estimated efficiently and classically-providing practical tools to study the loss landscapes of VQA models, including verifying whether or not a circuit/observable induces barren plateaus. In particular, our results can readily be leveraged to rule out barren plateaus for a realistic class of ansätze and mixed observables, namely, observables containing a non-vanishing local term. This insight has direct implications for hybrid Quantum Generative Adversarial Networks (qGANs). We prove that designing the discriminator appropriately leads to 1-local weights that stay constant in the number of qubits, regardless of discriminator depth. This implies that qGANs with appropriately chosen generators do not suffer from barren plateaus even at scale-making them a promising candidate for applications in generative quantum machine learning. We demonstrate this result by training a qGAN to learn a 2D mixture of Gaussian distributions with up to 16 qubits, and provide numerical evidence that global contributions to the gradient, while initially exponentially small, may kick in substantially over the course of training.

Deciphering gene expression patterns using large-scale transcriptomic data and its applications.

Gene expression varies stochastically across genders, racial groups, and health statuses. Deciphering these patterns is crucial for identifying informative genes, classifying samples, and understanding diseases like cancer. This study analyzes 11,252 bulk RNA-seq samples to explore expression patterns of 19,156 genes, including 10,512 cancer tissue samples and 740 normal samples. Additionally, 4,884 single-cell RNA-seq samples are examined. Statistical analysis using 16 probability distributions shows that normal samples display a wider range of distributions compared to cancer samples. Cancer samples tend to favor asymmetric distributions such as generalized extreme value, logarithmic normal, and Gaussian mixture distributions. In contrast, certain genes in normal samples exhibit symmetric distributions. Remarkably, more than 95.5% of genes exhibit non-normal distributions, which challenges traditional assumptions. Furthermore, distributions differ significantly between bulk and single-cell RNA-seq data. Many cancer driver genes exhibit distinct distribution patterns across sample types, suggesting potential for gene selection and classification based on distribution characteristics. A novel skewness-based metric is proposed to quantify distribution variation across datasets, showing genes with significant skewness differences have biological relevance. Finally, an improved naïve Bayes method incorporating gene-specific distributions demonstrates superior performance in simulations over traditional methods. This work enhances understanding of gene expression and its application in omics-based gene selection and sample classification.

Mix-DDPM: Enhancing Diffusion Models through Fitting Mixture Noise with Global Stochastic Offset

Denoising diffusion probabilistic models (DDPM) have shown impressive performance in various domains as a class of deep generative models. In this article, we introduce the mixture noise-based DDPM (Mix-DDPM), which considers the Markov diffusion posterior as a Gaussian mixture model. Specifically, Mix-DDPM randomly selects a Gaussian component and then adds the chosen Gaussian noise, which can be demonstrated as a more efficient way to perturb the signals into a simple known distribution. We further define the reverse probabilistic model as a parameterized Gaussian mixture kernel. Due to the intractability in calculating the KL divergence between Gaussian mixture models, we derive a variational bound to maximize the likelihood, offering a concise formulation for optimizing the denoising model and valuable insights for designing the sampling strategies. Our theoretical derivation highlights that Mix-DDPM need only shift image which requires the inclusion of a global stochastic offset in both the diffusion and reverse processes , which can be efficiently implemented with just several lines of code. The global stochastic offset effectively fits a Gaussian mixture distribution enhancing the degrees of freedom of the entire diffusion model. Furthermore, we present three streamlined sampling strategies that interface with diverse fast dedicated solvers for diffusion ordinary differential equations, boosting the efficacy of image representation in the sampling phase and alleviating the issue of slow generation speed, thereby enhancing both efficiency and accuracy. Extensive experiments on benchmark datasets demonstrate the effectiveness of Mix-DDPM and its superiority over the original DDPM.

Domain knowledge-driven encoder–decoder for nasopharyngeal carcinoma segmentation

Accurate image segmentation is vital for the diagnosis and treatment of nasopharyngeal carcinoma (NPC). Although deep neural networks have shown promising performance in NPC segmentation, they depend on large-scale pixel-level annotation datasets for purely data-driven model training, which poses practical limitations. To address this challenge, this paper explores the domain knowledge, namely the expert knowledge from radiologists, and proposes a domain knowledge-driven encoder–decoder architecture. Specifically, the domain knowledge on image spatial information is formulated as Gaussian mixture distribution and then transformed into an optimal transport-based expert-prior regularization of the encoder, which enhances the model’s ability in capturing discriminative features. To project the encoded features onto pixel space and obtain the segmentation maps, a cross-scale feature refinement module is built in the decoder with theoretical justification, which integrates the domain knowledge that radiologists segment NPC in a gradual refinement process. Experimental results verify the effectiveness of the proposed method for NPC segmentation. Remarkably, despite using only 10% of pixel-level annotation data, the domain knowledge-driven model outperforms recent deep neural networks that use the entire training data.

Bounded Flexible Scale Mixture of Normal Distributions with Application to Image Segmentation

Bounded Flexible Scale Mixture of Normal Distributions with Application to Image Segmentation

Mixture Gaussian process model with Gaussian mixture distribution for big data

In the era of chemical big data, the high complexity and strong interdependencies present in the datasets pose considerable challenges when constructing accurate parametric models. The Gaussian process model, owing to its non-parametric nature, demonstrates better adaptability when confronted with complex and interdependent data. However, the standard Gaussian process has two significant limitations. Firstly, the time complexity of inverting its kernel matrix during the inference process is O(n)3. Secondly, all data share a common kernel function parameter, which mixes different data types and reduces the model accuracy in mixing-category data identification problems. In light of this, this paper proposes a mixture Gaussian process model that addresses these limitations. This model reduces time complexity and distinguishes data based on different data features. It incorporates a Gaussian mixture distribution for the inducing variables to approximate the original data distribution. Stochastic Variational Inference is utilized to reduce the computational time required for parameter inference. The inducing variables have distinct parameters for the kernel function based on the data category, leading to improved analytical accuracy and reduced time complexity of the Gaussian process model. Numerical experiments are conducted to analyze and compare the performance of the proposed model on different-sized datasets and various data category cases.

Spatial Autoregressive Model with Mixture of Gaussian Distribution for the Random Effect: Formulation, Estimation and Application

Spatial Autoregressive Model with Mixture of Gaussian Distribution for the Random Effect: Formulation, Estimation and Application

Semiparametric regression models under skew scale mixtures of normal distributions

Semiparametric models (SM) are an important tool in modeling environmental data where generally a covariate presents an unknown nonlinear behavior. Usually, the error component is assumed to follow a normal distribution. However, in some situations, the response variable is skewed and heavy-tailed. This paper aims to extend the SMs allowing the errors to follow a skew scale mixture of normal distributions, increasing the model’s flexibility. In particular, we develop the EM algorithm for the proposed model, diagnostic analysis via global, local influence, and generalized leverage. A simulation study is also conducted to evaluate the efficiency of the EM algorithm. Finally, a suitable transformation is applied in a data set on ragweed pollen concentration to illustrate the utility of the proposed model.

A probabilistic proof of some integral formulas involving incomplete gamma functions

The theory of normal variance mixture distributions is used to provide elementary derivations of closed-form expressions for the definite integrals ∫ 0 ∞ x − 2 ν cos ⁡ ( b x ) γ ( ν , α x 2 ) d x (for ν > 1 / 2 , b > 0, α > 0) and ∫ 0 ∞ x 2 ν − 1 cos ⁡ ( b x ) Γ ( − ν , α x 2 ) d x (for ν > 0, b > 0, α > 0), where γ(a, x) and Γ(a, x) are the lower and upper incomplete gamma functions, respectively. The method of proof is of independent interest and could be used to derive further new definite integral formulas.

PPMGS: An efficient and effective solution for distributed privacy-preserving semi-supervised learning

Recently, distributed semi-supervised learning has attracted increasing research attention due to its tremendous practical value. A promising distributed semi-supervised learning method should not only achieve desirable classification performance but also protect data privacy in distributed scenarios. Existing approaches typically capture the similarities between data instances with privacy-preserving computations. This paradigm introduces extra computation and heuristic changes to the algorithm, resulting in sub-optimal solutions that are time-consuming. In current distributed semi-supervised learning, instance similarities are widely used to capture the underlying manifold or guide label propagation. This paper emphasizes that instance similarities are not necessary because the structure of data connections can be estimated using coarser-grained information. We propose a Privacy-preserving Mixture-distribution based Graph Smoothing (PPMGS) model for distributed privacy-preserving semi-supervised learning. Our motivation is to construct a graph based on a Gaussian mixture distribution instead of individual data instances, which better captures the underlying data distribution and improves model efficiency. PPMGS includes a privacy-preserving expectation-maximization (EM) phase to estimate the Gaussian mixture distribution depicting the input data and a mixture-distribution-based graph smoothing algorithm to learn a distribution-based classifier by fitting a few labeled samples. Experimental results show that the proposed PPMGS achieves 5%-10% higher accuracy and macro-F1 than state-of-the-art privacy-preserving semi-supervised learning methods. In terms of efficiency, it reduces time cost by 97% and communication cost by 96% in the most complex dataset. The numerical results demonstrate that our proposal outperforms state-of-the-art baselines in both efficiency and effectiveness.

Simulation algorithms for stationary sequences with distributions in the form of a mixture of Gaussian distributions

Abstract In this paper, we present three algorithms for simulation of intervals of stationary vector and scalar sequences with partial distributions of their subsequences of fixed length in the form of a mixture of Gaussian distributions. The first algorithm is based on superposition of two Gaussian vector processes and the second and third ones use the method of conditional distributions and the method of superpositions to simulate the mixtures and to select realizations for approximate construction of conditional realizations. Some properties of these algorithms are presented.

On Univariate Gaussian Mixture Model for Diabetic Patients in Yola, Adamawa State

Finite Mixture Model serves as a potent statistical tool for modeling intricate data distributions through the amalgamation of multiple Gaussian components. The aim of this work is to use finite mixture of normal distribution to model data on diabetic patients from Modibbo Adama University Teaching Hospital, Yola, Adamawa State, so as to expose latent sub-groups. The model is first used on synthetic data and subsequently on diabetic patients data. The diabetic data set consists of 553 cases of diabetes mellitus. The variables measured: Age(years), Mass of a patient(kg/meters), glucose level (plasma glucose con- centration, a 2-hour in an oral glucose tolerance test), pressure (Diastolic blood pressure mmHg), insulin (2-hour serum insulin mu U/ml) and class variable (0 or 1) treating 0 as false or negative and 1 treated as true or positive test for diabetes. The synthetic data is obtained from Gaussian distribution. The results showed that there is presence of heterogeneity in the data by showing the presence of 2 sub-groups.

Predictive Model for EV Charging Load Incorporating Multimodal Travel Behavior and Microscopic Traffic Simulation

A predictive model for the spatiotemporal distribution of electric vehicle (EV) charging load is proposed in this paper, considering multimodal travel behavior and microscopic traffic simulation. Firstly, the characteristic variables of travel time are fitted using advanced techniques such as Gaussian mixture distribution. Simultaneously, the user’s multimodal travel behavior is delineated by introducing travel purpose transfer probabilities, thus establishing a comprehensive travel spatiotemporal model. Secondly, the improved Floyd algorithm is employed to select the optimal path, taking into account various factors including signal light status, vehicle speed, and the position of starting and ending sections. Moreover, the approach of multi-lane lane change following and the utilization of cellular automata theory are introduced. To establish a microscopic traffic simulation model, a real-time energy consumption model is integrated with the aforementioned techniques. Thirdly, the minimum regret value is leveraged in conjunction with various other factors, including driving purpose, charging station electricity price, parking cost, and more, to simulate the decision-making process of users regarding charging stations. Subsequently, an EV charging load predictive framework is proposed based on the approach driven by electricity prices and real-time interaction of coupled network information. Finally, this paper conducts large-scale simulations to analyze the spatiotemporal distribution characteristics of EV charging load using a regional transportation network in East China and a typical power distribution network as case studies, thereby validating the feasibility of the proposed method.

An Approximate Algorithm for Simulating Stationary Discrete Random Processes with Bivariate Distributions of Their Consecutive Components in the Form of Mixtures of Gaussian Distributions

An Approximate Algorithm for Simulating Stationary Discrete Random Processes with Bivariate Distributions of Their Consecutive Components in the Form of Mixtures of Gaussian Distributions

Type I Tobit Bayesian Additive Regression Trees for censored outcome regression

Censoring occurs when an outcome is unobserved beyond some threshold value. Methods that do not account for censoring produce biased predictions of the unobserved outcome. This paper introduces Type I Tobit Bayesian Additive Regression Tree (TOBART-1) models for censored outcomes. Simulation results and real data applications demonstrate that TOBART-1 produces accurate predictions of censored outcomes. TOBART-1 provides posterior intervals for the conditional expectation and other quantities of interest. The error term distribution can have a large impact on the expectation of the censored outcome. Therefore, the error is flexibly modeled as a Dirichlet process mixture of normal distributions. An R package is available at https://github.com/EoghanONeill/TobitBART.

Integrated deviance information criterion for spatial autoregressive models with heteroskedasticity

In this study, we introduce the integrated deviance information criterion (DIC) for nested and non-nested model selection problems in heteroskedastic spatial autoregressive models. In a Bayesian estimation setting, we assume that the idiosyncratic error terms of our spatial autoregressive model have a scale mixture of normal distributions, where the scale mixture variables are latent variables that induce heteroskedasticity. We first derive the integrated likelihood function by analytically integrating out the scale mixture variables from the complete-data likelihood function. We then use the integrated likelihood function to formulate the integrated DIC measure. We investigate the finite sample performance of the integrated DIC in selecting the true model in a simulation study. The simulation results show that the integrated DIC performs satisfactorily and can be useful for selecting the correct model in specification search exercises. Finally, in a spatially augmented economic growth model, we use the integrated DIC to choose the spatial weights matrix that leads to better predictive accuracy.