Mixture Density Networks Research Articles

Flexible Bayesian inference on partially observed epidemics.

Individual-based models of contagious processes are useful for predicting epidemic trajectories and informing intervention strategies. In such models, the incorporation of contact network information can capture the non-randomness and heterogeneity of realistic contact dynamics. In this article, we consider Bayesian inference on the spreading parameters of an SIR contagion on a known, static network, where information regarding individual disease status is known only from a series of tests (positive or negative disease status). When the contagion model is complex or information such as infection and removal times is missing, the posterior distribution can be difficult to sample from. Previous work has considered the use of Approximate Bayesian Computation (ABC), which allows for simulation-based Bayesian inference on complex models. However, ABC methods usually require the user to select reasonable summary statistics. Here, we consider an inference scheme based on the Mixture Density Network compressed ABC, which minimizes the expected posterior entropy in order to learn informative summary statistics. This allows us to conduct Bayesian inference on the parameters of a partially observed contagious process while also circumventing the need for manual summary statistic selection. This methodology can be extended to incorporate additional simulation complexities, including behavioural change after positive tests or false test results.

Knowledge-guided mixture density network for chlorophyll-a retrieval and associated pixel-by-pixel uncertainty assessment in optically variable inland waters

Machine learning has been increasingly used to retrieve chlorophyll-a (Chl-a) in optically variable waters. However, without the guidance of physical principles or expert knowledge, machine learning may produce biased mapping relationships, or waste considerable time searching for physically infeasible hyperparameter domains. In addition, most Chl-a retrieval models cannot evaluate retrieval uncertainty when ground observations are not available, and the retrieval uncertainty is crucial for understanding the model limitations and evaluating the reliability of retrieval results. In this study, we developed a novel knowledge-guided mixture density network to retrieve Chl-a in optically variable inland waters based on Sentinel-3 Ocean and Land Color Instrument (OLCI) imagery. The proposed method embedded prior knowledge derived from spectral shape classification into the mixture density network. Compared to another deterministic model, the knowledge-guided mixture density network outputted the conditional distribution of Chl-a given an input spectrum, enabling us to estimate the optimal retrieval and the associated uncertainty. The proposed method showed favorable correspondence with the field Chl-a, with root mean square error (RMSE) of 6.56 μg/L, and mean absolute percentage error (MAPE) of 43.64 %. Calibrated against Sentinel-3 OLCI spectrum, the proposed method also performed well when applied to field spectrum (RMSE = 4.58 μg/L, MAPE = 72.70 %), suggesting its effectiveness and good generalization. The proposed method provided the standard deviation of each estimated Chl-a, which enabled us to inspect the reliability of the estimated results and understand the model limitations. Overall, the proposed method improved the Chl-a retrieval in terms of model accuracy and uncertainty evaluation, providing a more comprehensive Chl-a observation of inland waters.

Probabilistic rotation modeling based on directional mixture density networks

Predicting 3D rotations from a single image presents a significant challenge, primarily due to the inherent uncertainty arising from factors such as high symmetry, self-obscuration, and noise in the 3D environment. In this work, we propose a novel multimodal-based probabilistic model that integrates the matrix Fisher distribution and von Mises Fisher distribution into a mixture density network. Our model not only captures the inherent uncertainty of the object but also learns this uncertainty directly from the training data, thereby enhancing the robustness, flexibility, and efficiency of the model. To further refine the model's ability to handle ambiguities and recognize multiple distinct modes, we introduce a relaxed version of the winner-take-all loss function. This adaptation significantly improves the model's capability in accurately representing complex multimodal distributions. The performance of our model is rigorously assessed using two challenging datasets: Pascal3D+ and ModelNet10-SO(3). Extensive experimental analysis highlights the model's exceptional capability to fit complex multimodal distributions. Notably, when tested on the ModelNet10-SO(3) dataset, which is characterized by its ambiguity, and the more unequivocal Pascal3D+ dataset, our model outperforms the prevailing top baseline models by achieving accuracy improvements of 2.7% and 3.4%, respectively, at the minimum angle threshold. These results not only demonstrate our model's advanced capabilities in fitting complex distributions but also validate its effectiveness in accurately predicting 3D rotations in both ambiguous and unambiguous scenarios.

Neural networks for evaluating induced fission product yields

Fission product yields (FPYs) play a crucial role in various aspects of nuclear science and technology, including nuclear structure and reactions. However, the inherent constraints of traditional computational methods used in theoretical models, and lack of experimental access to key observables pose challenges in obtaining accurate and comprehensive fission data. Neural Networks (NNs) present a promising solution to address these challenges by effectively modeling and acquiring energy-dependent fission yields. Mixture Density Networks (MDNs) enable learning from available data, predicting unknowns, and quantifying uncertainties simultaneously. Our study demonstrates the effectiveness of MDNs in evaluating fission product yields, particularly in scenarios where experimental data are incomplete. Machine learning algorithms like Gaussian Process Regression (GPR) can capture the distribution of single-fission yields and generate high-quality samples. These samples serve as valuable inputs for MDN networks. This study introduces an MDN approach for evaluating energy-dependent fission mass yields. The results of MDN evaluations indicate satisfactory accuracy in determining both the distribution positions and energy dependencies of FPYs.

Galaxy 3D shape recovery using mixture density network

Abstract Since the turn of the century, astronomers have been exploiting the rich information afforded by combining stellar kinematic maps and imaging in an attempt to recover the intrinsic, three-dimensional (3D) shape of a galaxy. A common intrinsic shape recovery method relies on an expected monotonic relationship between the intrinsic misalignment of the kinematic and morphological axes and the triaxiality parameter. Recent studies have, however, cast doubt about underlying assumptions relating shape and intrinsic kinematic misalignment. In this work, we aim to recover the 3D shape of individual galaxies using their projected stellar kinematic and flux distributions using a supervised machine learning approach with mixture density network (MDN). Using a mock dataset of the EAGLE hydrodynamical cosmological simulation, we train the MDN model for a carefully selected set of common kinematic and photometric parameters. Compared to previous methods, we demonstrate potential improvements achieved with the MDN model to retrieve the 3D galaxy shape along with the uncertainties, especially for prolate and triaxial systems. We make specific recommendations for recovering galaxy intrinsic shapes relevant for current and future integral field spectroscopic galaxy surveys.

Transfer Learning-Assisted Inverse Modeling in Nanophotonics Based on Mixture Density Networks

Transfer Learning-Assisted Inverse Modeling in Nanophotonics Based on Mixture Density Networks

Machine learning analysis of fission product yields

Analyzing fission product yields (FPY) is challenging because traditional models, while effective in certain conditions, have limitations in predictive accuracy and handling evolving fission modes. To overcome the limitations, especially in scenarios of limited data availability, machine learning models like gaussian process regression (GPR) and gaussian mixture model (GMM) are used for single-fission yield prediction and uncertainty quantification. The application of machine learning techniques demonstrates their practical utility in areas with constrained data, offering a novel approach for future computational advancements in nuclear physics. Our research aims to identify the most effective method for capturing the distribution of the dataset and extracting high-quality samples. These samples could serve as valuable inputs for more complex probabilistic neural networks like Mixture Density Networks (MDNs).

Temporal Mixture Density Networks for Enhanced Investment Modeling

Temporal Mixture Density Networks for Enhanced Investment Modeling

XRMDN: An Extended Recurrent Mixture Density Network for Short-Term Probabilistic Rider Demand Forecasting Considering High Volatility

XRMDN: An Extended Recurrent Mixture Density Network for Short-Term Probabilistic Rider Demand Forecasting Considering High Volatility

Ensemble of Pruned Bagged Mixture Density Networks for Improved Water Quality Retrieval Using Sentinel-2 and Landsat-8 Remote Sensing Data

Ensemble of Pruned Bagged Mixture Density Networks for Improved Water Quality Retrieval Using Sentinel-2 and Landsat-8 Remote Sensing Data

Adversarial Mixture Density Network and Uncertainty-Based Joint Learning for 360$^\circ$ Monocular Depth Estimation

Adversarial Mixture Density Network and Uncertainty-Based Joint Learning for 360$^\circ$ Monocular Depth Estimation

Deep Drug Synergy Prediction Network Using Modified Triangular Mutation-Based Differential Evolution.

Drug combination therapy is crucial in cancer treatment, but accurately predicting drug synergy remains a challenge due to the complexity of drug combinations. Machine learning and deep learning models have shown promise in drug combination prediction, but they suffer from issues such as gradient vanishing, overfitting, and parameter tuning. To address these problems, the deep drug synergy prediction network, named as EDNet is proposed that leverages a modified triangular mutation-based differential evolution algorithm. This algorithm evolves the initial connection weights and architecture-related attributes of the deep bidirectional mixture density network, improving its performance and addressing the aforementioned issues. EDNet automatically extracts relevant features and provides conditional probability distributions of output attributes. The performance of EDNet is evaluated over two well-known drug synergy datasets, NCI-ALMANAC and deep-synergy. The results demonstrate that EDNet outperforms the competing models. EDNet facilitates efficient drug interactions, enhancing the overall effectiveness of drug combinations for improved cancer treatment outcomes.

Using mixture density networks to emulate a stochastic within-host model of Francisella tularensis infection.

For stochastic models with large numbers of states, analytical techniques are often impractical, and simulations time-consuming and computationally demanding. This limitation can hinder the practical implementation of such models. In this study, we demonstrate how neural networks can be used to develop emulators for two outputs of a stochastic within-host model of Francisella tularensis infection: the dose-dependent probability of illness and the incubation period. Once the emulators are constructed, we employ Markov Chain Monte Carlo sampling methods to parameterize the within-host model using records of human infection. This inference is only possible through the use of a mixture density network to emulate the incubation period, providing accurate approximations of the corresponding probability distribution. Notably, these estimates improve upon previous approaches that relied on bacterial counts from the lungs of macaques. Our findings reveal a 50% infectious dose of approximately 10 colony-forming units and we estimate that the incubation period can last for up to 11 days following low dose exposure.

A multi-domain mixture density network for tool wear prediction under multiple machining conditions

Accurate tool wear prediction is an essential task in machining processes because it helps to schedule efficient tool maintenance and maximise the tool's useful life, thus contributing to sustainable production via zero defect manufacturing (ZDM). However, there are limitations to existing methods; these cannot be used under multiple machining conditions, which is common practice. This problem not only hinders accurate tool wear monitoring but also necessitates the use of multiple models, which increases operation and modelling costs. Therefore, the multi-domain learning problem should be addressed to enable tool wear prediction under various machining conditions. To this end, this work presents a novel method, a multi-domain mixture density network (MD2N). In particular, a Bayesian learning-based feature extractor is proposed to learn domain-invariant representations. Additionally, an adversarial learning approach is developed to lead the predictive model in learning domain-invariant features. Lastly, a mixture density network-based predictor is used to generate probabilistic tool wear outputs. Experiments that use datasets from real-world milling processes under multiple conditions prove the proposed method's promising efficacy, with the best mean absolute error (MAE), root mean squared error (RMSE), and mean absolute percentage error (MAPE) of 2.1748, 5.6422, and 0.0350, respectively, indicating the ability to learn multi-domain representations.

Probabilistic seismic inversion based on physics-guided deep mixture density network

Deterministic inversion based on deep learning has been widely utilized in model parameters estimation. Constrained by logging data, seismic data, wavelet and modeling operator, deterministic inversion based on deep learning can establish nonlinear relationships between seismic data and model parameters. However, seismic data lacks low-frequency and contains noise, which increases the non-uniqueness of the solutions. The conventional inversion method based on deep learning can only establish the deterministic relationship between seismic data and parameters, and cannot quantify the uncertainty of inversion. In order to quickly quantify the uncertainty, a physics-guided deep mixture density network (PG-DMDN) is established by combining the mixture density network (MDN) with the deep neural network (DNN). Compared with Bayesian neural network (BNN) and network dropout, PG-DMDN has lower computing cost and shorter training time. A low-frequency model is introduced in the training process of the network to help the network learn the nonlinear relationship between narrowband seismic data and low-frequency impedance. In addition, the block constraints are added to the PG-DMDN framework to improve the horizontal continuity of the inversion results. To illustrate the benefits of proposed method, the PG-DMDN is compared with existing semi-supervised inversion method. Four synthetic data examples of Marmousi II model are utilized to quantify the influence of forward modeling part, low-frequency model, noise and the pseudo-wells number on inversion results, and prove the feasibility and stability of the proposed method. In addition, the robustness and generality of the proposed method are verified by the field seismic data.

Modeling Uncertainty of GEDI Clear-Sky Terrain Height Retrievals Using a Mixture Density Network

Early spaceborne laser altimetry mission development starts in pre-phase A design, where diverse ideas are evaluated against mission science requirements. A key challenge is predicting realistic instrument performance through forward modeling at an arbitrary spatial scale. Analytical evaluations compromise accuracy for speed, while radiative transfer modeling is not applicable at the global scale due to computational expense. Instead of predicting the arbitrary properties of a lidar measurement, we develop a baseline theory to predict only the distribution of uncertainty, specifically for the terrain elevation retrieval based on terrain slope and fractional canopy cover features through a deep neural network Gaussian mixture model, also known as a mixture density network (MDN). Training data were created from differencing geocorrected Global Ecosystem Dynamics Investigation (GEDI) L2B elevation measurements with 32 independent reference lidar datasets in the contiguous U.S. from the National Ecological Observatory Network. We trained the MDN and selected hyperparameters based on the regional distribution predictive capability. On average, the relative error of the equivalent standard deviation of the predicted regional distributions was 15.9%, with some anomalies in accuracy due to generalization and insufficient feature diversity and correlation. As an application, we predict the percent of elevation residuals of a GEDI-like lidar within a given mission threshold from 60°S to 78.25°N, which correlates to a qualitative understanding of prediction accuracy and instrument performance.

Deep learning methods for solving non-uniqueness of inverse design in photonics

Deep learning has been an efficient technique for inverse design in integrated photonics. However, there is a typical non-uniqueness or one-to-many problem that impacts the convergence of network models. In this paper, we construct a multilayer structure and implement the inverse design through three main deep learning algorithms: tandem neural networks (TNNs), mixture density networks (MDNs), and conditional generative adversarial networks (CGANs). In our design, although the MDN and the CGAN is more complicated, they are more accurate and could generate multiple solutions. In particular, the MDN shows the best performance, and we proposed a new way to locate all the possible positions related to multiple solutions. As a result, for one desired input response in our test, the MDN could generate more than one design option with the mean square error (MSE) less than 0.005.

Towards global long-term water transparency products from the Landsat archive

Secchi Disk Depth (Zsd) is one of the most fundamental and widely used water-quality indicators quantifiable via optical remote sensing. Despite decades of research, development, and demonstrations, currently, there is no operational model that enables the retrieval of Zsd from the rich archive of Landsat, the long-standing civilian Earth-observation program (1972 – present). Devising a robust Zsd model requires a comprehensive in situ dataset for testing and validation, enabling consistent mapping across optically varying global aquatic ecosystems. This study utilizes Mixture Density Networks (MDNs) trained with a large in situ dataset (N = 5689) from 300+ water bodies to formulate and implement a global Zsd algorithm for Landsat sensors, including the Thematic Mapper (TM), Enhanced Thematic Mapper Plus (ETM+), and Operational Land Imager (OLI) aboard Landsat-5, -7, -8, and -9, respectively. Through an extensive Monte Carlo cross-validation with in situ data, we showed that MDNs improved Zsd retrieval when compared to other commonly used machine-learning (ML) models and recently developed semi-analytical algorithms, achieving a median symmetric accuracy (ε) of ∼29% and median bias (β) of ∼3%). A fully trained MDN model was then applied to atmospherically corrected Landsat data (i.e., remote sensing reflectance; Rrs) to both further validate our MDN-estimated Zsd products using an independent global satellite-to-in situ matchup dataset (N = 3534) and to demonstrate their utility in time-series analyses (1984 – present) via selected lakes and coastal estuaries. The quality of Rrs products rigorously assessed for the Landsat sensors indicated sensor-/band-dependent ε ranging from 8% to 37%. For our Zsd products, we found ε ∼ 39% and β ∼ 8% for the Landsat-8/OLI matchups. We observed higher errors and biases for TM and ETM+, which are explained by uncertainties in Rrs products induced by uncertainties in atmospheric correction and instrument calibration. Once these sources of uncertainty are, to the extent possible, characterized and accounted for, our developed model can then be employed to evaluate long-term trends in water transparency across unprecedented spatiotemporal scales, particularly in poorly studied regions of the world in a consistent manner.

Machine-learning Inferences of the Interior Structure of Rocky Exoplanets from Bulk Observational Constraints

Characterizing the interiors of rocky exoplanets is important to understand planetary populations and further investigate planetary habitability. New observable constraints and inference techniques have been explored for this purpose. In this work, we design and train mixture density networks (MDNs) to predict the interior properties of rocky exoplanets with large compositional diversity. In addition to measurements of mass and radius, bulk refractory elemental abundance ratios and the static Love number k 2 are used to constrain the interior of rocky exoplanets. It is found that the MDNs are able to infer the interior properties of rocky exoplanets from the available measurements of exoplanets. Compared with powerful inversion methods based on Bayesian inference, the trained MDNs provide a more rapid characterization of planetary interiors for each individual planet. The MDN model offers a convenient and practical tool for probabilistic inferences of planetary interiors.

A probabilistic track model for tropical cyclone risk assessment using multitask learning

Tropical cyclone (TC) track forecasting is critical for wind risk assessment. This work proposes a novel probabilistic TC track forecasting model based on mixture density network (MDN) and multitask learning (MTL). The existing NN-based probabilistic TC track prediction models focus on directly modeling the distribution of the future TC positions. Multitask learning has been shown to boost the performance of single tasks when the tasks are relevant. This work divides the probabilistic track prediction task into two sub-tasks: a deterministic prediction of the future TC position and a probabilistic prediction of the residual between the deterministic prediction and the actual TC location. The MDN is employed to realize the probabilistic prediction task. Since the target values of the MDN in this work are the residuals, which depend on the prediction result of the deterministic task, a novel training method is developed to train the MTL model properly. The proposed model is tested against statistical and other learning-based models on historical TC data. The results show that the proposed model outperforms other models in making probabilistic predictions. This approach advances TC track forecasting by integrating MDN and MTL, showing promise in enhancing probabilistic predictions and improving disaster preparedness.