High-dimensional Parameter Space Research Articles

Optimizing data acquisition: a Bayesian approach for efficient machine learning model training

Abstract Acquiring a substantial number of data points for training accurate machine learning (ML) models is a big challenge in scientific fields where data collection is resource-intensive. Here, we propose a novel approach for constructing a minimal yet highly informative database for training ML models in complex multi-dimensional parameter spaces. To achieve this, we mimic the underlying relation between the output and input parameters using Gaussian process regression (GPR). Using a set of known data, GPR provides predictive means and standard deviation for the unknown data. Given the predicted standard deviation by GPR, we select data points using Bayesian optimization to obtain an efficient database for training ML models. We compare the performance of ML models trained on databases obtained through this method, with databases obtained using traditional approaches. Our results demonstrate that the ML models trained on the database obtained using Bayesian optimization approach consistently outperform the other two databases, achieving high accuracy with a significantly smaller number of data points. Our work contributes to the resource-efficient collection of data in high-dimensional complex parameter spaces, to achieve high precision machine learning predictions.

Zooming by in the CARPoolGP Lane: New CAMELS-TNG Simulations of Zoomed-in Massive Halos

Galaxy formation models within cosmological hydrodynamical simulations contain numerous parameters with nontrivial influences over the resulting properties of simulated cosmic structures and galaxy populations. It is computationally challenging to sample these high dimensional parameter spaces with simulations, in particular for halos in the high-mass end of the mass function. In this work, we develop a novel sampling and reduced variance regression method, CARPoolGP, which leverages built-in correlations between samples in different locations of high dimensional parameter spaces to provide an efficient way to explore parameter space and generate low-variance emulations of summary statistics. We use this method to extend the Cosmology and Astrophysics with machinE Learning Simulations to include a set of 768 zoom-in simulations of halos in the mass range of 1013–1014.5 M ⊙ h −1 that span a 28-dimensional parameter space in the IllustrisTNG model. With these simulations and the CARPoolGP emulation method, we explore parameter trends in the Compton Y–M, black hole mass–halo mass, and metallicity–mass relations, as well as thermodynamic profiles and quenched fractions of satellite galaxies. We use these emulations to provide a physical picture of the complex interplay between supernova and active galactic nuclei feedback. We then use emulations of the Y–M relation of massive halos to perform Fisher forecasts on astrophysical parameters for future Sunyaev–Zeldovich observations and find a significant improvement in forecasted constraints. We publicly release both the simulation suite and CARPoolGP software package.

Accelerating multilayer frequency selective surface absorber design by combining equivalent circuit models with machine learning

Designing high-performance electromagnetic functional materials and artificial structures is of great significance for electromagnetic wave modulation applications. Optimizing performance in the high-dimensional parameter space of electromagnetic functional materials has posed a challenging problem. This paper proposes the use of reinforcement and transfer learning methods to facilitate the quick and accurate design of wideband circuit analog absorbers (CAA). A trained reinforcement learning network is utilized to comprehend the relationship between reflectivity performance and changes in impedance parameter. Furthermore, the transfer learning method is applied to accelerate the training process, leading in a 20-fold reduction in the number of simulations. To validate the effectiveness of this approach, a double-layer absorber aiming for reflectivity less than −20 dB at 3–10 GHz is designed, requiring only 50 simulations. This demonstrates that the proposed method provides a flexible strategy for the interactive design of equivalent circuit and electromagnetic structural parameters. Moreover, it presents a novel and efficient solution for microwave absorber design, with potential applications across various microwave design fields.

Evolutionary reinforcement learning with action sequence search for imperfect information games

Deep Reinforcement Learning (DRL) has achieved remarkable success in perfect information games. However, when applied to imperfect information games like Contract Bridge, DRL faces challenges not only from unobservable partial information but also from the lack of efficient exploration. Although several Evolutionary Reinforcement Learning algorithms (ERLs) have harnessed Evolutionary Algorithms (EAs) to enhance exploration capabilities, the practical performance of EAs is limited either by the high-dimensional parameter space or possibly inaccurate guidance from value networks. In this paper, we introduce a novel ERL algorithm that employs the Particle Swarm Optimization (PSO) algorithm to search for superior action sequences, thereby aiding agents in exploring uncharted territory. We conduct the search in action space and evaluate action sequences through interactions with the environment to avoid the limitations of parameter space and value networks, respectively. The diverse experiences collected in the search and evaluation can boost the learning of the DRL agent. In addition, the action sequence search is executed only when the agent converges to local optima, which can reduce the overall cost of action evaluation and avoid influencing the optimization process of DRL. Through experimental comparisons conducted on Contract Bridge, our method demonstrates superior performance when compared with several state-of-the-art DRL and ERL algorithms. Furthermore, we utilize our method in the Bridge Competition of AAMAS 2023 Imperfect Information Card Games Competition and rank the first.

Deep learning-based predictive models for laser direct drive at the Omega Laser Facility

The rich and complex physics of inertial confinement fusion provides a unique and challenging space for high-fidelity first-principles modeling. Consequently, simulation codes that are used to design experiments are computationally expensive and lack the predictive capability required for extensive parameter exploration in search of a high-performing design for laser direct drive. In this article, we present two deep-learning-based predictive models intended to address these difficulties. The first model (TL DNN) acts as a fast emulator of simulations as well as experiments at the Omega Laser Facility. This model is trained on a simulation database and subsequently calibrated on experimental data using transfer learning. To facilitate the development of this model, an autoencoder is developed to reduce the dimensionality of the input space by compressing the laser pulse input. The model predicts key experimental scalar observables of Omega experiments with high accuracy and minimal computational cost. This deep neural net enables rapid exploration of a high-dimensional input parameter space for an optimal implosion design. The second model (DNN SM+) aims to extend the statistical modeling work of Lees et al. [Phys. Rev. Lett. 127, 105001 (2021)], by increasing the complexity of the model space and allowing for coupling between degradation terms. Since the model capacity of DNN SM+ is higher than the model of Lees et al., DNN SM+ can potentially provide an improvement in predictive capability, and we use this model to provide insight into complicated degradation dependencies.

Merging experiment data and simulation data for parameter identification of shaft seal

A parameter identification method which merges experimentally monitored signals and physics-based simulation data is proposed, targeting the identification tasks in shaft seals which are challenging due to the high-dimensional parameter space. Features were extracted from the monitored acoustic emission signals following a proposed cross-timescale analysis routine, and the simulation data were augmented using Kriging surrogate model to obtain a dataset with stratified fidelity. Then, a transferable architecture of convolutional neural network modified for periodical data was proposed, with which part of the parameters trained by simulation data were reserved when training model using acoustic emission data acquired in experiments. Cross validation shows that transfer learning can effectively improve the performance, provided data augmentation and proper transfer mode. In conclusion, the study provides an effective parameter identification method which merges the simulation data which carry the physical knowledge and the experiment data which carry directly monitored results.

Review and experimental benchmarking of machine learning algorithms for efficient optimization of cold atom experiments

The generation of cold atom clouds is a complex process which involves the optimization of noisy data in high dimensional parameter spaces. Optimization can be challenging both in and especially outside of the lab due to lack of time, expertise, or access for lengthy manual optimization. In recent years, it was demonstrated that machine learning offers a solution since it can optimize high dimensional problems quickly, without knowledge of the experiment itself. In this paper we present results showing the benchmarking of nine different optimization techniques and implementations, alongside their ability to optimize a rubidium (Rb) cold atom experiment. The investigations are performed on a 3D 87Rb molasses with 10 and 18 adjustable parameters, respectively, where the atom number obtained by absorption imaging was chosen as the test problem. We further compare the best performing optimizers under different effective noise conditions by reducing the signal-to-noise ratio of the images via adapting the atomic vapor pressure in the 2D+ magneto-optical trap and the detection laser frequency stability.

Cluster sampling and scalable Bayesian optimization with constraints for negative tone development resist model calibration

As the semiconductor technology node continues to shrink, achieving smaller critical dimension in lithography becomes increasingly challenging. Negative tone development (NTD) process is widely employed in advanced node due to their large process window. However, the unique characteristics of NTD, such as shrinkage effect, make the NTD resist model calibration more complex. Gradient descent (GD) and heuristic methods have been applied for calibration of NTD resist model. Nevertheless, these methods depend on initial parameter selection and tend to fall into local optima, resulting in poor accuracy of the NTD model and massive computational time. In this paper, we propose cluster sampling and scalable Bayesian optimization (BO) with constraints method for NTD resist model calibration. This approach utilizes cluster sampling strategy to enhance the capability for global initial sampling and employs scalable BO with constraints for global optimization of high-dimensional parameter space. With this approach, the calibration accuracy is significantly enhanced in comparison with results from GD and heuristic methods, and the computational efficiency is substantially improved compared with GD. By gearing up cluster sampling strategy and scalable BO with constraints, this method offers a new and efficient resist model calibration.

Search for Extreme Mass Ratio Inspirals Using Particle Swarm Optimization and Reduced Dimensionality Likelihoods

Extreme-mass-ratio inspirals (EMRIs) are significant observational targets for spaceborne gravitational wave detectors, namely, LISA, Taiji, and Tianqin, which involve the inspiral of stellar-mass compact objects into massive black holes (MBHs) with a mass range of approximately 104∼107M⊙. EMRIs are estimated to produce long-lived gravitational wave signals with more than 105 cycles before plunge, making them an ideal laboratory for exploring the strong-gravity properties of the spacetimes around the MBHs, stellar dynamics in galactic nuclei, and properties of the MBHs itself. However, the complexity of the waveform model, which involves the superposition of multiple harmonics, as well as the high-dimensional and large-volume parameter space, make the fully coherent search challenging. In our previous work, we proposed a 10-dimensional search using Particle Swarm Optimization (PSO) with local maximization over the three initial angles. In this study, we extend the search to an 8-dimensional PSO with local maximization over both the three initial angles and the angles of spin direction of the MBH, where the latter contribute a time-independent amplitude to the waveforms. Additionally, we propose a 7-dimensional PSO search by using a fiducial value for the initial orbital frequency and shifting the corresponding 8-dimensional Time Delay Interferometry responses until a certain lag returns the corresponding 8-dimensional log-likelihood ratio’s maximum. The reduced dimensionality likelihoods enable us to successfully search for EMRI signals with a duration of 0.5 years and signal-to-noise ratio of 50 within a wider search range than our previous study. However, the ranges used by both the LISA Data Challenge (LDC) and Mock LISA Data Challenge (MLDC) to generate their simulated signals are still wider than the those we currently employ in our direct searches. Consequently, we discuss further developments, such as using a hierarchical search to narrow down the search ranges of certain parameters and applying Graphics Processing Units to speed up the code. These advances aim to improve the efficiency, accuracy, and generality of the EMRI search algorithm.

Data-driven prediction of dimensionless quantities for semi-infinite target penetration by integrating machine-learning and feature selection methods

This study employs a data-driven methodology that embeds the principle of dimensional invariance into an artificial neural network to automatically identify dominant dimensionless quantities in the penetration of rod projectiles into semi-infinite metal targets from experimental measurements. The derived mathematical expressions of dimensionless quantities are simplified by the examination of the exponent matrix and coupling relationships between feature variables. As a physics-based dimension reduction methodology, this way reduces high-dimensional parameter spaces to descriptions involving only a few physically interpretable dimensionless quantities in penetrating cases. Then the relative importance of various dimensionless feature variables on the penetration efficiencies for four impacting conditions is evaluated through feature selection engineering. The results indicate that the selected critical dimensionless feature variables by this synergistic method, without referring to the complex theoretical equations and aiding in the detailed knowledge of penetration mechanics, are in accordance with those reported in the reference. Lastly, the determined dimensionless quantities can be efficiently applied to conduct semi-empirical analysis for the specific penetrating case, and the reliability of regression functions is validated.

Mathematics of Epidemics: On the General Solution of SIRVD, SIRV, SIRD, and SIR Compartment Models

The susceptible–infected–recovered–vaccinated–deceased (SIRVD) epidemic compartment model extends the SIR model to include the effects of vaccination campaigns and time-dependent fatality rates on epidemic outbreaks. It encompasses the SIR, SIRV, SIRD, and SI models as special cases, with individual time-dependent rates governing transitions between different fractions. We investigate a special class of exact solutions and accurate analytical approximations for the SIRVD and SIRD compartment models. While the SIRVD and SIRD equations pose complex integro-differential equations for the rate of new infections and the fractions as a function of time, a simpler approach considers determining equations for the sum of ratios for given variations. This approach enables us to derive fully exact analytical solutions for the SIRVD and SIRD models. For nonlinear models with a high-dimensional parameter space, such as the SIRVD and SIRD models, analytical solutions, exact or accurately approximative, are of high importance and interest, not only as suitable benchmarks for numerical codes, but especially as they allow us to understand the critical behavior of epidemic outbursts as well as the decisive role of certain parameters. In the second part of our study, we apply a recently developed analytical approximation for the SIR and SIRV models to the more general SIRVD model. This approximation offers accurate analytical expressions for epidemic quantities, such as the rate of new infections and the fraction of infected persons, particularly when the cumulative fraction of infections is small. The distinction between recovered and deceased individuals in the SIRVD model affects the calculation of the death rate, which is proportional to the infected fraction in the SIRVD/SIRD cases but often proportional to the rate of new infections in many SIR models using an a posteriori approach. We demonstrate that the temporal dependence of the infected fraction and the rate of new infections differs when considering the effects of vaccinations and when the real-time dependence of fatality and recovery rates diverge. These differences are highlighted for stationary ratios and gradually decreasing fatality rates. The case of stationary ratios allows one to construct a new powerful diagnostics method to extract analytically all SIRVD model parameters from measured COVID-19 data of a completed pandemic wave.

On the design of searching algorithm for parameter plateau in quantitative trading strategies using particle swarm optimization

Quantitative trading, relying on diverse parameter combinations, is becoming increasingly the norm for trading strategies in financial investments. The performance of these strategies is intricately linked to these parameters. However, the performance on the training set after backtesting does not ensure success on a test set and may lead to overfitting. This study emphasizes enhancing stability and robustness in trading-strategy parameters by introducing a ’parameter plateau.’ Traditional brute-force methods for exploring high-dimensional parameter spaces can be intricate and time-consuming. To address this challenge, we present an efficient alternative that identifies stable and robust parameters by configuring parameter plateaus to mitigate overfitting risks. A step-by-step search algorithm is proposed to determine the optimal parameters, leveraging the power of particle-swarm optimization. In continuous, multi-dimensional solution spaces, particle-swarm optimization is invaluable for the swift and effective discovery of the desired solutions. Experiments underscore the substantial influence of the parameter plateau concept on parameter selection, highlighting the pivotal role of particle-swarm optimization in efficiently navigating complex solution spaces and thereby enabling the discovery of stable and profitable trading strategies.

Interpreting generative adversarial networks to infer natural selection from genetic data.

Understanding natural selection and other forms of non-neutrality is a major focus for the use of machine learning in population genetics. Existing methods rely on computationally intensive simulated training data. Unlike efficient neutral coalescent simulations for demographic inference, realistic simulations of selection typically require slow forward simulations. Because there are many possible modes of selection, a high dimensional parameter space must be explored, with no guarantee that the simulated models are close to the real processes. Finally, it is difficult to interpret trained neural networks, leading to a lack of understanding about what features contribute to classification. Here we develop a new approach to detect selection and other local evolutionary processes that requires relatively few selection simulations during training. We build upon a generative adversarial network trained to simulate realistic neutral data. This consists of a generator (fitted demographic model), and a discriminator (convolutional neural network) that predicts whether a genomic region is real or fake. As the generator can only generate data under neutral demographic processes, regions of real data that the discriminator recognizes as having a high probability of being "real" do not fit the neutral demographic model and are therefore candidates for targets of selection. To incentivize identification of a specific mode of selection, we fine-tune the discriminator with a small number of custom non-neutral simulations. We show that this approach has high power to detect various forms of selection in simulations, and that it finds regions under positive selection identified by state-of-the-art population genetic methods in three human populations. Finally, we show how to interpret the trained networks by clustering hidden units of the discriminator based on their correlation patterns with known summary statistics.

Inferring networks from time series: A neural approach.

Network structures underlie the dynamics of many complex phenomena, from gene regulation and foodwebs to power grids and social media. Yet, as they often cannot be observed directly, their connectivities must be inferred from observations of the dynamics to which they give rise. In this work, we present a powerful computational method to infer large network adjacency matrices from time series data using a neural network, in order to provide uncertainty quantification on the prediction in a manner that reflects both the degree to which the inference problem is underdetermined as well as the noise on the data. This is a feature that other approaches have hitherto been lacking. We demonstrate our method's capabilities by inferring line failure locations in the British power grid from its response to a power cut, providing probability densities on each edge and allowing the use of hypothesis testing to make meaningful probabilistic statements about the location of the cut. Our method is significantly more accurate than both Markov-chain Monte Carlo sampling and least squares regression on noisy data and when the problem is underdetermined, while naturally extending to the case of nonlinear dynamics, which we demonstrate by learning an entire cost matrix for a nonlinear model of economic activity in Greater London. Not having been specifically engineered for network inference, this method in fact represents a general parameter estimation scheme that is applicable to any high-dimensional parameter space.

ANOVA-GP Modeling for High-Dimensional Bayesian Inverse Problems

Markov chain Monte Carlo (MCMC) stands out as an effective method for tackling Bayesian inverse problems. However, when dealing with computationally expensive forward models and high-dimensional parameter spaces, the challenge of repeated sampling becomes pronounced. A common strategy to address this challenge is to construct an inexpensive surrogate of the forward model, which cuts the computational cost of individual samples. While the Gaussian process (GP) is widely used as a surrogate modeling strategy, its applicability can be limited when dealing with high-dimensional input or output spaces. This paper presents a novel approach that combines the analysis of variance (ANOVA) decomposition method with Gaussian process regression to handle high-dimensional Bayesian inverse problems. Initially, the ANOVA method is employed to reduce the dimension of the parameter space, which decomposes the original high-dimensional problem into several low-dimensional sub-problems. Subsequently, principal component analysis (PCA) is utilized to reduce the dimension of the output space on each sub-problem. Finally, a Gaussian process model with a low-dimensional input and output is constructed for each sub-problem. In addition to this methodology, an adaptive ANOVA-GP-MCMC algorithm is proposed, which further enhances the adaptability and efficiency of the method in the Bayesian inversion setting. The accuracy and computational efficiency of the proposed approach are validated through numerical experiments. This innovative integration of ANOVA and Gaussian processes provides a promising solution to address challenges associated with high-dimensional parameter spaces and computationally expensive forward models in Bayesian inference.

Time-series forecasting through recurrent topology

Time-series forecasting is a practical goal in many areas of science and engineering. Common approaches for forecasting future events often rely on highly parameterized or black-box models. However, these are associated with a variety of drawbacks including critical model assumptions, uncertainties in their estimated input hyperparameters, and computational cost. All of these can limit model selection and performance. Here, we introduce a learning algorithm that avoids these drawbacks. A variety of data types including chaotic systems, macroeconomic data, wearable sensor recordings, and population dynamics are used to show that Forecasting through Recurrent Topology (FReT) can generate multi-step-ahead forecasts of unseen data. With no free parameters or even a need for computationally costly hyperparameter optimization procedures in high-dimensional parameter space, the simplicity of FReT offers an attractive alternative to complex models where increased model complexity may limit interpretability/explainability and impose unnecessary system-level computational load and power consumption constraints.

Anomaly detection using large-scale multimode industrial data: An integration method of nonstationary kernel and autoencoder

Kernel methods and neural networks (NNs) are two mainstream nonlinear data modeling methods and have been widely applied to industrial process monitoring. However, they both present imperfect properties, so the relevant applications are limited. On the one hand, kernels are not so reconstructable, scalable, and robust to hyperparameters that they suffer performance degradation for large-scale data modeling and monitoring. On the other hand, the high-dimensional parameter space of NNs that is sorted to parameter initialization presents severe anomaly detection performance inconsistency, which makes the industry cautious about using NNs. Motivated by these facts, we propose to integrate kernels and NNs, forming a new model structure that is scalable, reconstructable, and performance-consistent. Specifically, a novel autoencoder-based nonstationary pattern selection kernel (AE-NPSK) is proposed by (1) selecting from the training set the critical edges and interior data as the centers of the radial basis functions in the hidden layers and (2) adaptively adjusting the kernel width in the training procedure. Also, the new NN has strong performance consistency, which facilitates the search for optimal parameters. Finally, we test the performance of the proposed method on the challenging multimode processes. The results validate the efficacy of the proposed method.

Prototype Bayesian Meta-Learning for Few-Shot Image Classification.

Meta-learning aims to leverage prior knowledge from related tasks to enable a base learner to quickly adapt to new tasks with limited labeled samples. However, traditional meta-learning methods have limitations as they provide an optimal initialization for all new tasks, disregarding the inherent uncertainty induced by few-shot tasks and impeding task-specific self-adaptation initialization. In response to this challenge, this article proposes a novel probabilistic meta-learning approach called prototype Bayesian meta-learning (PBML). PBML focuses on meta-learning variational posteriors within a Bayesian framework, guided by prototype-conditioned prior information. Specifically, to capture model uncertainty, PBML treats both meta-and task-specific parameters as random variables and integrates their posterior estimates into hierarchical Bayesian modeling through variational inference (VI). During model inference, PBML employs Laplacian estimation to approximate the integral term over the likelihood loss, deriving a rigorous upper-bound for generalization errors. To enhance the model's expressiveness and enable task-specific adaptive initialization, PBML proposes a data-driven approach to model the task-specific variational posteriors. This is achieved by designing a generative model structure that incorporates prototype-conditioned task-dependent priors into the random generation of task-specific variational posteriors. Additionally, by performing latent embedding optimization, PBML decouples the gradient-based meta-learning from the high-dimensional variational parameter space. Experimental results on benchmark datasets for few-shot image classification illustrate that PBML attains state-of-the-art or competitive performance when compared to other related works. Versatility studies demonstrate the adaptability and applicability of PBML in addressing diverse and challenging few-shot tasks. Furthermore, ablation studies validate the performance gains attributed to the inference and model components.

Using a Neural Network to Approximate the Negative Log Likelihood Function

An increasingly frequent challenge faced in HEP data analysis is to characterize the agreement between a prediction that depends on a dozen or more model parameters—such as predictions coming from an effective field theory (EFT) framework—and the observed data. Traditionally, such characterizations take the form of a negative log likelihood (NLL) function, which can only be evaluated numerically. The lack of a closed-form description of the NLL function makes it difficult to convey results of the statistical analysis. Typical results are limited to extracting “best fit” values of the model parameters and 1D intervals or 2D contours extracted from scanning the higher dimensional parameter space. It is desirable to explore these high-dimensional model parameter spaces in more sophisticated ways. One option for overcoming this challenge is to use a neural network to approximate the NLL function. This approach has the advantage of being continuous and differentiable by construction, which are essential properties for an NLL function and may also provide useful handles in exploring the NLL as a function of the model parameters. In this talk, we describe the advantages and limitations of this approach in the context of applying it to a CMS data analysis using the framework of EFT.

Scalable stellar evolution forecasting

Many astrophysical applications require efficient yet reliable forecasts of stellar evolution tracks. One example is population synthesis, which generates forward predictions of models for comparison with observations. The majority of state-of-the-art rapid population synthesis methods are based on analytic fitting formulae to stellar evolution tracks that are computationally cheap to sample statistically over a continuous parameter range. The computational costs of running detailed stellar evolution codes, such as MESA, over wide and densely sampled parameter grids are prohibitive, while stellar-age based interpolation in-between sparsely sampled grid points leads to intolerably large systematic prediction errors. In this work, we provide two solutions for automated interpolation methods that offer satisfactory trade-off points between cost-efficiency and accuracy. We construct a timescale-adapted evolutionary coordinate and use it in a two-step interpolation scheme that traces the evolution of stars from zero age main sequence all the way to the end of core helium burning while covering a mass range from 0.65 to 300 M⊙. The feedforward neural network regression model (first solution) that we train to predict stellar surface variables can make millions of predictions, sufficiently accurate over the entire parameter space, within tens of seconds on a 4-core CPU. The hierarchical nearest-neighbor interpolation algorithm (second solution) that we hard-code to the same end achieves even higher predictive accuracy, the same algorithm remains applicable to all stellar variables evolved over time, but it is two orders of magnitude slower. Our methodological framework is demonstrated to work on the MESA ISOCHRONES AND STELLAR TRACKS (Choi et al. 2016) data set, but is independent of the input stellar catalog. Finally, we discuss the prospective applications of these methods and provide guidelines for generalizing them to higher dimensional parameter spaces.