Toy Problem Research Articles

Multi-Objective Deep Reinforcement Learning for Crowd Route Guidance Optimization

In this study, we propose an improved version of Pareto deep Q-network (PDQN), a multi-objective deep reinforcement learning method, and attempt to demonstrate its effectiveness in a real-world problem such as crowd route guidance strategy optimization. Overcrowding during crowd movement can sometimes lead to accidents; therefore, it is imperative to guide crowds to move safely and efficiently. Safety and efficiency are conflicting objectives, and how to dynamically determine guidance can be formulated as a multi-objective sequential decision-making problem. PDQN is suitable for solving these problems, but its applicability to complex real-world problems has not been fully verified. To apply PDQN to real problems, we propose to adjust the parameters of PDQN and improve the action selection criteria during learning. A toy problem and a crowd guidance problem using multi-agent crowd simulation are adopted to evaluate the performance of improved PDQN. Experimental results show that the improved PDQN can search for good strategies compared with the original PDQN and can obtain better strategies than simplified strategies.

Modifications of generalized response surface methodology for constrained stochastic optimization problems

To optimize problems with a stochastic objective function and some stochastic or deterministic constraints, the generalized response surface methodology (GRSM) is used. This study presents a novel GRSM that is fast and accurate enough to solve constrained stochastic optimization problems containing expensive simulations or too costly real experiments. Basic GRSM is a kind of interior point method which moves toward the optimum from only inside of the feasible region. However, the proposed approach can move toward the optimum from inside or outside the feasible region. In addition, basic GRSM is not always reliable and fails in problems including complicated stochastic functions; we improved the GRSM to enable it to make reliable solutions to complicated problems. We solve two examples: 1- a toy problem with two stochastic constraints, and 2- an optimization problem of inventory system with one stochastic constraint. The obtained results indicate the optimization method proposed in this study is much more efficient than OPT-QUEST and classic GRSM.

Online Planning for Constrained POMDPs with Continuous Spaces through Dual Ascent

Rather than augmenting rewards with penalties for undesired behavior, Constrained Partially Observable Markov Decision Processes (CPOMDPs) plan safely by imposing inviolable hard constraint value budgets. Previous work performing online planning for CPOMDPs has only been applied to discrete action and observation spaces. In this work, we propose algorithms for online CPOMDP planning for continuous state, action, and observation spaces by combining dual ascent with progressive widening. We empirically compare the effectiveness of our proposed algorithms on continuous CPOMDPs that model both toy and real-world safety-critical problems. Additionally, we compare against the use of online solvers for continuous unconstrained POMDPs that scalarize cost constraints into rewards and highlight the limitations of the default exploration scheme.

Efficient prediction of turbulent flow quantities using a Bayesian hierarchical multifidelity model

Multifidelity models (MFMs) can be used to construct predictive models for flow quantities of interest (QoIs) over the space of uncertain/design parameters, with the purpose of uncertainty quantification, data fusion and optimization. For numerical simulation of turbulence, there is a hierarchy of methodologies ranked by accuracy and cost, where each methodology may have several numerical/modelling parameters that control the predictive accuracy and robustness of its resulting outputs. Compatible with these specifications, the present hierarchical MFM strategy allows for simultaneous calibration of the fidelity-specific parameters in a Bayesian framework as developed by Gohet al.(Technometrics, vol. 55, no. 4, 2013, pp. 501–512). The purpose of the MFM is to provide an improved prediction, mainly interpolation over the range covered by training data, by combining lower- and higher-fidelity data in an optimal way for any number of fidelity levels; even providing confidence intervals for the resulting QoI. The capabilities of the MFM are first demonstrated on an illustrative toy problem, and it is then applied to three realistic cases relevant to engineering turbulent flows. The latter include the prediction of friction at different Reynolds numbers in turbulent channel flow, the prediction of aerodynamic coefficients for a range of angles of attack of a standard airfoil and the uncertainty propagation and sensitivity analysis of the separation bubble in the turbulent flow over periodic hills subject to geometrical uncertainties. In all cases, based on only a few high-fidelity data samples, the MFM leads to accurate predictions of the QoIs. The result of the uncertainty quantification and sensitivity analyses are also found to be accurate compared with the ground truth in each case.

Kinetics parameter optimization of hydrocarbon fuels via neural ordinary differential equations

Chemical kinetics mechanisms are essential for understanding, analyzing, and simulating complex combustion phenomena. In this study, a neural ordinary differential equation (Neural ODE) framework is employed to optimize the kinetics parameters of reaction mechanisms. Given experimental or high-cost simulated observations as training data, the proposed algorithm can optimally recover the hidden characteristics in the data. Different datasets of various sizes, types, and noise levels are systematically tested. A classic toy problem of stiff Robertson ODE is first used to demonstrate the learning capability, efficiency, and robustness of the Neural ODE approach. A 41-species, 232-reactions JP-10 skeletal mechanism and a 34-species, 121-reactions n-heptane skeletal mechanism are then optimized with species' temporal profiles and ignition delay times, respectively. Results show that the proposed algorithm can optimize stiff chemical models with sufficient accuracy, efficiency and robustness. It is noted that the trained mechanism not only fits the data perfectly but also retains its physical interpretability, which can be further integrated and validated in practical turbulent combustion simulations. In addition, as demonstrated with the stiff Robertson problem, it is promising to adopt Bayesian inference techniques with Neural ODE to estimate the kinetics parameter uncertainties from experimental data.

Exploiting Symmetry in Variational Quantum Machine Learning

Variational quantum machine learning is an extensively studied application of near-term quantum computers. The success of variational quantum learning models crucially depends on finding a suitable parametrization of the model that encodes an inductive bias relevant to the learning task. However, precious little is known about guiding principles for the construction of suitable parametrizations. In this work, we holistically explore when and how symmetries of the learning problem can be exploited to construct quantum learning models with outcomes invariant under the symmetry of the learning task. Building on tools from representation theory, we show how a standard gateset can be transformed into an equivariant gateset that respects the symmetries of the problem at hand through a process of gate symmetrization. We benchmark the proposed methods on two toy problems that feature a non-trivial symmetry and observe a substantial increase in generalization performance. As our tools can also be applied in a straightforward way to other variational problems with symmetric structure, we show how equivariant gatesets can be used in variational quantum eigensolvers.

Learning From Guided Play: Improving Exploration for Adversarial Imitation Learning With Simple Auxiliary Tasks

Adversarial imitation learning (AIL) has become a popular alternative to supervised imitation learning that reduces the distribution shift suffered by the latter. However, AIL requires effective exploration during an online reinforcement learning phase. In this work, we show that the standard, naïve approach to exploration can manifest as a suboptimal local maximum if a policy learned with AIL sufficiently matches the expert distribution without fully learning the desired task. This can be particularly catastrophic for manipulation tasks, where the difference between an expert and a non-expert state-action pair is often subtle. We present Learning from Guided Play (LfGP), a framework in which we leverage expert demonstrations of multiple exploratory, auxiliary tasks in addition to a main task. The addition of these auxiliary tasks forces the agent to explore states and actions that standard AIL may learn to ignore. Additionally, this particular formulation allows for the reusability of expert data between main tasks. Our experimental results in a challenging multitask robotic manipulation domain indicate that LfGP significantly outperforms both AIL and behaviour cloning, while also being more expert sample efficient than these baselines. To explain this performance gap, we provide further analysis of a toy problem that highlights the coupling between a local maximum and poor exploration, and also visualize the differences between the learned models from AIL and LfGP.

Quantum density peak clustering

Clustering algorithms are of fundamental importance when dealing with large unstructured datasets and discovering new patterns and correlations therein, with applications ranging from scientific research to medical imaging and marketing analysis. In this work, we introduce a quantum version of the density peak clustering algorithm, built upon a quantum routine for minimum finding. We prove a quantum speedup for a decision version of density peak clustering depending on the structure of the dataset. Specifically, the speedup is dependent on the heights of the trees of the induced graph of nearest-highers, i.e. the graph of connections to the nearest elements with higher density. We discuss this condition, showing that our algorithm is particularly suitable for high-dimensional datasets. Finally, we benchmark our proposal with a toy problem on a real quantum device.

Risk-averse optimization of reward-based coherent risk measures

In real-world problems such as robotics, finance and healthcare, randomness is always present, thus, it is important to take risk into consideration in order to limit the chance of rare but dangerous events. The literature on risk-averse reinforcement learning has produced many different approaches to tackle the problem, but they either struggle to scale up to complex instances, or they exhibit irrational behaviors. Here we present two novel risk-averse objectives that are both coherent and easy to optimize: the reward-based mean-mean absolute deviation (Mean-RMAD) and the reward-based conditional value at risk (RCVaR). Instead of reducing the return risk, these measures minimize the per-step reward one. We prove that these risk measures bound the corresponding return-based risk measures, so that they can be also used as proxies for their return-based versions. We develop safe algorithms for these risk measures with guaranteed monotonic improvement, and their practical trust-region versions. Furthermore, we propose a decomposition for the RCVaR optimization problem into a sequence of risk-neutral problems. Finally, we conduct an empirical analysis on the introduced approaches, demonstrating their effectiveness in retrieving a variety of risk-averse behaviors on both toy problems and more challenging ones, such as a simulated trading environment and robotic locomotion tasks.

Sparse Stable Outlier-Robust Signal Recovery Under Gaussian Noise

This paper presents a novel framework for sparse robust signal recovery integrating the sparse recovery using the minimax concave (MC) penalty and robust regression called sparse outlier-robust regression (SORR) using the MC loss. While the proposed approach is highly robust against huge outliers, the sparseness of estimates can be controlled by taking into consideration a tradeoff between sparseness and robustness. To accommodate the prior information about additive Gaussian noise and outliers, an auxiliary vector to model the noise is introduced. The remarkable robustness and stability come from the use of the MC loss and the squared <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ell _{2}$</tex-math></inline-formula> penalty of the noise vector, respectively. In addition, the simultaneous use of the MC and squared <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ell _{2}$</tex-math></inline-formula> penalties of the coefficient vector leads to a certain remarkable grouping effect. The necessary and sufficient conditions for convexity of the smooth part of the cost are derived under a certain nonempty-interior assumption via the product space formulation using the linearly-involved Moreau-enhanced-over-subspace (LiMES) framework. The efficacy of the proposed method is demonstrated by simulations in its application to speech denoising under highly noisy environments as well as to toy problems.

Hybrid Polynomial Chaos Expansion and proper generalized decomposition approach for uncertainty quantification problems in the frame of elasticity

Polynomial Chaos Expansion (PCE) is a powerful and flexible metamodeling technique, but suffers from the fact that the number of required training samples grows exponentially with the dimensionality of the problem. Recently, proper generalized decomposition (PGD) has become a popular way to address this challenge since its complexity grows linearly with the dimension of the problem. In this work, we present a new technique called PGD–PCE, which combines the advantages of both methods. The algorithm is based on separate representations and constructed by orthonormal polynomial functions. We test the proposed approach on simple toy problems and engineering finite element problems. The results show that PGD–PCE performs well in terms of accuracy and computational efficiency when dealing with large problems. • We propagate uncertainties in mechanical problems. • We develop a simple to implement and non-intrusive method. • We show that accuracy is preserved while cost drastically decrease. • We illustrate the capabilities of the approach on various numerical tests.

How and why non smooth solutions of the 3D Navier–Stokes equations could possibly develop

Fluid configurations in three-dimensions, displaying a plausible decay of regularity in a finite time, are suitably built and examined. Vortex rings are the primary ingredients in this study. The full Navier–Stokes system is converted into a 3D scalar problem, where appropriate numerical methods are implemented in order to figure out the behavior of the solutions. Further simplifications in 2D and 1D provide interesting toy problems, that may be used as a starting platform for a better understanding of blowup phenomena.

Bayesian probability updates using sampling/importance resampling: Applications in nuclear theory

We review an established Bayesian sampling method called sampling/importance resampling and highlight situations in nuclear theory when it can be particularly useful. To this end we both analyse a toy problem and demonstrate realistic applications of importance resampling to infer the posterior distribution for parameters of ΔNNLO interaction model based on chiral effective field theory and to estimate the posterior probability distribution of target observables. The limitation of the method is also showcased in extreme situations where importance resampling breaks.

A Self-learning Neural Network Approach for Radio Frequency Interference Detection and Removal in Radio Astronomy

We present a novel neural network (NN) method for the detection and removal of radio frequency interference (RFI) from the raw digitized signal in the signal processing chain of a typical radio astronomy experiment. The main advantage of our method is that it does not require a training set. Instead, our method relies on the fact that the true signal of interest coming from astronomical sources is thermal and therefore described as a Gaussian random process, which cannot be compressed. We employ a variational encoder/decoder network to find the compressible information in the data stream that can explain the most variance with the fewest degrees of freedom. We demonstrate it on a set of toy problems and stored ring buffers from the Baryon Mapping eXperiment prototype. We find that the RFI subtraction is effective at cleaning simulated time streams: While we find that the power spectra of the RFI-cleaned time streams output by the NN suffer from extra signal consistent with additive noise, we find that it is generally around percent level across the band and sub 10% in contaminated spectral channels even when RFI power is an order of magnitude larger than the signal. We discuss the advantages and limitations of this method and possible implementation in the front end of future radio experiments.

Active Learning for Computationally Efficient Distribution of Binary Evolution Simulations

Binary stars undergo a variety of interactions and evolutionary phases, critical for predicting and explaining observations. Binary population synthesis with full simulation of stellar structure and evolution is computationally expensive, requiring a large number of mass-transfer sequences. The recently developed binary population synthesis code POSYDON incorporates grids of MESA binary star simulations that are interpolated to model large-scale populations of massive binaries. The traditional method of computing a high-density rectilinear grid of simulations is not scalable for higher-dimension grids, accounting for a range of metallicities, rotation, and eccentricity. We present a new active learning algorithm, psy-cris, which uses machine learning in the data-gathering process to adaptively and iteratively target simulations to run, resulting in a custom, high-performance training set. We test psy-cris on a toy problem and find the resulting training sets require fewer simulations for accurate classification and regression than either regular or randomly sampled grids. We further apply psy-cris to the target problem of building a dynamic grid of MESA simulations, and we demonstrate that, even without fine tuning, a simulation set of only ∼1/4 the size of a rectilinear grid is sufficient to achieve the same classification accuracy. We anticipate further gains when algorithmic parameters are optimized for the targeted application. We find that optimizing for classification only may lead to performance losses in regression, and vice versa. Lowering the computational cost of producing grids will enable new population synthesis codes such as POSYDON to cover more input parameters while preserving interpolation accuracies.

Learning and Executing Re-Usable Behaviour Trees From Natural Language Instruction

Domestic and service robots have the potential to transform industries such as health care and small-scale manufacturing, as well as the homes in which we live. However, due to the overwhelming variety of tasks these robots will be expected to complete, providing generic out-of-the-box solutions that meet the needs of every possible user is clearly intractable. To address this problem, robots must therefore not only be capable of learning how to complete novel tasks at run-time, but the solutions to these tasks must also be informed by the needs of the user. In this letter we demonstrate how behaviour trees, a well established control architecture in the fields of gaming and robotics, can be used in conjunction with natural language instruction to provide a robust and modular control architecture for instructing autonomous agents to learn and perform novel complex tasks. We also show how behaviour trees generated using our approach can be generalised to novel scenarios, and can be re-used in future learning episodes to create increasingly complex behaviours. We validate this work against an existing corpus of natural language instructions, demonstrate the application of our approach on both a simulated robot solving a toy problem, as well as two distinct real-world robot platforms which, respectively, complete a block sorting scenario, and a patrol scenario.

Assessment of DeepONet for time dependent reliability analysis of dynamical systems subjected to stochastic loading

Time dependent reliability analysis and uncertainty quantification of structural system subjected to stochastic forcing function is a challenging endeavour as it necessitates considerable computational time. We investigate the efficacy of recently proposed DeepONet in solving time dependent reliability analysis and uncertainty quantification of systems subjected to stochastic loading. Unlike conventional machine learning and deep learning algorithms, DeepONet is an operator network and learns a function to function mapping and hence, is ideally suited to propagate the uncertainty from the stochastic forcing function to the output responses. We use DeepONet to build a surrogate model for the dynamical system under consideration. Multiple case studies, involving both toy and benchmark problems, have been conducted to examine the efficacy of DeepONet in time dependent reliability analysis and uncertainty quantification of linear and nonlinear dynamical systems. Comparisons have also been drawn with Recurrent Neural Network results and with results obtained from Proper Orthogonal Decomposition based Gaussian process. The results obtained indicate that the DeepONet architecture is accurate as well as efficient. Moreover, DeepONet posses zero shot learning capabilities and hence, a trained model easily generalizes to unseen and new environment with no further training.

Computational investigation into the effect of constitutive formulations for flow modeling of high elasticity

• Three constitutive formulations of the Leonov model are numerically investigated. • It is verified that keeping positive-definiteness of the conformation tensor is crucial for computational evolution while appropriate interpolation of exponentially convecting domain is essential for accuracy. • Swirling asymmetric corner vortices are described in 3D contraction flow modeling. For the purpose of defeating the high Deborah (or Weissenberg) number problems in viscoelastic flow modeling, the matrix-logarithmic formulation of the constitutive equations has been suggested and proven probably the most effective. This transformation of the conformation tensor endows proper resolution of the boundary layer with exponential convection as well as conservation of its positive definiteness. Herein we make an attempt to evaluate numerical consequences of these two features independently for the Leonov viscoelastic model mainly in the contraction flow. To this end, we from the viewpoint of computational stability and accuracy investigate three formulations written with variables such as the original conformation tensor c (c form), its matrix-logarithm (h form) and additive decomposition (b form). 2D planar 4:1 contraction flow in the limit of vanishing inertia in addition to its 1D toy problem is considered in the framework of finite element method. The h and b forms are designed to preserve the positive definiteness of c , and furthermore the h form grants efficient interpolation of exponentially convecting variables. The 1D analogy expresses the characteristics qualitatively equivalent to the original problem in a simpler fashion. The c form modeling of 2D flow exhibits typical nonevolutionary behavior when the Deborah number ( De ) exceeds a certain limit, and this degradation gets worse with finer mesh. While the h and b forms have always been computationally stable with proper description of steady behavior or elastic instability otherwise, the h form manifests faster mesh convergence. As a complementary study, the h form calculation of full 3D flow along an axisymmetric contraction channel has been performed, which also attests well-posedness without upper bound in De . Its steady result is corroborated with the one under axisymmetric flow approximation. Loss of convergence in the axisymmetric flow regime at high De is verified to result from the occurrence of 3D instability, which is then well portrayed in 3D modeling. In this instability, swirling of asymmetric structure for the corner vortices about the geometrical symmetry axis plausibly mimics experimental observation. Consequently, maintaining the positive definiteness in terms of h or b form is crucial for numerically stable evolution, and the h form interpolation delineating appropriately the exponential convection assures the fastest mesh convergence, hence enhancing the accuracy.

Co-Learning Bayesian Optimization.

Bayesian optimization (BO) is well known to be sample efficient for solving black-box problems. However, BO algorithms may get stuck in suboptimal solutions even with plenty of samples. Intrinsically, such a suboptimal problem of BO can attribute to the poor surrogate accuracy of the trained Gaussian process (GP), particularly that in the regions where the optimal solutions locate. Hence, we propose to build multiple GP models instead of a single GP surrogate to complement each other, thus resolving the suboptimal problem of BO. Nevertheless, according to the bias-variance tradeoff equation, the individual prediction errors can increase when increasing the diversity of models, which may lead even worse overall surrogate accuracy. On the other hand, based on the theory of the Rademacher complexity, it has been proven that exploiting the agreement of models on unlabeled information can reduce the complexity of hypothesis space, therefore achieving the required surrogate accuracy with fewer samples. Such value of model agreement has been extensively demonstrated for co-training style algorithms to boost model accuracy with a small portion of samples. Inspired by the above, we propose a novel BO algorithm labeled as co-learning BO (CLBO), which exploits both model diversity and agreement on unlabeled information to improve the overall surrogate accuracy with limited samples, therefore achieving more efficient global optimization. Through tests on five numerical toy problems and three engineering benchmarks, the effectiveness of the proposed CLBO has been well demonstrated.

Modal analysis and interface tracking of multiphase flows using Dynamic Mode Decomposition

Non-invasive visualization techniques for multiphase flows are critical to understanding primary atomization and sprays. We use back-lit imaging to identify the liquid–gas interface of two-phase flows at high temporal and spatial resolutions and employ Dynamic Mode Decomposition (DMD) to study the shape and frequency of instabilities of a liquid jet surrounded by a coaxial annular airblast atomizer. However, DMD is not suitable for interface tracking, so we develop a data-driven two-step approach using the optical sensor data. The method uses DMD on the optical flow field estimated from image snapshot pairs. We demonstrate our method to a representative toy problem of an oscillating drop and on the primary atomization of a numerical planar liquid jet. Finally, we apply our method to the experimental liquid jet from the coaxial airblast atomizer using back-lit imaging. Our method is able to accurately reconstruct and predict the flow and preserves the sharpness of the fluid interface.