Initial State Distribution Research Articles

Global Optimality Guarantees for Policy Gradient Methods

Policy gradient methods, which have powered a lot of recent success in reinforcement learning, search for an optimal policy in a parameterized policy class by performing stochastic gradient descent on the cumulative expected cost-to-go under some initial state distribution. Although widely used, these methods lack theoretical guarantees as the optimization objective is typically nonconvex even for simple control problems, and hence are understood to only converge to a stationary point. In “Global Optimality Guarantees for Policy Gradient Methods,” J. Bhandari and D. Russo identify structural properties of the underlying MDP that guarantee that despite nonconvexity, the optimization objective has no suboptimal stationary points, ensuring asymptotic convergence of policy gradient methods to globally optimal policies. Under stronger conditions, authors show the policy gradient objective to satisfy a Polyak-lojasiewicz (gradient dominance) condition that yields fast convergence rates. In addition, when some of the said conditions are relaxed, authors provide bounds on the suboptimality gap of any stationary point. The results rely on a key connection with policy iteration, a classic dynamic programming algorithm which solves a single period optimization problem at every step. The authors show how structure in the single period optimization problems solved by policy iteration translate into nice properties of the multiperiod policy gradient objective, making it amenable for first-order methods to find globally optimal solutions. The authors also instantiate their framework for several classical control problems including tabular and linear MDPs, linear quadratic control, optimal stopping, and finite-horizon inventor control problems.

A Multi-Agent Reinforcement Learning Method for Omnidirectional Walking of Bipedal Robots.

Achieving omnidirectional walking for bipedal robots is considered one of the most challenging tasks in robotics technology. Reinforcement learning (RL) methods have proved effective in bipedal walking tasks. However, most existing methods use state machines to switch between multiple policies and achieve omnidirectional gait, which results in shaking during the policy switching process for bipedal robots. To achieve a seamless transition between omnidirectional gait and transient motion for full-size bipedal robots, we propose a novel multi-agent RL method. Firstly, a multi-agent RL algorithm based on the actor-critic framework is designed, and policy entropy is introduced to improve exploration efficiency. By learning agents with parallel initial state distributions, we minimize reliance on gait planner effectiveness in the Robot Operating System (ROS). Additionally, we design a novel heterogeneous policy experience replay mechanism based on Euclidean distance. Secondly, considering the periodicity of bipedal robot walking, we develop a new periodic gait function. Including periodic objectives in the policy can accelerate the convergence speed of training periodic gait functions. Finally, to enhance the robustness of the policy, we construct a novel curriculum learning method by discretizing Gaussian distribution and incorporate it into the robot's training task. Our method is validated in a simulation environment, and the results show that our method can achieve multiple gaits through a policy network and achieve smooth transitions between different gaits.

Interpretable Policies and the Price of Interpretability in Hypertension Treatment Planning

Problem definition: Effective hypertension management is critical to reducing the consequences of atherosclerotic cardiovascular disease, a leading cause of death in the United States. Clinical guidelines for hypertension can be enhanced using decision-analytic approaches capable of capturing complexities in treatment planning. However, model-generated recommendations may be uninterpretable/unintuitive, limiting their clinical acceptability. We address this challenge by investigating interpretable treatment plans. Methodology/results: We formulate interpretable treatment plans as Markov decision processes (MDPs) and analyze the problems of optimizing monotone policies, which prohibit decreasing treatment intensity for sicker patients, and class-ordered monotone policies, which generalize monotone policies. We establish that both policies depend on initial state distributions and that optimal monotone policies can be generated tractably for many treatment planning problems. Next, we propose exact formulations for optimizing interpretable policies broadly. Then, we analyze the price of interpretability, proving that the class-ordered monotone policy’s price of interpretability does not exceed the monotone policy’s price of interpretability. Finally, we formulate and evaluate MDPs for hypertension treatment planning using a large nationally representative data set of the U.S. population. We compare the structure and performance of optimal monotone policies and class-ordered monotone policies with optimal MDP-based policies and current clinical guidelines. At the patient level, optimal MDP-based policies may be unintuitive, recommending more aggressive treatment for healthier patients than sicker patients. Conversely, monotone policies and class-ordered monotone policies never deescalate treatment, reflecting clinical intuition. Across 66.5 million patients, optimized monotone policies and class-ordered monotone policies outperform clinical guidelines, saving over 3,246 quality-adjusted life years per 100,000 patients, with both policies paying a low price of interpretability. Sensitivity analysis illustrates that monotone policies and class-ordered monotone policies are robust to various definitions of “interpretability.” Managerial implications: Interpretable policies can be tractably optimized, drastically outperform existing guidelines, and perform near optimally—potentially increasing the acceptability of decision-analytic approaches in practice. Funding: L. N. Steimle and W. J. Marrero received support from the National Science Foundation Graduate Research Fellowship [Grant DGE 1256260]. J. B. Sussman received support from the National Institutes of Health [Grants R01NS102715 and RF1AG068410], the U.S. Department of Veterans Affairs [Grants 1I01-HX003304 and 1I50-HX003251], and the Michigan Department of Health and Human Services. Supplemental Material: The e-companion is available at https://doi.org/10.1287/msom.2021.0373 .

Distributed consensus-based multi-agent temporal-difference learning

In this paper we propose two new distributed consensus-based algorithms for temporal-difference learning in multi-agent Markov decision processes. The algorithms are of off-policy type and are aimed at linear approximation of the value function. Restricting agents’ observations to local data and communications to their small neighborhoods, the algorithms consist of: (a) local updates of the parameter estimates based on either the standard TD(λ) or the emphatic ETD(λ) algorithm, and (b) dynamic consensus scheme implemented over a time-varying lossy communication network. The algorithms are completely decentralized, allowing efficient parallelization and applications where the agents may have different behavior policies and different initial state distributions while evaluating a common target policy. It is proved under nonrestrictive assumptions that the proposed algorithms weakly converge to the solutions of the mean ordinary differential equation (ODE) common for all the agents. It is also proved that the whole system may be stabilized by a proper choice of the network and that the parameter estimates weakly converge to consensus. Discussion is given on the asymptotic bias and variance of the estimates, on the projected forms of the proposed algorithms, as well as on restrictiveness of the adopted assumptions. Simulation results illustrate the main properties of the algorithms and provide comparisons with similar schemes.

Stochastic Magnetic Actuated Random Transducer Devices Based on Perpendicular Magnetic Tunnel Junctions

True random number generators are of great interest in many computing applications such as cryptography, neuromorphic systems and Monte Carlo simulations. Here we investigate perpendicular magnetic tunnel junction nanopillars (pMTJs) activated by short duration (ns) pulses in the ballistic limit for such applications. In this limit, a pulse can transform the Boltzmann distribution of initial free layer magnetization states into randomly magnetized down or up states, i.e. a bit that is 0 or 1, easily determined by measurement of the junction's tunnel resistance. It is demonstrated that bitstreams with millions of events: 1) are very well described by the binomial distribution; 2) can be used to create a uniform distribution of 8-bit random numbers; 3) pass multiple statistical tests for true randomness, including all the National Institute of Standards tests for random number generators with only one XOR operation; and 4) can have no drift in the bit probability with time. The results presented here show that pMTJs operated in the ballistic regime can generate true random numbers at GHz bitrates, while being more robust to environmental changes, such as their operating temperature, compared to other stochastic nanomagnetic devices.

Impact of Salinity on the Energy Transfer between Pigment-Protein Complexes in Photosynthetic Apparatus, Functions of the Oxygen-Evolving Complex and Photochemical Activities of Photosystem II and Photosystem I in Two Paulownia Lines.

The present study shows the effect of salinity on the functions of thylakoid membranes from two hybrid lines of Paulownia: Paulownia tomentosa x fortunei and Paulownia elongate x elongata, grown in a Hoagland solution with two NaCl concentrations (100 and 150 mM) and different exposure times (10 and 25 days). We observed inhibition of the photochemical activities of photosystem I (DCPIH2 → MV) and photosystem II (H2O → BQ) only after the short treatment (10 days) with the higher NaCl concentration. Data also revealed alterations in the energy transfer between pigment-protein complexes (fluorescence emission ratios F735/F685 and F695/F685), the kinetic parameters of the oxygen-evolving reactions (initial S0-S1 state distribution, misses (α), double hits (β) and blocked centers (SB)). Moreover, the experimental results showed that after prolonged treatment with NaCl Paulownia tomentosa x fortunei adapted to the higher concentration of NaCl (150 mM), while this concentration is lethal for Paulownia elongata x elongata. This study demonstrated the relationship between the salt-induced inhibition of the photochemistry of both photosystems and the salt-induced changes in the energy transfer between the pigment-protein complexes and the alterations in the Mn cluster of the oxygen-evolving complex under salt stress.

Softmax policy gradient methods can take exponential time to converge

AbstractThe softmax policy gradient (PG) method, which performs gradient ascent under softmax policy parameterization, is arguably one of the de facto implementations of policy optimization in modern reinforcement learning. For$$\gamma $$γ-discounted infinite-horizon tabular Markov decision processes (MDPs), remarkable progress has recently been achieved towards establishing global convergence of softmax PG methods in finding a near-optimal policy. However, prior results fall short of delineating clear dependencies of convergence rates on salient parameters such as the cardinality of the state space$${\mathcal {S}}$$Sand the effective horizon$$\frac{1}{1-\gamma }$$11-γ, both of which could be excessively large. In this paper, we deliver a pessimistic message regarding the iteration complexity of softmax PG methods, despite assuming access to exact gradient computation. Specifically, we demonstrate that the softmax PG method with stepsize$$\eta $$ηcan take$$\begin{aligned} \frac{1}{\eta } |{\mathcal {S}}|^{2^{\Omega \big (\frac{1}{1-\gamma }\big )}} ~\text {iterations} \end{aligned}$$1η|S|2Ω(11-γ)iterationsto converge, even in the presence of a benign policy initialization and an initial state distribution amenable to exploration (so that the distribution mismatch coefficient is not exceedingly large). This is accomplished by characterizing the algorithmic dynamics over a carefully-constructed MDP containing only three actions. Our exponential lower bound hints at the necessity of carefully adjusting update rules or enforcing proper regularization in accelerating PG methods.

A Neural Network Approach for High-Dimensional Optimal Control Applied to Multiagent Path Finding

We propose a neural network approach that yields approximate solutions for high-dimensional optimal control problems and demonstrate its effectiveness using examples from multi-agent path finding. Our approach yields controls in a feedback form, where the policy function is given by a neural network (NN). Specifically, we fuse the Hamilton-Jacobi-Bellman (HJB) and Pontryagin Maximum Principle (PMP) approaches by parameterizing the value function with an NN. Our approach enables us to obtain approximately optimal controls in real-time without having to solve an optimization problem. Once the policy function is trained, generating a control at a given space-time location takes milliseconds; in contrast, efficient nonlinear programming methods typically perform the same task in seconds. We train the NN offline using the objective function of the control problem and penalty terms that enforce the HJB equations. Therefore, our training algorithm does not involve data generated by another algorithm. By training on a distribution of initial states, we ensure the controls' optimality on a large portion of the state-space. Our grid-free approach scales efficiently to dimensions where grids become impractical or infeasible. We apply our approach to several multi-agent collision-avoidance problems in up to 150 dimensions. Furthermore, we empirically observe that the number of parameters in our approach scales linearly with the dimension of the control problem, thereby mitigating the curse of dimensionality.

A Data-Efficient Training Method for Deep Reinforcement Learning

Data inefficiency is one of the major challenges for deploying deep reinforcement learning algorithms widely in industry control fields, especially in regard to long-horizon sparse reward tasks. Even in a simulation-based environment, it is often prohibitive to take weeks to train an algorithm. In this study, a data-efficient training method is proposed in which a DQN is used as a base algorithm, and an elaborate curriculum is designed for the agent in the simulation scenario to accelerate the training process. In the early stage of the training process, the distribution of the initial state is set close to the goal so the agent can obtain an informative reward easily. As the training continues, the initial state distribution is set farther from the goal for the agent to explore more state space. Thus, the agent can obtain a reasonable policy through fewer interactions with the environment. To bridge the sim-to-real gap, the parameters for the output layer of the neural network for the value function are fine-tuned. An experiment on UAV maneuver control is conducted in the proposed training framework to verify the method. We demonstrate that data efficiency is different for the same data in different training stages.

Hybrid discrete-continuous truncated Wigner approximation for driven, dissipative spin systems

We present a systematic approach for the semiclassical treatment of many-body dynamics of interacting, open spin systems. Our approach overcomes some of the shortcomings of the recently developed discrete truncated Wigner approximation (DTWA) based on Monte-Carlo sampling in a discrete phase space that improves the classical treatment by accounting for lowest-order quantum fluctuations. We provide a rigorous derivation of the DTWA by embedding it in a continuous phase space, thereby introducing a hybrid discrete-continuous truncated Wigner approximation (DCTWA). We derive a set of operator-differential mappings that yield an exact equation of motion (EOM) for the continuous SU(2) Wigner function of spins. The standard DTWA is then recovered by a systematic neglection of specific terms in this exact EOM. The hybrid approach permits us to determine the validity conditions and to gain detailed understanding of the quality of the approximation, paving the way for systematic improvements. Furthermore, we show that the continuous embedding allows for a straightforward extension of the method to open spin systems subject to dephasing, losses and incoherent drive, while preserving the key advantages of the discrete approach, such as a positive definite Wigner distribution of typical initial states. We derive exact stochastic differential equations for processes which cannot be described by the standard DTWA due to the presence of non-classical noise. We illustrate our approach by applying it to the dissipative dynamics of Rydberg excitation of one-dimensional arrays of laser-driven atoms and compare it to exact results for small systems.

Assessing the Cost-effectiveness of a Hypothetical Disease-modifying Therapy With Limited Duration for the Treatment of Early Symptomatic Alzheimer Disease

PurposeClinical trials have produced promising results for disease-modifying therapies (DMTs) for Alzheimer’s disease (AD); however, the evidence on their potential cost-effectiveness is limited. This study assesses the cost-effectiveness of a hypothetical DMT with a limited treatment duration in AD. MethodsWe developed a Markov state–transition model to estimate the cost-effectiveness of a hypothetical DMT plus best supportive care (BSC) versus BSC alone among Americans living with mild cognitive impairment (MCI) due to AD or mild AD. AD states included MCI due to AD, mild AD, moderate AD, severe AD, and death. A hypothetical DMT was assumed to confer a 30% reduction in progression from MCI and mild AD. The base case annual drug acquisition cost was assumed to be $56,000. Other medical and indirect costs were obtained from published literature or list prices. Utilities for patients and caregivers were obtained from the published literature and varied by AD state and care setting (community care or long-term care). We considered 3 DMT treatment strategies: (1) treatment administered until patients reached severe AD (continuous strategy), (2) treatment administered for a maximum duration of 18 months or when patients reached severe AD (fixed-duration strategy), and (3) 40% of patients discontinuing treatment at 6 months because of amyloid plaque clearance and the remaining patients continuing treatment until 18 months or until they reached severe AD (test-and-discontinue strategy). Incremental cost-effectiveness ratios (ICERs) were calculated as the incremental cost per quality-adjusted life-year (QALY) gained. FindingsFrom the health care sector perspective, continuous treatment with a hypothetical DMT versus BSC resulted in an ICER of $612,354 per QALY gained. The ICER decreased to $157,288 per QALY gained in the fixed-duration strategy, driven by large reductions in treatment costs. With 40% of patients discontinuing treatment at 6 months (test-and-discontinue strategy), the ICER was $125,631 per QALY gained. In sensitivity and scenario analyses, the ICER was the most sensitive to changes in treatment efficacy, treatment cost, and the initial population AD state distribution. From the modified societal perspective, ICERs were 6.3%, 20.4%, and 25.1% lower than those from the health care sector perspective for the continuous, fixed-duration, and test-and-discontinue strategies, respectively. ImplicationsUnder a set of assumptions for annual treatment costs and the magnitude and duration of treatment efficacy, DMTs used for a limited duration may deliver value consistent with accepted US cost-effectiveness thresholds.

Automating Reinforcement Learning With Example-Based Resets

Deep reinforcement learning has enabled robots to learn motor skills from environmental interactions with minimal to no prior knowledge. However, existing reinforcement learning algorithms assume an episodic setting, in which the agent resets to a fixed initial state distribution at the end of each episode, to successfully train the agents from repeated trials. Such reset mechanism, while trivial for simulated tasks, can be challenging to provide for real-world robotics tasks. Resets in robotic systems often require extensive human supervision and task-specific workarounds, which contradicts the goal of autonomous robot learning. In this paper, we propose an extension to conventional reinforcement learning towards greater autonomy by introducing an additional agent that learns to reset in a self-supervised manner. The reset agent preemptively triggers a reset to prevent manual resets and implicitly imposes a curriculum for the forward agent. We apply our method to learn from scratch on a suite of simulated and real-world continuous control tasks and demonstrate that the reset agent successfully learns to reduce manual resets whilst also allowing the forward policy to improve gradually over time.

Constrained Discounted Stochastic Games

In this paper, we consider a large class of constrained non-cooperative stochastic Markov games with countable state spaces and discounted cost criteria. In one-player case, i.e., constrained discounted Markov decision models, it is possible to formulate a static optimisation problem whose solution determines a stationary optimal strategy (alias control or policy) in the dynamical infinite horizon model. This solution lies in the compact convex set of all occupation measures induced by strategies, defined on the set of state–action pairs. In case of n-person discounted games the occupation measures are induced by strategies of all players. Therefore, it is difficult to generalise the approach for constrained discounted Markov decision processes directly. It is not clear how to define the domain for the best-response correspondence whose fixed point induces a stationary equilibrium in the Markov game. This domain should be the Cartesian product of compact convex sets in a locally convex topological vector space. One of our main results shows how to overcome this difficulty and define a constrained non-cooperative static game whose Nash equilibrium induces a stationary Nash equilibrium in the Markov game. This is done for games with bounded cost functions and positive initial state distribution. An extension to a class of Markov games with unbounded costs and arbitrary initial state distribution relies on an approximation of the unbounded game by bounded ones with positive initial state distributions. In the unbounded case, we assume the uniform integrability of the discounted costs with respect to all probability measures induced by strategies of the players, defined on the space of plays (histories) of the game. Our assumptions are weaker than those applied in earlier works on discounted dynamic programming or stochastic games using the so-called weighted norm approach.

Machine learning product state distributions from initial reactant states for a reactive atom-diatom collision system.

A machine-learned model for predicting product state distributions from specific initial states (state-to-distribution or STD) for reactive atom-diatom collisions is presented and quantitatively tested for the N(4S) + O2(X3Σg -) → NO(X2Π) + O(3P) reaction. The reference dataset for training the neural network consists of final state distributions determined from quasi-classical trajectory (QCT) simulations for ∼2000 initial conditions. Overall, the prediction accuracy as quantified by the root-mean-squared difference (∼0.003) and the R2 (∼0.99) between the reference QCT and predictions of the STD model is high for the test set, for off-grid state-specific initial conditions, and for initial conditions drawn from reactant state distributions characterized by translational, rotational, and vibrational temperatures. Compared with a more coarse grained distribution-to-distribution (DTD) model evaluated on the same initial state distributions, the STD model shows comparable performance with the additional benefit of the state resolution in the reactant preparation. Starting from specific initial states also leads to a more diverse range of final state distributions, which requires a more expressive neural network compared with DTD. A direct comparison between QCT simulations, the STD model, and the widely used Larsen-Borgnakke (LB) model shows that the STD model is quantitative, whereas the LB model is qualitative at best for rotational distributions P(j') and fails for vibrational distributions P(v'). As such, the STD model can be well-suited for simulating nonequilibrium high-speed flows, e.g., using the direct simulation Monte Carlo method.

Formulation of temperature dependent effective Hartree potential incorporating quadratic over linear molecular DOFs-surface modes couplings and its effect on quantum dynamics of D2 (v = 0, j = 0)/D2 (v = 0, j = 2) on Cu(111) metal surface

The effect of surface temperature on the scattering and dissociation processes of D2 (v = 0, j = 0)/D2 (v = 0, j = 2) on Cu(111) metal plane is investigated by formulating a time and temperature dependent effective Hamiltonian within mean-field approach, where the molecular degrees of freedom and surface mode vibrations are assumed to be weakly coupled with each other. While constructing such effective Hamiltonian with a Hartree product type wavefunction, we incorporate: (i) an operator algebra method to circumscribe higher order-quadratic (anharmonic) molecular DOFs-surface modes couplings over the linear coupling case implemented previously [Adhikari and coworkers, J. Chem. Phys. 154, 104103 (2021)]; (ii) surface temperature by considering Bose–Einstein or Maxwell–Boltzmann probability factors for the initial state distribution of the surface vibrational modes. Finally, a six-dimensional (6D) quantum dynamical (QD) calculation is performed with such newly formulated effective Hartree potential and calculated transition/dissociation probabilities at various surface temperature situations. The quadratic (anharmonic) molecular DOFs-surface modes couplings modify the reaction and transition probabilities substantially with change of surface temperature, which indicates the crucial role of surface mode vibrations on the molecule-surface scattering phenomena.

Geant4 simulation of neutron displacement damage effect in InP

As the second-generation compound semiconductor material, indium phosphide (InP) has strong irradiation resistance and high photoelectric conversion efficiency. It has advantages in the field of photonics and radio frequency. In atmospheric space, high-energy cosmic rays enter into the earth’s atmosphere and interact with nitrogen (N), oxygen (O) and other elements to produce secondary cosmic rays. The irradiation particles in the atmosphere are mainly neutrons because the penetration of charged particles is weak. The InP semiconductor devices are affected by atmospheric neutron irradiation of various energy from all directions, which results in the internal defects in InP crystals, the degradation of device performance and the reduction of device lifetime. In this paper, Monte Carlo simulation software Geant4 is used to simulate the neutron irradiation effect, and the initial state distribution of displacement damage caused by neutrons with different energy is obtained, including the distribution of non-ionized energy loss (NIEL) with depth, the relationship between NIEL and the energy of incident neutrons, and the type, number and energy of primary knock-on atoms (PKA). The results show that 1) the NIEL is uniformly distributed when material thickness is on the order of μm and for the material thickness on the order of cm and more, the NIEL decreases as the depth increases and can be reduced to zero when the target material is thick enough; 2) by analyzing the NIEL produced by 1–20 MeV neutrons incident on 3-μm InP and their distribution with depth, it is found that the NIEL first increases and then decreases with incident neutron energy increasing. This trend is caused mainly by PKA produced through the inelastic scattering reaction; 3) by analyzing the type and the energy of PKA produced by 1–20 MeV neutrons incident on 3 μm InP, it is found that the PKA of In/P accounts for a large proportion, which causes displacement damage mainly, and the higher the neutron energy, the richer the variety of PKA is and the greater the maximum kinetic energy of PKA, but the PKAs mainly distribute in the low energy part. The present research has theoretical and guiding value for the long-term application of InP-based 5G devices in atmospheric neutron irradiation environment.

Discrete-time inverse linear quadratic optimal control over finite time-horizon under noisy output measurements

In this paper, the problem of inverse quadratic optimal control over finite time-horizon for discrete-time linear systems is considered. Our goal is to recover the corresponding quadratic objective function using noisy observations. First, the identifiability of the model structure for the inverse optimal control problem is analyzed under relative degree assumption and we show the model structure is strictly globally identifiable. Next, we study the inverse optimal control problem whose initial state distribution and the observation noise distribution are unknown, yet the exact observations on the initial states are available. We formulate the problem as a risk minimization problem and approximate the problem using empirical average. It is further shown that the solution to the approximated problem is statistically consistent under the assumption of relative degrees. We then study the case where the exact observations on the initial states are not available, yet the observation noises are known to be white Gaussian distributed and the distribution of the initial state is also Gaussian (with unknown mean and covariance). EM-algorithm is used to estimate the parameters in the objective function. The effectiveness of our results are demonstrated by numerical examples.

Sufficiency of Markov Policies for Continuous-Time Jump Markov Decision Processes

One of the basic facts known for discrete-time Markov decision processes is that, if the probability distribution of an initial state is fixed, then for every policy it is easy to construct a (randomized) Markov policy with the same marginal distributions of state-action pairs as for the original policy. This equality of marginal distributions implies that the values of major objective criteria, including expected discounted total costs and average rewards per unit time, are equal for these two policies. This paper investigates the validity of the similar fact for continuous-time jump Markov decision processes (CTJMDPs). It is shown in this paper that the equality of marginal distributions takes place for a CTJMDP if the corresponding Markov policy defines a nonexplosive jump Markov process. If this Markov process is explosive, then at each time instance, the marginal probability, that a state-action pair belongs to a measurable set of state-action pairs, is not greater for the described Markov policy than the same probability for the original policy. These results are applied in this paper to CTJMDPs with expected discounted total costs and average costs per unit time. It is shown for these criteria that, if the initial state distribution is fixed, then for every policy, there exists a Markov policy with the same or better value of the objective function.

Brief Paper: Augmentation of Hidden Markov Chain for Complex Sequential Data in Context

The classical HMM is defined by a parameter triple <italic>λ</italic> = (<italic>π</italic>, <italic>A</italic>, <italic>B</italic>), where each parameter represents a collection of probability distributions: initial state, state transition and output distributions in order. This paper proposes a new stationary parameter <italic>e</italic> =(<italic>e</italic>₁,<italic>e</italic>₂,…,<italic>e_N</italic>) where <italic>N</italic> is the number of states and <italic>e_t</italic> = <italic>P</italic>(|<italic>x_t</italic> = <italic>i</italic>,<italic>y</italic>) for describing how an input pattern <italic>y</italic> ends in state <italic>x_t</italic> = <italic>i</italic> at time <italic>t</italic> followed by nothing. It is often said that all is well that ends well. We argue here that all should end well. The paper sets the framework for the theory and presents an efficient inference and training algorithms based on dynamic programming and expectation-maximization. The proposed model is applicable to analyzing any sequential data with two or more finite segmental patterns are concatenated, each forming a context to its neighbors. Experiments on online Hangul handwriting characters have proven the effect of the proposed augmentation in terms of highly intuitive segmentation as well as recognition performance and 13.2% error rate reduction.

Effect of surface temperature on quantum dynamics of H2 on Cu(111) using a chemically accurate potential energy surface.

The effect of surface atom vibrations on H2 scattering from a Cu(111) surface at different temperatures is being investigated for hydrogen molecules in their rovibrational ground state (v = 0, j = 0). We assume weakly correlated interactions between molecular degrees of freedom and surface modes through a Hartree product type wavefunction. While constructing the six-dimensional effective Hamiltonian, we employ (a) a chemically accurate potential energy surface according to the static corrugation model [M. Wijzenbroek and M. F. Somers, J. Chem. Phys. 137, 054703 (2012)]; (b) normal mode frequencies and displacement vectors calculated with different surface atom interaction potentials within a cluster approximation; and (c) initial state distributions for the vibrational modes according to Bose-Einstein probability factors. We carry out 6D quantum dynamics with the so-constructed effective Hamiltonian and analyze sticking and state-to-state scattering probabilities. The surface atom vibrations affect the chemisorption dynamics. The results show physically meaningful trends for both reaction and scattering probabilities compared to experimental and other theoretical results.