Discrete State Space Research Articles

Near-optimal energy management for plug-in hybrid fuel cell and battery propulsion using deep reinforcement learning

Plug-in hybrid fuel cell and battery propulsion systems appear promising for decarbonising transportation applications such as road vehicles and coastal ships. However, it is challenging to develop optimal or near-optimal energy management for these systems without exact knowledge of future load profiles. Although efforts have been made to develop strategies in a stochastic environment with discrete state space using Q-learning and Double Q-learning, such tabular reinforcement learning agents’ effectiveness is limited due to the state space resolution. This article aims to develop an improved energy management system using deep reinforcement learning to achieve enhanced cost-saving by extending discrete state parameters to be continuous. The improved energy management system is based upon the Double Deep Q-Network. Real-world collected stochastic load profiles are applied to train the Double Deep Q-Network for a coastal ferry. The results suggest that the Double Deep Q-Network acquired energy management strategy has achieved a further 5.5% cost reduction with a 93.8% decrease in training time, compared to that produced by the Double Q-learning agent in discrete state space without function approximations. In addition, this article also proposes an adaptive deep reinforcement learning energy management scheme for practical hybrid-electric propulsion systems operating in changing environments.

Learning From Demonstrations Using Signal Temporal Logic in Stochastic and Continuous Domains

Learning control policies that are safe, robust and interpretable are prominent challenges in developing robotic systems. Learning-from-demonstrations with formal logic is an arising paradigm in reinforcement learning to estimate rewards and extract robot control policies that seek to overcome these challenges. In this approach, we assume that mission-level specifications for the robotic system are expressed in a suitable temporal logic such as Signal Temporal Logic (STL). The main idea is to automatically infer rewards from user demonstrations (that could be suboptimal or incomplete) by evaluating and ranking them w.r.t. the given STL specifications. In contrast to existing work that focuses on deterministic environments and discrete state spaces, in this letter, we propose significant extensions that tackle stochastic environments and continuous state spaces.

Realizing Active Inference in Variational Message Passing: The Outcome-Blind Certainty Seeker.

Active inference is a state-of-the-art framework in neuroscience that offers a unified theory of brain function. It is also proposed as a framework for planning in AI. Unfortunately, the complex mathematics required to create new models can impede application of active inference in neuroscience and AI research. This letter addresses this problem by providing a complete mathematical treatment of the active inference framework in discrete time and state spaces and the derivation of the update equations for any new model. We leverage the theoretical connection between active inference and variational message passing as described by John Winn and Christopher M. Bishop in 2005. Since variational message passing is a well-defined methodology for deriving Bayesian belief update equations, this letter opens the door to advanced generative models for active inference. We show that using a fully factorized variational distribution simplifies the expected free energy, which furnishes priors over policies so that agents seek unambiguous states. Finally, we consider future extensions that support deep tree searches for sequential policy optimization based on structure learning and belief propagation.

Enhanced geometrically-nonlinear poro-FG shear-deformable beams under moving load in discrete state-space

ABSTRACT Using Hamilton’s principle, the motion-governing partial differential equations of enhanced geometrically-nonlinear porous functionally-graded beam subjected to a concentrated moving force were derived and compared with the corresponding geometrically-nonlinear beam. The beam density and elastic modulus vary continuously along the web direction from those of metal to those of ceramic. The motion-governing difference equations were achieved via generalized differential quadrature and the Newmark’s methods, assembled via state-spatial variables and their velocities, and solved via Newton-Raphson stabilized iterations. Components of the beam dynamic elastica, shearing force and flexural moment were obtained along the span. Proportionality of nonporous-FG beam deflection, to porosity, to material parameter, and to metal portion vis-à-vis ceramic, and that of poro-FG beamanti-symmetry in deflection, to load speed was revealed. The critical speed of the load that causes the locally-greatest displacements was found. Cross-sectional shearing and normal stresses were obtained at mid-span, across the web. As a particular case, the beam motion-governing difference equations were reduced to equilibrium-governing ones and solved. The lateral displacement of the beam static elastica was compared with that of the corresponding nonlinear and linear beams.

Approaching probabilistic laws

In the general problem of verisimilitude, we try to define the distance of a statement from a target, which is an informative truth about some domain of investigation. For example, the target can be a state description, a structure description, or a constituent of a first-order language (Sect. 1). In the problem of legisimilitude, the target is a deterministic or universal law, which can be expressed by a nomic constituent or a quantitative function involving the operators of physical necessity and possibility (Sect. 2). The special case of legisimilitude, where the target is a probabilistic law (Sect. 3), has been discussed by Roger Rosenkrantz (Synthese, 1980) and Ilkka Niiniluoto (Truthlikeness, 1987, Ch. 11.5). Their basic proposal is to measure the distance between two probabilistic laws by the Kullback–Leibler notion of divergence, which is a semimetric on the space of probability measures. This idea can be applied to probabilistic laws of coexistence and laws of succession, and the examples may involve discrete or continuous state spaces (Sect. 3). In this paper, these earlier studies are elaborated in four directions (Sect. 4). First, even though deterministic laws are limiting cases of probabilistic laws, the target-sensitivity of truthlikeness measures implies that the legisimilitude of probabilistic laws is not easily reducible to the deterministic case. Secondly, the Jensen-Shannon divergence is applied to mixed probabilistic laws which entail some universal laws. Thirdly, a new class of distance measures between probability distributions is proposed, so that their horizontal differences are taken into account in addition to vertical ones (Sect. 5). Fourthly, a solution is given for the epistemic problem of estimating degrees of probabilistic legisimilitude on the basis of empirical evidence (Sect. 6).

Improved estimations of stochastic chemical kinetics by finite-state expansion

Stochastic reaction networks are a fundamental model to describe interactions between species where random fluctuations are relevant. The master equation provides the evolution of the probability distribution across the discrete state space consisting of vectors of population counts for each species. However, since its exact solution is often elusive, several analytical approximations have been proposed. The deterministic rate equation (DRE) gives a macroscopic approximation as a compact system of differential equations that estimate the average populations for each species, but it may be inaccurate in the case of nonlinear interaction dynamics. Here we propose finite-state expansion (FSE), an analytical method mediating between the microscopic and the macroscopic interpretations of a stochastic reaction network by coupling the master equation dynamics of a chosen subset of the discrete state space with the mean population dynamics of the DRE. An algorithm translates a network into an expanded one where each discrete state is represented as a further distinct species. This translation exactly preserves the stochastic dynamics, but the DRE of the expanded network can be interpreted as a correction to the original one. The effectiveness of FSE is demonstrated in models that challenge state-of-the-art techniques due to intrinsic noise, multi-scale populations and multi-stability.

Optimal resource allocation in networked control systems using viterbi algorithm

This paper presents an optimal bandwidth allocation method for a networked control system (NCS) which includes time-driven sensor, event-driven controller and random channels. A hidden markov model (HMM) with a discretized state space is formulated for the random traffic to predict the network states using a suitable data window. Network bandwidth is allocated based on the predicted traffic state subject to bounds on the deterministic traffic that guarantee acceptable NCS performance and do not exceed hardware limitations. Bandwidth allocation uses minimization of unmet bandwidth demand. A stability condition is derived for a variable but bounded sampling period interval. Computer simulation results show the effect of varying the number of discrete states for the HMM and the window width on bandwidth allocation. The results compare favorably with a published approach based on fuzzy logic.

The Research of Markov Chain Application under Two Common Real World Examples

Markov chain is a random process with Markov characteristics, which exists in the discrete index set and state space in probability theory and mathematical statistics. Based on probability theory, the Markov chain model is a quantitative prediction model for stationary random phenomena using autoregressive process methods. This article first introduces the Markov Chain and its related principles, then in order to study the applicability of Markov Chain, two common life situations are used in practical applications, and the conclusion that Markov Chain can accurately predict the probability is drawn; finally evaluated the Markov chain model and advocated for its wide application.

Robust state estimation for uncertain linear discrete systems with d‐step state delay

This paper discusses the state estimation problems of an uncertain linear discrete time-varying state space model with d-step state delay. Based on the principle of minimising the expectation of estimation errors and the method of state augmentation, a robust state estimation algorithm is proposed. Specially, this estimator retains the form of Kalman-like filter and the characteristics of fast recursive calculation. Moreover, the conditions of bounded estimation error covariance and the proof of asymptotic unbiasedness of the filter are given. Finally, numerical examples are used to verify the effectiveness and the wide applicability of this estimator.

Analysis and Suppression Strategy of Transient Frequency Deviation by Soft Guide Vane Control in Hydro Turbine

The hydro-dominant characteristic is notable after the asynchronous operation in Southwest China Power Grid which brings worse transient frequency deviation caused by water hammer effect. To prevent the possible cascading failure due to the worse transient frequency deviation in the secondary frequency regulation, the soft guide vane control law is proposed to suppress the water hammer effect in the transient regulation process based on the detailed turbine-penstock model. The discrete state space expression of the turbine-penstock model is built to simulate the regulation effect in the time domain and the PSO (Particle Swarm Optimization) algorithm is applied to optimize the soft guide vane control adjustment factor considering multi-object. Finally, the emergency AGC (Automatic Generation Control) adjustment in the single machine with load transmission system is conducted to verify the effect of transient frequency deviation suppression by optimized soft guide vane control.

Long Short Term Memory Model Based Position-Stiffness Control of Antagonistically Driven Twisted-Coiled Polymer Actuators Using Model Predictive Control

Twisted-coiled polymer actuators (TCA) have many interesting properties that show potentials for making high performance bionic devices. This letter presents a long short term memory (LSTM) model to predict the behavior of an antagonistic joint driven by hybrid TCA bundles made from Spandex and nylon fibers. By using automatic differentiation, which can be done with PyTorch, a linearized discrete state space model can be formulated and then be utilized in model predictive control (MPC). The experimental results show that by combining LSTM and MPC in modeling and control of the TCA, both high prediction performance and control performance can be achieved. It is verified that using the LSTM model, the joint angle and actuator temperatures can be predicted accurately with an avarage steady state error less than 0.1 deg and 0.2 Cel. deg., respectively. The MPC control results show the controller's ability to reach the desired joint angle with an avarage steady state error of 0.21 degrees in set-point regulation and track a sinusoidal waveform at composite frequencies of 0.1 Hz to 0.15 Hz with the steady-state error less than 0.38 degrees while changing joint stiffness.

Motion Planning and Control of an Omnidirectional Mobile Robot in Dynamic Environments

Motion control in dynamic environments is one of the most important problems in using mobile robots in collaboration with humans and other robots. In this paper, the motion control of a four-Mecanum-wheeled omnidirectional mobile robot (OMR) in dynamic environments is studied. The robot’s differential equations of motion are extracted using Kane’s method and converted to discrete state space form. A nonlinear model predictive control (NMPC) strategy is designed based on the derived mathematical model to stabilize the robot in desired positions and orientations. As a main contribution of this work, the velocity obstacles (VO) approach is reformulated to be introduced in the NMPC system to avoid the robot from collision with moving and fixed obstacles online. Considering the robot’s physical restrictions, the parameters and functions used in the designed control system and collision avoidance strategy are determined through stability and performance analysis and some criteria are established for calculating the best values of these parameters. The effectiveness of the proposed controller and collision avoidance strategy is evaluated through a series of computer simulations. The simulation results show that the proposed strategy is efficient in stabilizing the robot in the desired configuration and in avoiding collision with obstacles, even in narrow spaces and with complicated arrangements of obstacles.

On rapid oscillations driving biological processes at disparate timescales

We consider a generic biological process described by a dynamical system, subject to an input signal with a high-frequency periodic component. The rapid oscillations of the input signal induce inherently multiscale dynamics, motivating order-reduction techniques. It is intuitive that the system behaviour is well approximated by its response to the averaged input signal. However, changes to the high-frequency component that preserve the average signal are beyond the reach of such intuitive reasoning. In this study, we explore system response under the influence of such an input signal by exploiting the timescale separation between high-frequency input variations and system response time. Employing the asymptotic method of multiple scales, we establish that, in some circumstances, the intuitive approach is simply the leading-order asymptotic contribution. We focus on higher-order corrections that capture the response to the details of the high-frequency component beyond its average. This approach achieves a reduction in system complexity while providing valuable insight into the structure of the response to the oscillations. We develop the general theory for nonlinear systems, while highlighting the important case of systems affine in the state and the input signal, presenting examples of both discrete and continuum state spaces. Importantly, this class of systems encompasses biochemical reaction networks described by the chemical master equation and its continuum approximations. Finally, we apply the framework to a nonlinear system describing mRNA translation and protein expression previously studied in the literature. The analysis shines new light on several aspects of the system quantification and both extends and simplifies results previously obtained.

Optimal filtering for state space model with time-integral measurements

Filtering for state space model with time-integral measurement is studied. In this model, the measurement is an integral function, rather than an algebraic function, of the state vector. To be more specific, in the problem studied, the measurement is the between-epoch integral of a linear combination of the state vector. This is a new measurement type which arises in several navigation applications with velocity as part of the state vector and with between-epoch displacement as the measurement of the state space model. A rigorous discrete state space model is constructed from first principles to correctly represent the problem to be solved. Compared to this rigorous model, the model used in conventional method is inferior due to the following: a) the covariance of the measurement noise is wrongly determined; b) the between-epoch correlation in the measurement noise is ignored; and c) the cross correlation between the process noise and the measurement noises is ignored. Optimal filtering algorithm is designed according to the rigorous model. A simulation study is conducted to validate the superior performance of the proposed method to the conventional method.

On the general conditions of existence for linear MMSE filters: Wiener and Kalman

State estimation is a fundamental task in a plethora of applications. Two important classes of linear minimum mean square error (MMSE) estimators are Wiener filters (WFs) and Kalman filters (KFs), the latter being the recursive form of the former for linear discrete state-space (LDSS) dynamic models. The minimum set of uncorrelation conditions regarding states and measurements which allows to formulate a general KF form has been recently introduced in the literature, but for both WFs and KFs a standard key assumption is that the measurement noise covariance is invertible. In this contribution we provide i) an alternative derivation of the WF in the case of a non-invertible measurement covariance matrix, and ii) the proof that under the minimum set of uncorrelation conditions lately released, the KF can always be formulated irrespective of the measurement covariance matrix rank. Notice that these results also apply for Kalman predictors and smoothers, linearly constrained WFs and minimum variance distortionless response (MVDR) filters, therefore being a remarkable result of broad interest.

Rotational Probability Flux from Discrete Differential Forms of Stochastic Reaction Kinetics in High Dimensional Reaction Networks

Biochemical reaction networks often display different behaviors and exhibit different phenotypes. In non-equilibrium networks, switching between different phenotypes can occur spontaneously under certain conditions. While there have been significant efforts towards understanding the mechanism of dynamic phenotype switching, a full understanding in many cases remains lacking. Calculation of the rotational probability flux can help elucidating the mechanism of dynamic switching. However, when the copy numbers of molecular species are small, the assumption of a system with continuous state is invalid, and the calculation of rotational probability flux becomes difficult. To the best of our knowledge, rotational probability flux has not been formulated in discrete state space. Here, we first develop a theoretical framework of discrete differential forms for stochastic reaction kinetics. We then introduce the concept of discrete rotational probability flux in high dimensional reaction networks and show how it can be computed. Results of discrete rotational probability flux in the toggle switch network are presented and compared with the amount of entropy production to facilitate understanding of dynamic phenotype switching.

A Deep Reinforcement Learning Approach to Dynamic Loading Strategy of Repairable Multistate Systems

As multistate system (MSS) reliability models can characterize the multistate deteriorating nature of engineering systems, they have received considerable attention in the past decade. The states of a multistate system/component can be distinguished by its performance capacity, which deteriorates over time and can be restored by maintenance activities. On the other hand, the deterioration of a system/component is, oftentimes, controllable by setting a loading strategy. In this article, a dynamic load optimization problem for repairable MSSs is investigated to achieve the maximum expected cumulative performance within a finite time horizon and limited maintenance resources. The degraded components in a system are dynamically maintained to recover to their better conditions, whereas the performance rate of each component can also be dynamically specified. The resulting sequential decision problem is formulated as a Markov decision process with a continuous action space and a mixed integer discrete continuous state space. The deep deterministic policy gradient algorithm, which is a specific deep reinforcement learning algorithm in the actor−critic framework, is customized to overcome the “curse of dimensionality” and mitigate the uncountable state and action spaces. The effectiveness of the proposed method is examined by two illustrative examples, and a set of comparative studies are conducted to demonstrate the advantage of the proposed dynamic loading strategy.

A weak semantic approach to bisimulation metrics in models with nondeterminism and continuous state spaces

Bisimulation metrics are a successful instrument used to estimate the behavioural distance between probabilistic concurrent systems. They have been defined in both discrete and continuous state space models. However, the weak semantics approach, where non-observable actions are abstracted away, has been adopted only in the discrete case. In this paper we fill this gap and provide a weak bisimulation metric for models with continuous state spaces. A technical difficulty is to provide a suitable notion of weak transition, which requires to lift transitions leaving from states to transitions leaving from a continuous distribution over states. We prove that our weak bisimulation metric is non-expansive, thus allowing for compositional reasoning. We prove that systems at distance zero are equated by a suitable notion of probabilistic weak bisimulation. We apply our theory in a case study where continuous distributions derive from the evolution of the physical environment.

Noise covariance matrix estimation with subspace model identification for Kalman filtering

SummaryA problem frequently encountered in Kalman filtering is the tuning of the noise covariance matrices. Indeed, misspecifying their values can drastically reduce the performance of the Kalman filter. Unfortunately, in most practical cases, noise statistics are not known a priori. This paper focuses on a method relying on subspace model identification theory to determine them accurately. This solution is developed for linear time invariant systems with stationary random disturbances having constant covariance matrices. Practically, these noise covariance matrices are determined from the comparison between an estimated state space representation and the discrete time state space representation involved in the Kalman filter equations. The method developed in this paper departs from most of the solutions available in the literature by the fact that it does not need any tuning parameter to be chosen by the user. After discussing theoretical results, several numerical examples are given to demonstrate the efficiency of the approach.

An optimal property of the hyperplane system in a finite cubing

Motivated by navigation and control problems in robotics, Ghrist and Peterson introduced a class of non-positively curved (NPC) cubical complexes arising as configuration spaces of reconfigurable systems, best regarded as discretized state space representations of embodied agents such as a multi-jointed robotic arm. In current real world applications, agents are increasingly required to respond autonomously to sensory input in order for them to contend with a priori unknown obstacles to navigation. In particular, the configuration spaces in question may not be known in advance. This motivates the following problem formulation: Given a NPC cubical complex $$\mathcal {C}$$ and a point-separating collection $$\varSigma $$ of Boolean queries on its 0-skeleton, $$\mathcal {C}^{(0)}$$ , find an efficient algorithm for learning $$\mathcal {C}$$ from the outputs provided by $$\varSigma $$ along an appropriately chosen path in $$\mathcal {C}$$ . In this note, we tackle the problem of identifying $$\mathcal {C}$$ when it is known that $$\mathcal {C}$$ is CAT(0). We show that the collection of canonical hyperplanes of $$\mathcal {C}$$ is the unique solution of a sub-modular minmax problem over the space of point-separating systems of Boolean queries on $$\mathcal {C}^{(0)}$$ , which may also be formulated in terms of the quadratic form associated with the graph Laplacian of $$\mathcal {C}^{(1)}$$ .