Discrete State Space Research Articles

Learning from different perspectives for regret reduction in reinforcement learning: A free energy approach

Reinforcement learning (RL) is the core method for interactive learning in living and artificial creatures. Nevertheless, in contrast to humans and animals, artificial RL agents are very slow in learning and suffer from the curse of dimensionality. This is partially due to using RL in isolation; i.e. lack of social learning and social diversity. We introduce a free energy-based social RL for learning novel tasks. Society is formed by the learning agent and some diverse virtual ones. That diversity is in their perception while all agents use the same interaction samples for learning and share the same action set. Individual difference in perception is mostly the cause of perceptual aliasing however, it can result in virtual agents’ faster learning in early trials. Our free energy method provides a knowledge integration method for the main agent to benefit from that diversity to reduce its regret. It rests upon Thompson sampling policy and behavioral policy of main and virtual agents. Therefore, it is applicable to a variety of tasks, discrete or continuous state space, model-free, and model-based tasks as well as to different reinforcement learning methods. Through a set of experiments, we show that this general framework highly improves learning speed and is clearly superior to previous existing methods. We also provide convergence proof.

Variational Bayesian Monte Carlo simulation for rare event assessment

Estimating rare events in structural engineering demands intensive computational resources, and the complexity of the performance function poses challenges to accurate probability estimation. To efficiently conduct the structural reliability analysis on rare events, we propose a novel variational Bayesian Monte Carlo (VBMC)-based subset simulation (SS) method. The proposed approach can be divided into two aspects: the Monte Carlo simulation (MCS) is first applied to estimate the first-layer unconditional failure probability, then VBMC is employed to estimate subsequent conditional failure probabilities. Finally, the original small failure probability is obtained by the product of the first-layer unconditional failure probability and a series of conditional failure probabilities. The proposed approach inherits the merits of SS as well as VBMC, which converts the tricky rare event estimation into a series of high-frequency event estimations and leverages VBMC instead of the canonical Markov chain to produce the independent and informative next-generation failure-conditional samples. Four case studies, including high-dimensional and discrete state space scenarios, are performed to illustrate the feasibility and generality of the proposed method.

A Cooperative Multi-Agent Q-Learning Control Framework for Real-Time Energy Management in Energy Communities

Residential and commercial buildings are responsible for 35% of the EU energy-related greenhouse gas (GHG) emissions. Reducing their emissions is crucial for meeting the challenging EU objective of the agenda for becoming a net-zero continent by 2050. The diffusion and integration of distributed renewable energy sources (RESs) and energy storage systems (ESSs), as well as the creation of energy communities (ECs), have proven to be crucial aspects in reducing GHG emissions. In this context, this article proposes a multi-agent AI-based control framework to solve the EC’s energy management problem in the presence of distributed RESs and ESSs as well as considering a shared ESS. The objectives of the proposed control framework are to satisfy the EC members’ load demand to maximize self-consumption and to manage ESSs charging and discharging processes, to enforce cooperative behavior among the EC members by adopting fair and personalized strategies and to maximize EC members’ profits. The proposed control procedure is based on three sequential stages, each solved by a dedicated local RL agent exploiting the Q-Learning algorithm. To reduce the computational complexity of the proposed approach, specifically defined state aggregation criteria were defined to map the RL agents’ continuous state spaces into discrete state spaces of limited dimensions. During the training phase, the EC members’ profiles and the ESSs’ and RESs’ characteristics were randomly changed to allow the RL agents to learn the correct policy to follow in any given scenario. Simulations proved the effectiveness of the proposed approach for different costumers’ load demand profiles and different EC configurations. Indeed, the trained RL agents proved to be able to satisfy the EC members’ load demands to maximize self-consumption, to correctly use the distributed and shared ESSs, to charge them according to respective personalized criteria and to sell the energy surplus, prioritizing sales to the EC. The proposed control framework also proved to be a useful tool for understanding EC performance in different configurations and, thus, for properly dimensioning the EC elements.

Deep-learning-assisted optical communication with discretized state space of structured light

Abstract The rich structure of transverse spatial modes of structured light has facilitated their extensive applications in quantum information and optical communication. The Laguerre-Gaussian (LG) modes, which carry a well-defined orbital angular momentum (OAM), consist of a complete orthogonal basis describing the transverse spatial modes of light. The application of OAM in free-space optical communication is restricted due to the experimentally limited OAM numbers and the complex OAM recognition methods. Here, we present a novel method that uses the advanced deep learning technique for LG modes recognition. By discretizing the spatial modes of structured light, we turn the OAM state regression into classification. A proof-of-principle experiment is also performed, showing that our method effectively categorizes OAM states with small training samples and the accuracy exceeds 99% from 3-dimensional to 15-dimensional space. By assigning each category a classical information, we further apply our approach to an image transmission task, achieving a transmission accuracy of 99.58%, which demonstrates the ability to encode large data with low OAM number. This work opens up a new avenue for achieving high-capacity optical communication with low OAM number based on structured light.

Development of a Mathematical Model of a Highly Maneuverable Robot for Simulation of Robots with Different Types of Designs

The article presents the results of developing a mathematical model and simulation modeling of a three-wheeled highly maneuverable mobile robot. A mathematical model of the robot in the state space has been developed. The possibility of using this robot with rotary modules for analyzing and testing control algorithms for robots of various designs has been considered. The relevance of the study is justified by the high costs of creating physical models of robots for experimental verification of control algorithms. The proposed three-wheeled mobile robot with rotary modules, where each wheel is simultaneously both a driving and a steering wheel, allows reducing these costs, while maintaining the ability to simulate part of the real operating conditions. Simplified diagrams of the robot are given, the main parameters and equations characterizing the change in linear and angular velocity are described. Particular attention is paid to modeling various types of robot drives: automotive type and with a differential drive. In the discrete state space, equations describing the dynamics of the robot were presented, which allows them to be used in software products for modeling. The results of simulation modeling obtained in the SimInTech software environment confirm the effectiveness of the proposed model. The analysis of the obtained results showed that the use of a three-wheeled robot with rotary modules allows successfully simulating the behavior of robots with other types of drives and creating conditions close to real ones for testing control algorithms. The proposed model can be used to improve the quality of design and testing of control algorithms in robotics, while reducing the costs of creating physical models of robots, as well as in the educational process.

Research on Longitudinal Control of Electric Vehicle Platoons Based on Robust UKF–MPC

In a V2V communication environment, the control of electric vehicle platoons faces issues such as random communication delays, packet loss, and external disturbances, which affect sustainable transportation systems. In order to solve these problems and promote the development of sustainable transportation, a longitudinal control algorithm for the platoon based on robust Unscented Kalman Filter (UKF) and Model Predictive Control (MPC) is designed. First, a longitudinal kinematic model of the vehicle platoon is constructed, and discrete state–space equations are established. The robust UKF algorithm is derived by enhancing the UKF algorithm with Huber-M estimation. This enhanced algorithm is then used to estimate the state information of the leading vehicle. Based on the vehicle state information obtained from the robust UKF estimation, feedback correction and compensation are added to the MPC algorithm to design the robust UKF–MPC longitudinal controller. Finally, the effectiveness of the proposed controller is verified through CarSim/Simulink joint simulation. The simulation results show that in the presence of communication delay and data loss, the robust UKF–MPC controller outperforms the MPC and UKF–MPC controllers in terms of MSE and IAE metrics for vehicle spacing error and acceleration tracking error and exhibits stronger robustness and stability.

A Weak MLMC Scheme for Lévy-Copula-Driven SDEs with Applications to the Pricing of Credit, Equity and Interest Rate Derivatives

This paper develops a novel weak multilevel Monte-Carlo (MLMC) approximation scheme for Lévy-driven Stochastic Differential Equations (SDEs). The scheme is based on the state space discretization (via a continuous-time Markov chain approximation [Mijatović, Aleksandar, Matija Vidmar, and Saul Jacka. 2012. “Markov Chain Approximations for Transition Densities of Lévy Processes.” Electronic Journal of Probability 19]) of the pure-jump component of the driving Lévy process and is particularly suited if the multidimensional driver is given by a Lévy copula. The multilevel version of the algorithm requires a new coupling of the approximate Lévy drivers in the consecutive levels of the scheme, which is defined via a coupling of the corresponding Poisson point processes. The multilevel scheme is weak in the sense that the bound on the level variances is based on the coupling alone without requiring strong convergence. Moreover, the coupling is natural for the proposed discretization of jumps and is easy to simulate. The approximation scheme and its multilevel analogous are applied to examples taken from mathematical finance, including the pricing of credit, equity and interest rate derivatives.

Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees

Researchers have introduced a new algorithm to estimate structural models of dynamic decisions by human agents, addressing the challenge of high computational complexity. Traditionally, this task involves a nested structure: an inner problem identifying an optimal policy and an outer problem maximizing a measure of fit. Previous methods have struggled with large discrete state spaces or high-dimensional continuous state spaces, often sacrificing reward estimation accuracy. The new approach combines policy improvement with a stochastic gradient step for likelihood maximization, ensuring accurate reward estimation without compromising computational efficiency. This single-loop algorithm, designed to handle high-dimensional state spaces, converges to a stationary solution with finite-time guarantees. When the reward is linearly parameterized, it approximates the maximum likelihood estimator sublinearly, offering a robust solution for complex decision modeling tasks.

Duality and the well-posedness of a martingale problem

For two Polish state spaces EX and EY, and an operator GX, we obtain existence and uniqueness of a GX-martingale problem provided there is a bounded continuous duality function H on EX×EY together with a dual process Y on EY which is the unique solution of a GY-martingale problem. For the corresponding solutions (Xt)t≥0 and (Yt)t≥0, duality with respect to a function H in its simplest form means that the relation Ex[H(Xt,y)]=Ey[H(x,Yt)] holds for all (x,y)∈EX×EY and t≥0. While duality is well-known to imply uniqueness of the GX-martingale problem, we give here a set of conditions under which duality also implies existence without using approximating sequences of processes of a different kind (e.g. jump processes to approximate diffusions) which is a widespread strategy for proving existence of solutions of martingale problems. Given the process (Yt)t≥0 and a duality function H, to prove existence of (Xt)t≥0 one has to show that the r.h.s. of the duality relation defines for each y a measure on EX, i.e. there are transition kernels (μt)t≥0 from EX to EX such that Ey[H(x,Yt)]=∫μt(x,dx′)H(x′,y) for all (x,y)∈EX×EY and all t≥0.As examples, we treat resampling and branching models, such as the Fleming–Viot measure-valued diffusion and its spatial counterparts (with both, discrete and continuum space), as well as branching systems, such as Feller’s branching diffusion. While our main result as well as all examples come with (locally) compact state spaces, we discuss the strategy to lift our results to genealogy-valued processes or historical processes, leading to non-compact (discrete and continuum) state spaces. Such applications will be tackled in forthcoming work based on the present article.

Coordinated configuration of hybrid energy storage for electricity-hydrogen integrated energy system

This paper proposes an optimal coordinated configuration method of hybrid electricity and hydrogen storage for the electricity‑hydrogen integrated energy system (EH-ES) to promote the renewable energy source (RES) utilization and reduce the deployment cost. To simulate the practical operation of EH-ES, an energy hub framework with a discrete state space system matrix is designed to characterize the internal interconnection topology and quantify the steady-state production, storage, conversion and utilization processes of electricity and hydrogen energy carriers within EH-ES. A chronological operation simulation based electricity and hydrogen storage configuration model over a year-round time horizon is formulated to collaboratively optimize the capacity of the electrolyzer, fuel cell, battery energy storage (BES) and hydrogen storage tank. In this model, the power and capacity complementarity of BES and generalized hydrogen storage is fully exploited to compensate for power and energy imbalances, and additional capacity configuration of BES is considered to mitigate the drastic electrolytic power fluctuation. Besides, a typical source-load scenario reduction method based on K-means and ordered clustering is developed to avoid a large number of operation variables due to year-round hourly simulation. Comparative studies demonstrate that the proposed methodology can achieve the RES consumption rate of 98.873 % while the equipment investment cost can be reduced by 10.034 %.

Reinforcement learning based autonomous multi-rotor landing on moving platforms

Multi-rotor UAVs suffer from a restricted range and flight duration due to limited battery capacity. Autonomous landing on a 2D moving platform offers the possibility to replenish batteries and offload data, thus increasing the utility of the vehicle. Classical approaches rely on accurate, complex and difficult-to-derive models of the vehicle and the environment. Reinforcement learning (RL) provides an attractive alternative due to its ability to learn a suitable control policy exclusively from data during a training procedure. However, current methods require several hours to train, have limited success rates and depend on hyperparameters that need to be tuned by trial-and-error. We address all these issues in this work. First, we decompose the landing procedure into a sequence of simpler, but similar learning tasks. This is enabled by applying two instances of the same RL based controller trained for 1D motion for controlling the multi-rotor’s movement in both the longitudinal and the lateral directions. Second, we introduce a powerful state space discretization technique that is based on i) kinematic modeling of the moving platform to derive information about the state space topology and ii) structuring the training as a sequential curriculum using transfer learning. Third, we leverage the kinematics model of the moving platform to also derive interpretable hyperparameters for the training process that ensure sufficient maneuverability of the multi-rotor vehicle. The training is performed using the tabular RL method Double Q-Learning. Through extensive simulations we show that the presented method significantly increases the rate of successful landings, while requiring less training time compared to other deep RL approaches. Furthermore, for two comparison scenarios it achieves comparable performance than a cascaded PI controller. Finally, we deploy and demonstrate our algorithm on real hardware. For all evaluation scenarios we provide statistics on the agent’s performance. Source code is openly available at https://github.com/robot-perception-group/rl_multi_rotor_landing.

Impact load identification method based on frequency response pattern recognition and dynamic sensor filter strategy

Identification of impact loads plays important role in marine structures health monitoring but is difficult to be measured directly most time. This study investigates a two-stage framework for impact load localization and reconstruction, consisting of load region identification and local refined nodal search. For the region identification, a novel frequency response feature preprocessing method based on FFT is proposed and incorporated into a multi-layer perceptron (MLP) neural network as the embedding function of the Matching Network (MN), the core model adopted for pattern recognition. Based on the region probabilities predicted by MN, a local refined nodal search strategy is provided, which is initialized by a region correction method for amending the possible region misclassification and further guided by error metrics with iteration search strategy. Moreover, the inverse problem in this study is formulated in the discretized state space expression with the reduced modal coordinates. For improving the load inverse accuracy affected by Zero Order Hold (ZOH) simplification in this formulation, a dynamic sensor filter strategy is provided. Eventually, a numerical experiment of impact load identification on a steel plate is performed and discussed, whose results indicate the validity and robustness of the proposed method.

A novel approach to model nance drying considering its shrinkage

Air drying is a viable preservation method to boost the accessibility of nance, a seasonal drupe with very short shelf life; however, information about nance drying is scarce, especially from a mass transfer (MT) standpoint. Besides, the lack of simple models considering the stone fraction (radius of inner stone/radius of nance) and the mesocarp shrinkage in drupes has caused most investigations rely on sphere and cube geometries, contributing to the unreliable estimation of MT properties. This study developed a discrete state-space (SS) formulation with analytical solution to describe nance drying which, unlike a variable separation (VS) solution, can consider a variable stone fraction, caused by outer boundary moving while flesh-pit interface remains still. Nance fruits were air dried (60, 70 and 80 °C) to obtain their moisture kinetics and shrinking behavior. Drying data were used to estimate water diffusivity (D) under rigid (VS and SS solutions) and shrinkable (SS and numerical solution) mesocarp assumptions. Nances lost about 60% of their initial volume at the end of drying, while stone fraction increased from 0.44 to 0.62. D values ranged from 5.3 to 6.7 (×10–10) m2/s with SS approach and variable stone fraction. The major source of error in D was not the formulation (SS or VS) when a rigid mesocarp was considered, but the values assigned to the nance radius (length for diffusion) and stone fraction, an initial or average value along process. SS diffusivities did not differ to those estimated with the full numerical solution under variable stone fraction (p>0.05), while VS parameters were overestimated by 44–46 % as a minimum. SS formulation can be applied to hollow cylinders and other geometries with 2D or 3D MT, while able to include non-isotropic shrinkage and evolving external conditions, a problem normally reserved to pure numerical implementations, offering substantial advantages over VS equations.

Stability prediction method of time-varying real-time hybrid testing system on vehicle-bridge coupled system

In recent years, real-time hybrid testing (RTHT) has been applied for the dynamic testing of high-speed trains running on bridges. A guarantee of stability for the RTHT system is essential to achieve a safe and reliable result. However, the inherent time-varying characteristics of the vehicle-bridge coupled system pose challenges to RTHT stability prediction. This study aims to develop a stability prediction method specifically tailored for time-varying RTHT system. Firstly, the vehicle-bridge coupled RTHT was modelled using a discrete state–space representation with a comprehensive consideration of the time-varying vehicle-bridge interaction and the dynamics of the shaking table. Subsequently, a time-varying stability criterion was derived from the periodic time-varying state matrix, forming the basis for a relative stability prediction method. The validity of the proposed method was confirmed through simulations and experiments employing a single-axle interaction within the vehicle-bridge coupled RTHT system. The coupled system consisted of a quarter-car model and simply supported beams were used as an example to evaluate the stability and accuracy of the time-varying RTHT. The results showed that the stability increased with increasing vehicle speed. Reducing the pure time delay of the shaking table improved both stability and accuracy. Increasing the effective frequency of the shaking table improved the accuracy but may reduce the stability of the RTHT.

Explicit machine learning-based model predictive control of nonlinear processes via multi-parametric programming

Machine learning-based model predictive control (ML-MPC) has been developed to control nonlinear processes with unknown first-principles models. While ML models can capture nonlinear dynamics of complex systems, the complexity of ML models leads to increased computation time for real-time implementation of ML-MPC. To address this issue, in this work, we propose an explicit ML-MPC framework for nonlinear processes using multi-parametric programming. Specifically, a self-adaptive approximation algorithm is first developed to obtain a piecewise linear affine function that approximates the behaviors of ML models. Then, multi-parametric quadratic programming (mpQP) problems are formulated to generate the solution map for states in discretized state–space. Furthermore, to accelerate the implementation of explicit ML-MPC, a neighbor-first search algorithm is developed. Finally, an example of a chemical reactor is used to demonstrate the effectiveness of the explicit ML-MPC.

Optimizing the Finishing Treatment of Potato Harvester Transmission Shafts

The potato harvester’s distinctive specifications include the spatially segregated drive of its units from an external engine, operating under stochastic conditions. This setup complicates the loading of its components, including transmission shafts, and creates the prerequisites for decreased durability, leading to an upsurge in fatigue failures. To counteract this process, transmission shafts undergo a special treatment aimed at mitigating concentrated defects. (Research purpose) The purpose of this research is to formulate an objective function for controlling the finishing treatment of transmission shafts in a potato harvester. This function is designed to maximize potato yield while monitoring concentrated defects. (Materials and methods) The implementation of the method requires equipment capable of being controlled through a technical vision channel. The control scheme employs mathematical tools for describing discrete and continuous random variables, Markov processes with discrete time and continuous state space, the maximum likelihood method, and methods of numerical optimization in multifactor space. (Results and discussion) The research has yielded formulas for calculating the probabilities of successful and unsuccessful outcomes of finishing treatment based on the criterion of concentrated defect presence. The paper provides the results of testing the proposed algorithm for predicting the yield of viable products using the proposed method based on interim measurements of defect characteristics during the treatment process. Additionally, a scheme is proposed for integrating the obtained results into the production process of manufacturing transmission shafts for potato harvesters. (Conclusions) An algorithm has been proposed to control the finishing treatment of the transmission shaft based on a predictive assessment of the probability of manufacturing a product that adheres to the specified technical criteria regarding geometry and surface cleanliness. The predictive probability of achieving a surface with the required cleanliness without exceeding permissible defect limits is considered as the objective function. The solution to the problem is achieved through the selection of appropriate technological parameters.

Real-time estimation of barycenter position for fuel cell distributed drive electric tractor based on derived UKF

ABSTRACT The FCDET is considered as a promising and advanced agricultural machinery due to its convenience and controllability. To enhance the control performance of the FCDET unit dynamic system, understanding the accurate position of the center of gravity (CoG) of the tractor unit in longitudinal motion and the operating environment is essential. In this paper, we proposed a real-time estimation method of CoG position of longitudinal motion for FCDET plowing and transportation conditions. CoG height and longitudinal axis distance distribution coefficients are used as state estimation variables, and the data obtained from displacement and velocity sensors are used as state observation variables, establishing a discrete system state space model and proving the observability. The proposed Derived UKF algorithm introduces the Sage-Husa adaptive noise equation within the Square-Root UKF framework to address the non-positive definite matrix problem. The research shows that the precise CoG longitudinal position effectively reflects changes in operational pavement load. The Derived UKF algorithm exhibits high state estimation accuracy in plowing and transportation conditions, with mean absolute errors of 14.2% and 5.0% of the state amplitude, respectively. This study supports the practical application of real-time state parameter estimation algorithms in controlling advanced agricultural machinery distributed drive systems.

Discrete fractional order chaotic systems synchronization based on the variable structure control with a new discrete reaching-law

In this paper, we directly derive a new discrete state space expression of the fractional order chaotic system based on the fractional order Grnwald-Letnikow (G-L) definition and design a variable structure controller with a new faster reaching-law. The new reaching-law has the advantages of weakening the high frequency shake. Firstly, the condition of the discrete sliding mode surface is demonstrated. Then a multi-parametric function for sliding mode surface is constructed for weakening the high frequency shake through improving the Gao discrete reaching-law. Finally, the newly designed variable structure controller is applied to realize the synchronization of two different order discrete fractional chaotic systems. The simulation results show that the designed controller in this paper is effective, as it can achieve the synchronization of the discrete fractional order chaotic systems with external disturbances. Theoretical analysis and simulation results prove the effectiveness and robustness of this control method.

Numerical analysis of the dynamic characteristics of a fractional-order viscoelastic beam with fixed supports at both ends

In this paper, we present a generalized model of a viscoelastic beam, which takes into account the influence of axial forces and incorporates a fractional constitutive relationship. In addition, we propose a novel numerical calculation method for analyzing fractional-order viscoelastic beams. This method takes the transverse displacement and bending moment of the beam as state variables, transforms the beam model into a discrete state space equation through the application of the central difference method, and utilizes an improved precise integration algorithm to solve the equation. To evaluate the performance of the method, the responses of the beam under two types of excitations, namely uniformly distributed transverse load and the motion of the support at both ends, are calculated under fixed hinge conditions. The results demonstrate that the present method has excellent accuracy and convergence, and also reveal some nonlinear phenomena of the system.

Algorithm for Diffuse TSK Modeling of SNL MIMO with Undefined Operation Points

This paper presents an algorithm for constructing fuzzy models in linear state subspaces from the nonlinear MIMO dynamic model of plants whose operating points are not defined within the permissible physical range for the system. It is based on the fuzzy Takagi-Sugeno-Kang model and Kawamoto's ideas of non-linearity sectors. The relevant functions in the antecedent are modeled with linear functions, while functions model the consequent in Discrete State Space. The application of the algorithm to the model of a thermoelectric plant widely studied in the specialized literature is discussed.