Unknown Transition Probabilities Research Articles

Spatio‐temporal dynamic navigation for electric vehicle charging using deep reinforcement learning

AbstractThis paper considers the real‐time spatio‐temporal electric vehicle charging navigation problem in a dynamic environment by utilizing a shortest path‐based reinforcement learning approach. In a data sharing system including transportation network, an electric vehicle (EV) and EV charging stations (EVCSs), it is aimed to determine the most convenient EVCS and the optimal path for reducing the travel, charging and waiting costs. To estimate the waiting times at EVCSs, Gaussian process regression algorithm is integrated using a real‐time dataset comprising of state‐of‐charge and arrival‐departure times of EVs. The optimization problem is modelled as a Markov decision process with unknown transition probability to overcome the uncertainties arising from time‐varying variables. A recently proposed on‐policy actor–critic method, phasic policy gradient (PPG) which extends the proximal policy optimization algorithm with an auxiliary optimization phase to improve training by distilling features from the critic to the actor network, is used to make EVCS decisions on the network where EV travels through the optimal path from origin node to EVCS by considering dynamic traffic conditions, unit value of EV owner and time‐of‐use charging price. Three case studies are carried out for 24 nodes Sioux‐Falls benchmark network. It is shown that phasic policy gradient achieves an average of 9% better reward compared to proximal policy optimization and the total time decreases by 7–10% when EV owner cost is considered.

Switching Unscented Kalman Filters With Unknown Transition Probabilities for Remaining Useful Life Prediction of Bearings

Switching Unscented Kalman Filters With Unknown Transition Probabilities for Remaining Useful Life Prediction of Bearings

Event-Based Asynchronous H∞ Control for Nonhomogeneous Markov Jump Systems With Imperfect Transition Probabilities.

The event-based H∞ control problem is investigated for a class of nonhomogeneous Markov jump systems (MJSs) with partially unknown transition probabilities (TPs). The MJS is characterized by a piecewise nonhomogeneous Markovian chain, where the switching of the system TP matrix is governed by a higher-level chain. A hidden Markov model (HMM) is employed to observe the system mode, which cannot always be correctly detected in practice. Under this framework, the partially unknown TPs existing in both higher-level TPs (HTPs) and conditional TPs (CTPs) are taken into account for practical consideration. Additionally, an observed-mode-dependent event-triggered mechanism (ETM) is employed to design an asynchronous controller, which is expected to alleviate the burden of the communication network. Evidently, the considered scenario is fairly general and covers some special cases. With the above consideration, sufficient conditions are established to guarantee stochastic stability of the resulting closed-loop system with a prescribed H∞ performance. Finally, two examples are presented to demonstrate the effectiveness and applicability of the proposed method.

Event-triggered sliding mode control for singular discrete-time fuzzy Markov jump networked systems

In this present paper, the event-triggered sliding mode control (SMC) issue for a type of singular networked Markov jump systems with partly unknown transition probabilities (TPs) is mainly investigated, and it is modeled by discrete-time Takagi–Sugeno fuzzy systems. To save network resources, a mode-dependent event-triggered scheme (ETS) is applied to improve transmission efficiency. Subsequently, a common sliding surface is designed, which can effectively reduce the impact of the jumping process. Therein, the stochastic admissibility criterion for the system with [Formula: see text] performance [Formula: see text] is derived in the sense of partly unknown TPs. Moreover, a novel SMC law that guarantees the reachability of the quasi-sliding mode is given in view of the reaching condition. Finally, the validity of the presented theorems is demonstrated by a numerical example.

Finite-Time H∞ Controllers Design for Stochastic Time-Delay Markovian Jump Systems with Partly Unknown Transition Probabilities.

This paper concentrates on the finite-time H∞ control problem for a type of stochastic discrete-time Markovian jump systems, characterized by time-delay and partly unknown transition probabilities. Initially, a stochastic finite-time (SFT) H∞ state feedback controller and an SFT H∞ observer-based state feedback controller are constructed to realize the closed-loop control of systems. Then, based on the Lyapunov-Krasovskii functional (LKF) method, some sufficient conditions are established to guarantee that closed-loop systems (CLSs) satisfy SFT boundedness and SFT H∞ boundedness. Furthermore, the controller gains are obtained with the use of the linear matrix inequality (LMI) approach. In the end, numerical examples reveal the reasonableness and effectiveness of the proposed designing schemes.

H∞$$ {H}_{\infty } $$ optimal output tracking control for Markov jump systems: A reinforcement learning‐based approach

AbstractIn this paper, the optimal output tracking control problem for Markov jump systems is investigated, where the two cases with known or completely unknown transition probabilities are both considered. Based on game theory and performance, quadratic cost is considered, where a discount parameter is introduced into the quadratic cost in order to track unstable systems and eliminate the assumption that the noise energy is bounded. The game coupled algebraic Riccati equation and the corresponding controller are presented by dynamic programming. The stochastic stability of the tracking error system is further investigated. Moreover, iterative and reinforcement learning‐based algorithms are proposed for solving the optimal tracking controller with known or completely unknown transition probabilities, respectively. Finally, some numerical simulations on a DC motor are performed to validate the effectiveness of the proposed results.

Learning-based scheduling of integrated charging-storage-discharging station for minimizing electric vehicle users' cost

Charging cost is an important concern for electric vehicle (EV) users. The ordered charging behavior, such as the reasonable selection of charging period and charging power, can greatly decrease users' charging cost. Towards the integrated charging-storage-discharging station (ICSDS), a learning-based method is proposed in this paper to minimize EV users' cost. The physical constraints of ICSDS and the user's demand are first built, and the charging scheduling problem of ICSDS is formulated as a Markov Decision Process (MDP) with unknown transition probability. Second, in order to generate optimal schedule through the learning network, the deep features of the future electricity price are extracted by using a long short-term memory (LSTM) network. Third, a twofold deep deterministic policy gradient (TDDPG) algorithm is proposed to generate the continuous charging actions and avoid the Q value overestimation. In addition, the TDDPG-based scheduling strategy is designed on the basis of the extracted features of electricity price. Finally, the validations through the real-world data demonstrate that the features of electricity price are extracted with a satisfied accuracy. Moreover, compared with many benchmarked methods, the experimental results demonstrate that the charging scheduling by TDDPG possesses excellent performance in minimizing EV users' cost.

Selling Quality-Differentiated Products in a Markovian Market with Unknown Transition Probabilities

How to Price a Product Line When Customer Preferences Change over Time Quality-differentiated products can help sellers increase their profits through market segmentation. However, in many business applications, such as online search and consumer lending, customer preferences evolve over time, making it difficult for sellers to use market segmentation. In their study “Selling Quality-Differentiated Products in a Markovian Market with Unknown Transition Probabilities,” Keskin and Li analyze dynamic pricing of a vertically differentiated product line when customer preferences for quality can shift over time. Keskin and Li show that data-driven learning is essential when operating in a changing market with unknown customer heterogeneity. Keskin and Li also develop a bounded learning policy that implements near-optimal data-driven learning in a Markov-modulated demand environment.

Stability Analysis of Delayed Discrete Singular Piecewise Homogeneous Markovian Jump Systems With Unknown Transition Probabilities via Sliding-Mode Approach

Stability Analysis of Delayed Discrete Singular Piecewise Homogeneous Markovian Jump Systems With Unknown Transition Probabilities via Sliding-Mode Approach

Base on -Learning Pareto Optimality for Linear Itô Stochastic Systems With Markovian Jumps

This article investigate the cooperative differential game (CDG) for continuous-time linear Itô stochastic systems with markovian jumps (SSMJ) to obtain the Pareto solutions. Different from most existing works studying nonzero-sum games, this article studies the CDG on the quadratic infinite horizon for the Itô-type SSMJ with unknown system matrix and transition probability. A novel <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -learning online algorithm is developed, which consists of that (i) the optimal control problem is equivalent to solving a stochastic algebraic Riccatic equation (ARE); (ii) the joint cost function is approximated by a critic neural network (NN) and Pareto efficient is approximated by two actor NNs. The rigorous stability analysis shows that the system state for SSMJ and the NN weight errors are uniformly ultimately bounded (UUB). Finally, the theory analysis is validated by a numerical example with detailed discussions. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —In practical applications, many systems are often affected by the change of external environment or the failure of internal components, which leads to the random jump of system parameters. Markovian jump system can effectively describe the above problems. And when the system model is disturbed by the internal parameters, the system control input and external environment, the random errors of state measurement, and other random factors, the deterministic model can no longer accurately describe the controlled system. Therefore, the SSMJ can describe practical problems more accurately. By cooperation, in general, the cost one specific player incurs is not uniquely determined anymore. If all players decide, for example, to use their control variables to reduce the cost of player 1 as much as possible, a different minimum is attained for player 1 compared with that in the case where all players agree collectively to help a different player in minimizing his cost. So, depending on how the players choose to ‘divide’ their control efforts, a player incurs different ‘minima’. Therefore, we will design an online learning algorithm to obtain pareto solutions with different weights. On the other hand, in practice, it is difficult to obtain an accurate system model. In order to solve this problem, a novel scheme is designed by using <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$</tex-math> </inline-formula> -learning technology, which does not need system matrix.

Optimal Priority Rule-Enhanced Deep Reinforcement Learning for Charging Scheduling in an Electric Vehicle Battery Swapping Station

For a battery swapping station (BSS) with solar generation, N charging bays, and an inventory of M batteries, we study the charging scheduling problem under random EV arrivals, renewable generation, and electricity prices. To minimize the expected weighted sum of charging cost (sum of electricity and battery degradation costs) and EV owners’ waiting cost, we formulate the problem as a Markov decision process with unknown state transition probability. Under a mild heavy-traffic assumption, we rigorously establish the optimality of the Less Demand First (LDF) priority rule under arbitrary system dynamics: batteries with less demand shall be charged first. The optimality result enables us to integrate the LDF rule into a state-of-the-art deep reinforcement learning (DRL) method, proximal policy optimization (PPO), reducing the dimensionality of its output from O(M+N) to O(1), without loss of optimality in the heavy-traffic scenario. Numerical results (on real-world data) demonstrate that the proposed LDF enhanced PPO approach significantly outperforms classical DRL methods and FCFS (first come, first served) priority rule based DRL counterparts.

Stochastic Dynamic Information Flow Tracking game using supervised learning for detecting advanced persistent threats

Advanced persistent threats (APTs) are organized prolonged cyberattacks by sophisticated attackers with the intent of stealing critical information. Although APT activities are stealthy and evade detection by traditional detection tools, they interact with the system components to make progress in the attack. These interactions lead to information flows that are recorded in the form of a system log. Dynamic Information Flow Tracking (DIFT) has been shown to be an effective way to detect APTs using information flows. A DIFT-based detection mechanism dynamically performs security analysis on the information flows to detect possible attacks. However, wide range security analysis using DIFT results in a significant increase in performance overhead and high rates of false-positives and false-negatives. In this paper, we model the strategic interaction between APT and DIFT as a non-cooperative stochastic game. The game unfolds on a state space constructed from an information flow graph (IFG) that is extracted from the system log. The objective of the APT in the game is to choose transitions in the IFG to find an optimal path in the IFG from an entry point of the attack to an attack target. On the other hand, the objective of DIFT is to dynamically select nodes in the IFG to perform security analysis for detecting APT. Our game model has imperfect information as the players are unaware of the actions of the opponent. We consider two scenarios of the game (i) the false-positive and false-negative rates of DIFT (i.e., transition probabilities of the game) are known and (ii) the false-positive and false-negative rates are unknown. For case (i), we propose a value iteration-based algorithm and prove that the solution converges to the optimal solution (Nash equilibrium). Case (ii) translates to an incomplete information game with unknown transition probabilities. For case (ii), we propose a supervised learning-based algorithm, referred to as Hierarchical Supervised Learning (HSL) algorithm. HSL integrates a neural network, to predict the value vector of the game, with a policy iteration algorithm to compute an approximate equilibrium. We implemented our algorithms for cases (i) and (ii) on real attack datasets for nation state and ransomware attacks and validated the performance of our approach. We compared the performance of the HSL algorithm when the transition probabilities are unknown with instances with known transition probabilities and demonstrated that HSL algorithm converges to a solution close to optimal (i.e., optimal value vector) while the value vector obtained using greedy does not converge to optimal for 44.4% of the states and the mean absolute error is almost 200 times that of the HSL.

Dissipative fault detection for time‐delay nonlinear Markov jump systems with measurement outliers under stochastic communication protocol

SummaryThis article investigates the dissipative fault detection (FD) problem for time‐delay nonlinear Markov jump systems with measurement outliers in the case of partially unknown transition probabilities. The stochastic communication protocol is utilized to save network bandwidth, where the scheduling model is described via a Markov chain. An outlier‐resistant FD filter is constructed with the help of adaptive saturation function technology. The sufficient conditions are derived to ensure that the FD system satisfies the stochastic stability and stochastic strict dissipativity. In addition, an FD filter without saturation constraint is also designed to compare with the outlier‐resistant FD filter, which verifies that the outlier‐resistant FD filter weakens the influence of measurement outliers effectively. Finally, two examples are provided to demonstrate the feasibility and effectiveness of the designed FD scheme.

Mode-Dependent Adaptive Event-Triggered Control for Stabilization of Markovian Memristor-Based Reaction-Diffusion Neural Networks.

This article focuses on the design of a mode- dependent adaptive event-triggered control (AETC) scheme for the stabilization of Markovian memristor-based reaction-diffusion neural networks (RDNNs). Different from the existing works with completely known transition probabilities, partly unknown transition probabilities (PUTPs) are considered here. The switching conditions and values of memristive connection weights are all correlated with Markovian jumping. A mode-dependent AETC scheme is newly proposed, in which different adaptive event-triggered mechanisms will be applied for different Markovian jumping modes and memristor switching modes. For each given mode, the corresponding event-triggered mechanism can efficiently reduce the number of transmission signals by adaptively adjusting the threshold. Thus, the mode-dependent AETC scheme can effectively save the limited network communication resources for the considered system. Based on the proposed control scheme, a new stabilization criterion is set up for Markovian memristor-based RDNNs with PUTPs. Meanwhile, a memristor-dependent AETC scheme is devised for memristor-based RDNNs. Finally, simulation results are presented to verify the effectiveness and superiority of the analysis results.

Event-triggered finite-time [formula omitted] control for positive Markov jump systems with partly known transition probability

The article studies the event-triggered finite-time L1 control design for positive Markov jump systems(PMJSs) with partly known transition probability. Firstly, with regard to the unique positive characteristic, a novel 1-norm based event-triggering condition is proposed for PMJSs. Then, sufficient conditions of positivity and finite-time L1 boundedness are provided for the closed-loop system under the constructed even-triggered control law. Based on this, by applying the matrix decomposition technology to the event-triggered controller gain matrix, an effective mode-dependent event-triggered finite-time L1 control scheme in the solvable linear programming(LP) form is presented for PMJSs while preserving the positivity and finite-time L1 boundedness in closed-loop. Further, an effective LP-based event-triggered finite-time L1 control scheme is also obtained for PMJSs with completely known transition probability and completely unknown transition probability, respectively. Finally, two examples are presented to verify the validity of the obtained results.

Expectation‐maximization algorithm for bilinear state‐space models with time‐varying delays under non‐Gaussian noise

SummaryIn this paper, the parameter identification of bilinear state‐space model (SSM) in the presence of random outliers and time‐varying delays is investigated. Under the basis of the observable canonical form of the bilinear model, the system output can be written as a regressive form, and a bilinear state observer is applied to estimate the unknown states. To eliminate the influence of outliers and time‐varying delays on parameter estimation, we employ the Student's distribution to deal with the measurement noise and use a first‐order Markov chain to model the delays. In the framework of expectation‐maximization (EM) algorithm, the unknown parameters, delays, noise variance, states and transition probability matrix can be estimated iteratively. A numerical simulation and a continuous stirred tank reactor (CSTR) process demonstrate that the proposed algorithm has good immunity against outliers and time‐varying delays and offers good estimation accuracy for the bilinear SSM.

Finite-Time Event-Triggered Stabilization for Discrete-Time Fuzzy Markov Jump Singularly Perturbed Systems.

The finite-time event-triggered stabilization is studied for a class of discrete-time nonlinear Markov jump singularly perturbed models with partially unknown transition probabilities (TPs). T-S fuzzy strategy is adopted to characterize the related nonlinear Markov jump singularly perturbed models. The control objective is to make sure that the system states remain within a bounded domain during a fixed-time interval. First, a mode-dependent event-triggered scheme is constructed to reduce the communication burden and save the network bandwidth. On that basis, by using a new Lyapunov function, a developed finite-time stability criterion is derived for the corresponding system to avoid an ill-conditioned issue due to a small singular perturbation parameter. Moreover, the mode-dependent fuzzy controller gain and the event-triggered parameter are co-designed under the framework of partially unknown TPs. Finally, the feasibility of the main results is provided to verify the finite-time event-triggered control strategy.

Quantitative elemental analysis of bismuth brass with target-enhanced orthogonal double-pulse LIBS combined with variant one-point calibration.

Self-absorption and unknown transition probabilities of the analytical lines hinder the accurate quantitative elemental analysis of bismuth brass with conventional calibration-free laser-induced breakdown spectroscopy (LIBS). In this work, target-enhanced orthogonal double-pulse LIBS combined with a variant one-point calibration method was used to solve this problem and realize quantitative elemental analysis of bismuth brass with a relative error of less than 4%. This approach is able to reduce the influence of self-absorption and capable of using analytical lines with unknown transition probabilities while using a calibration-free algorithm, which is helpful for accurate quantitative elemental analysis of bismuth brass and other samples.

Adaptive Underwater Acoustic OFDM Modulation Method Based on Improved Deep Reinforcement Learning

High environmental noise, narrow bandwidth, low carrier frequency, large transmission delay and other factors restrict the data transmission rate in underwater acoustic communication. To this end, improving communication quality and efficiency through adaptive adjustment is necessary for underwater acoustic communication system because of the restrict factor. The proposed method uses Deep Reinforcement Learning (DRL) to select the optimal transportation parameters and maximize the transmission efficiency with. The adaptive communication parameter setting is mapped into a Markov decision process with an unknown state transition probability. The system returns channel information including Channel State Information (CSI) and Signal Noise Ratio (SNR). We validate our estimation and adaptation techniques using the well-known simulator SWellEx_96 in varying underwater environments. Our results show that, for many practical cases, a dynamic adjustment of the prefix length, number of sub-carriers, filter length, and over-sampling factor may enable the utilization of OFDM modulations in underwater communications.

Conservatism reduction in robust control of Markov jump delay systems

A new method is developed for the robust stabilization of a Markov jump system with general discrete state time-delay and partly unknown transition probabilities. In contrast to most existing literature, a set of numerical testable conditions, rather than a huge matrix inequality, are established for the resulting closed-loop system in order to justify the stochastic stability and disturbance attenuation capability with the aid of a non-monotonic design approach. To be specific, by constructing a stochastic Lyapunov function via the application of an n-samples variation, sufficient conditions for the existence of a dissipative state feedback controller are derived with less conservativeness, that is, with larger stochastic stable region and better attenuation level. Three examples including a numerical example, a pest’s structured population model and an F-404 test aircraft model are presented to demonstrate the advantage and practical potential of the proposed methodology.