True Value Of State Research Articles

Mild Policy Evaluation for Offline Actor-Critic.

In offline actor-critic (AC) algorithms, the distributional shift between the training data and target policy causes optimistic Q value estimates for out-of-distribution (OOD) actions. This leads to learned policies skewed toward OOD actions with falsely high Q values. The existing value-regularized offline AC algorithms address this issue by learning a conservative value function, leading to a performance drop. In this article, we propose a mild policy evaluation (MPE) by constraining the difference between the Q values of actions supported by the target policy and those of actions contained within the offline dataset. The convergence of the proposed MPE, the gap between the learned value function and the true one, and the suboptimality of the offline AC with MPE are analyzed, respectively. A mild offline AC (MOAC) algorithm is developed by integrating MPE into off-policy AC. Compared with existing offline AC algorithms, the value function gap of MOAC is bounded by the existence of sampling errors. Moreover, in the absence of sampling errors, the true state value function can be obtained. Experimental results on the D4RL benchmark dataset demonstrate the effectiveness of MPE and the performance superiority of MOAC compared to the state-of-the-art offline reinforcement learning (RL) algorithms.

State Conditional Filtering

In many dynamical state estimation problems, not all the values that the state can take have the same importance; hence, missing to deliver an appropriate estimate has more severe consequences for certain state values than for others. In many applications, such important state values correspond to events that have low a priori probability to happen (e.g., unsafe situations or conditions that one tries to avoid by design). Provably, Kalman filtering techniques are inadequate to correctly estimate such rare events. In this article, a new state estimation paradigm is introduced to build confidence regions that contain the true state value, whatever this value is, with a user-chosen probability. Among regions having this property, an algorithm is introduced that generates in a Gaussian setup the region that satisfies a minimum-volume condition.

ScaleUp: middleware for intelligent environments.

The development of the Internet of Things (IoT) expands to an ultra-large-scale, which provides numerous services across different domains and environments. The use of middleware eases application development by providing the necessary functional capability. This paper presents a new form of middleware for controlling smart devices installed in an intelligent environment. This new form of middleware functioned seamlessly with any manufacturer API or bespoke controller program. It acts as an all-encompassing top layer of middleware in an intelligent environment control system capable of handling numerous different types of devices simultaneously. This protected de-synchronization of data stored in clone devices. It showed that in this middleware, the clone devices were regularly synchronized with their original master such as locally stored representations were continuously updated with the known true state values.

Closing the loop: Dynamic state estimation and feedback optimization of power grids

This paper considers the problem of online feedback optimization to solve the AC Optimal Power Flow in real-time in power grids. This consists in continuously driving the controllable power injections and loads towards the optimal set-points in time-varying conditions based on real-time measurements performed on the grid. However, instead of assuming noise-free full state measurement like in recently proposed feedback optimization schemes, we connect a dynamic State Estimation using available measurements, and study its dynamic interaction with the optimization scheme. We certify stability of this interconnection and the convergence in expectation of the state estimate and the control inputs towards the true state values and optimal set-points respectively. Additionally, we bound the resulting stochastic error. Finally, we show the effectiveness of the approach on a test case using high resolution consumption data.

Dissipative interval observer design for discrete‐time nonlinear systems

AbstractThis paper studies the problem of designing interval observers for a family of discrete‐time nonlinear systems subject to parametric uncertainties and external disturbances. The design approach states that the interval observers are constituted by a couple of preserving order observers, one providing an upper estimation of the state while the other provides a lower one. The design aim is to apply the cooperative and dissipative properties to the discrete‐time estimation error dynamics in order to guarantee that the upper and lower estimations are always above and below the true state trajectory for all times, while both estimations asymptotically converge towards a neighborhood of the true state values. The approach represents an extension to the original method proposed by the authors, which focuses on the continuous‐time nonlinear systems. In some situations, the design conditions can be formulated as bilinear matrix inequalities (BMIs) and/or linear matrix inequalities (LMIs). Two simulation examples are provided to show the effectiveness of the design approach.

A nonlinear second-order filtering strategy for state estimation of uncertain systems

In this paper, a new strategy referred to as the nonlinear second-order (NSO) filter is presented and used for estimation of linear and nonlinear systems in the presence of uncertainties. Similar to the popular Kalman filter estimation strategy, the proposed strategy is model-based and formulated as a predictor-corrector. The NSO filter is based on variable structure theory that utilizes a switching term and gain that ensures some level of estimation stability. It offers improvements in terms of robustness to modeling uncertainties and errors. The proof of stability is derived based on Lyapunov that demonstrates convergence of estimates towards the true state values. The proposed filtering strategy is based on a second-order Markov process that utilizes information from the current and past two time steps. An experimental system was setup and characterized in order to demonstrate the proposed filtering strategy's performance. The strategy was compared with the popular Kalman filter (and its nonlinear form) and the smooth variable structure filter (SVSF). Experimental results demonstrate that the proposed nonlinear second-order filter provides improvements in terms of state estimation accuracy and robustness to modeling uncertainties and external disturbances.

Adaptive estimation of nonlinear parameters of a nonholonomic spherical robot using a modified fuzzy-based speed gradient algorithm

This paper deals with adaptive estimation of the unknown parameters and states of a pendulum-driven spherical robot (PDSR), which is a nonlinear in parameters (NLP) chaotic system with parametric uncertainties. Firstly, the mathematical model of the robot is deduced by applying the Newton–Euler methodology for a system of rigid bodies. Then, based on the speed gradient (SG) algorithm, the states and unknown parameters of the robot are estimated online for different step length gains and initial conditions. The estimated parameters are updated adaptively according to the error between estimated and true state values. Since the errors of the estimated states and parameters as well as the convergence rates depend significantly on the value of step length gain, this gain should be chosen optimally. Hence, a heuristic fuzzy logic controller is employed to adjust the gain adaptively. Simulation results indicate that the proposed approach is highly encouraging for identification of this NLP chaotic system even if the initial conditions change and the uncertainties increase; therefore, it is reliable to be implemented on a real robot.

Performance Degradation Due to Particle Impoverishment in Particle Filtering

Particle filtering (PF) has shown its outperforming results compared to that of classical Kalman filtering (KF), particularly for highly nonlinear problems. However, PF may not be universally superior to the extended KF (EKF) although the case (i.e. an example that the EKF outperforms PF) is seldom reported in the literature. Particularly, PF approaches show degraded performance for problems where the state noise is very small or zero. This is because particles become identical within a few iterations, which is so called particle impoverishment (PI) phenomenon; consequently, no matter how many particles are employed, we do not have particle diversity regardless of if the impoverished particle is close to the true state value or not. In this paper, we investigate this PI phenomenon, and show an example problem where a classical KF approach outperforms PF approaches in terms of mean squared error (MSE) criterion. Furthermore, we compare the processing speed of the EKF and PF approaches, and show the better speed performance of classical EKF approaches. Therefore, PF approaches may not be always better option than the classical EKF for nonlinear problems. Specifically, we show the outperforming result of unscented Kalman filter compared to that of PF approaches (which are shown in Fig. 7(c) for processing speed performance, and Fig. 6 for MSE performance in the paper).

Set-Membership Fuzzy Filtering for Nonlinear Discrete-Time Systems

This paper is concerned with the set-membership filtering (SMF) problem for discrete-time nonlinear systems. We employ the Takagi-Sugeno (T-S) fuzzy model to approximate the nonlinear systems over the true value of state and to overcome the difficulty with the linearization over a state estimate set rather than a state estimate point in the set-membership framework. Based on the T-S fuzzy model, we develop a new nonlinear SMF estimation method by using the fuzzy modeling approach and the S-procedure technique to determine a state estimation ellipsoid that is a set of states compatible with the measurements, the unknown-but-bounded process and measurement noises, and the modeling approximation errors. A recursive algorithm is derived for computing the ellipsoid that guarantees to contain the true state. A smallest possible estimate set is recursively computed by solving the semidefinite programming problem. An illustrative example shows the effectiveness of the proposed method for a class of discrete-time nonlinear systems via fuzzy switch.

The effect of management structure on the performance of interconnected packet-switched networks

A multi-layer model is used to study the effect of management structure on the performance of connection-oriented packet-switched networks managed via fixed threshold call admission policies. Call admission decisions are based on estimates of the number and characteristics of currently held calls, which may be inaccurate due to uncertainties in the measurement process or to the use of untimely information. Let the state estimate be represented as the true state value offset by a noise factor. The standard deviation of the estimation error serves as a key parameter in representing the complexity, coverage, extensiveness, and cost of the implemented network management and information collection procedure. For a single-domain network the effects of Gaussian noise on blocking, throughput, and the probability of excess calls are examined and used to define a measure of performance, the throughput capacity trajectory, which gives maximum throughput levels for fixed packet blocking, packet delay, and probability of excess call constraints. In a multi-domain network a particular network manager/controller may have complete information about its own domain, but limited, aggregated, or untimely information about other domains. Tradeoffs between centralized and distributed decision-making are discussed and a mechanism is provided for comparing various management structures as well as determining good values for admission control thresholds.