Partial Observations Research Articles

Reinforcement Learning for Blast Furnace Ironmaking Operation With Safety and Partial Observation Considerations.

Making proper decision online in complex environment during the blast furnace (BF) operation is a key factor in achieving long-term success and profitability in the steel manufacturing industry. Regulatory lags, ore source uncertainty, and continuous decision requirement make it a challenging task. Recently, reinforcement learning (RL) has demonstrated state-of-the-art performance in various sequential decision-making problems. However, the strict safety requirements make it impossible to explore optimal decisions through online trial and error. Therefore, this article proposes a novel offline RL approach designed to ensure safety, maximize return, and address issues of partially observed states. Specifically, it utilizes an off-policy actor-critic framework to infer the optimal decision from expert operation trajectories. The "actor" in this framework is jointly trained by the supervision and evaluation signals to make decision with low risk and high return. Furthermore, we investigate a recurrent version of the actor and critic networks to better capture the complete observations, which solves the partially observed Markov decision process (POMDP) arising from sensor limitations. Verification within the BF smelting process demonstrates the improvements of the proposed algorithm in performance, i.e., safety and return.

A Mean-Field Game for a Forward-Backward Stochastic System with Partial Observation and Common Noise

A Mean-Field Game for a Forward-Backward Stochastic System with Partial Observation and Common Noise

CTDS: Centralized Teacher With Decentralized Student for Multiagent Reinforcement Learning

Due to the partial observability and communication constraints in many multi-agent reinforcement learning (MARL) tasks, centralized training with decentralized execution (CTDE) has become one of the most widely used MARL paradigms. In CTDE, centralized information is dedicated to learning the allocation of the team reward with a mixing network, while the learning of individual Q-values is usually based on local observations. The insufficient utility of global observation will degrade performance in challenging environments. To this end, this work proposes a novel Centralized Teacher with Decentralized Student (CTDS) framework, which consists of a teacher model and a student model. Specifically, the teacher model allocates the team reward by learning individual Q-values conditioned on global observation, while the student model utilizes the partial observations to approximate the Q-values estimated by the teacher model. In this way, CTDS balances the full utilization of global observation during training and the feasibility of decentralized execution for online inference. Our CTDS framework is generic which is ready to be applied upon existing CTDE methods to boost their performance. We conduct experiments on a challenging set of StarCraft II micromanagement tasks to test the effectiveness of our method and the results show that CTDS outperforms the existing value-based MARL methods.

Active flow control for bluff body drag reduction using reinforcement learning with partial measurements

Active flow control for drag reduction with reinforcement learning (RL) is performed in the wake of a two-dimensional square bluff body at laminar regimes with vortex shedding. Controllers parametrised by neural networks are trained to drive two blowing and suction jets that manipulate the unsteady flow. The RL with full observability (sensors in the wake) discovers successfully a control policy that reduces the drag by suppressing the vortex shedding in the wake. However, a non-negligible performance degradation ( $\sim$ 50 % less drag reduction) is observed when the controller is trained with partial measurements (sensors on the body). To mitigate this effect, we propose an energy-efficient, dynamic, maximum entropy RL control scheme. First, an energy-efficiency-based reward function is proposed to optimise the energy consumption of the controller while maximising drag reduction. Second, the controller is trained with an augmented state consisting of both current and past measurements and actions, which can be formulated as a nonlinear autoregressive exogenous model, to alleviate the partial observability problem. Third, maximum entropy RL algorithms (soft actor critic and truncated quantile critics) that promote exploration and exploitation in a sample-efficient way are used, and discover near-optimal policies in the challenging case of partial measurements. Stabilisation of the vortex shedding is achieved in the near wake using only surface pressure measurements on the rear of the body, resulting in drag reduction similar to that in the case with wake sensors. The proposed approach opens new avenues for dynamic flow control using partial measurements for realistic configurations.

Quantitative data and indicators on agricultural activity

Statistical coverage and quantitative presentation of agricultural activity are of particular importance for planning agricultural, strategic, and production activities. Food production is a fundamental existential condition, that is, the essence of the existence and development of the human population, as well as animal and plant life on Earth. For all the reasons mentioned above, it is utterly important to have adequate, valid, up-to-date, and comprehensive data and indicators. Obtaining such data is a complex, difficult, time-consuming, and relatively expensive process. If the data are missing, inaccurate, out-of-date, or if they are not collected properly, then all the analyses, estimates, forecasts, and plans are bound to contain errors, resulting in wrong conclusions, decisions, and ultimately leading to wrong solutions. The problem of missing data and the presence of outliers - atypical values is particularly emphasized. In this text the most important aspects of the quantitative (statistical) coverage of agricultural activity are presented. This primarily refers to the method of obtaining the data, which can be done as a complete or a partial observation, that is, sampling. The two methods are presented through censuses, surveying and sampling, with special reference to the upcoming comprehensive census of agriculture. The process of obtaining of indicators is presented in the text, with a special focus on descriptive statistical measures. Data and indicators according to the Global Strategy for the Improvement of Agricultural and Rural Statistics, as well as data and indicators of the Food and Agriculture Organization (FAO), indicators of the Common Agricultural Policy (CAP), and indicators of the World Bank and Eurostat are presented. Based on all the means of obtaining data, while respecting the constant need to align and harmonize the system of our agricultural statistics with the requirements of the European Union, as well as considering the experiences of all the mentioned (and most important) institutions for the production of statistical data in Europe and the world, a proposal of indicators and required data for quantitative presentation of agricultural activity in the Republic of Srpska is given. In addition, the need to introduce statistical registers of agriculture, that should enable continuous quantitative monitoring of agricultural activity, was particularly emphasized.

An Incomplete Observability-Constrained PMU Allocation Problem by Using Mathematical and Evolutionary Algorithms

The purpose of this paper is to introduce several optimization algorithms that can be used to address optimization models in the power network, where the level of observability may be either complete or incomplete. These algorithms include discrete, continuous and metaheuristic methods. Initially, the optimization problem is approached by implementing a zero-one mixed integer linear program solved by several methods, including branch and bound revised simplex and primal dual-simplex in combination with interior point algorithms. To solve the problem of depth-one-unobservability (DoOU), a nonlinear program is proposed using Sequential Quadratic Programming (SQP), Interior-Point methods (IPMs) or YALMIP\\s branch-and-bound algorithm. Additionally, the paper proposes the use of metaheuristic algorithms, such as Genetic Algorithms (GAs) and Binary Particle Swarm Optimization (BPSO), to solve optimization problems under incomplete observability. The proposed algorithms are tested using simulations on IEEE standard systems to illustrate their efficiency and reliability in solving the optimization problem under partial observability. Overall, the paper concludes that these algorithms can efficiently lead to the optimum point in a reasonable runtime. Hence, this work examines the problem of putting a restricted PMUs number to make the DoOU and to give a feedback to the state estimation routine accuracy.

Cooperative Multiagent Reinforcement Learning With Partial Observations

In this paper, we propose a distributed zeroth-order policy optimization method for Multi-Agent Reinforcement Learning (MARL). Existing MARL algorithms often assume that every agent can observe the states and actions of all the other agents in the network. This can be impractical in large-scale problems, where sharing the state and action information with multi-hop neighbors may incur significant communication overhead. The advantage of the proposed zeroth-order policy optimization method is that it allows the agents to compute the local policy gradients needed to update their local policy functions using local estimates of the global accumulated rewards that depend on partial state and action information only and can be obtained using consensus. Specifically, to calculate the local policy gradients, we develop a new distributed zeroth-order policy gradient estimator that relies on one-point residual-feedback which, compared to existing zeroth-order estimators that also rely on one-point feedback, significantly reduces the variance of the policy gradient estimates improving, in this way, the learning performance. We show that the proposed distributed zeroth-order policy optimization method with constant stepsize converges to the neighborhood of a policy that is a stationary point of the global objective function. The size of this neighborhood depends on the agents' learning rates, the exploration parameters, and the number of consensus steps used to calculate the local estimates of the global accumulated rewards. Moreover, we provide numerical experiments that demonstrate that our new zeroth-order policy gradient estimator is more sample-efficient compared to other existing one-point estimators.

Discrete-Time Approximation of Stochastic Optimal Control with Partial Observation

Discrete-Time Approximation of Stochastic Optimal Control with Partial Observation

Double jeopardy: FCPA enforcement and MNC risk-mitigation strategies

Does transnational anti-bribery enforcement affect the risk-mitigation strategies of firms? This paper uses an original dataset on the enforcement actions of the Foreign Corrupt Practices Act (FCPA) to examine the law’s impact on corporate behavior and political risks for multinational corporations (MNCs). I argue that corrupt institutions are not necessarily undesirable for foreign investors. Foreign firms seek above-normal returns in high-risk markets through informal exchanges with the host government. FCPA enforcement provides a “fire alarm” that affects firms differently given their sensitivity to corruption concerns. FCPA enforcement has unequal deterrence against corporate misconduct, encouraging some firms to adopt transparency norms while incentivizing other firms to be more insidious in their corrupt business practices. I use a partial observability bivariate probit model to estimate the unobservable propensity of firms to engage in corrupt exchanges. Then I examine the impact of FCPA enforcement on Chinese FDI, and find that Chinese investments are deterred from markets with robust legal institutions. The FCPA’s deterrence effects against corrupt competitors is a positive outcome for U.S. MNCs. However, American companies experience diminished returns in countries with strong investor protection regimes. External legal interventions under the FCPA generate regulatory burdens on U.S. that limit their business opportunities.

Multi-UAV simultaneous target assignment and path planning based on deep reinforcement learning in dynamic multiple obstacles environments.

Target assignment and path planning are crucial for the cooperativity of multiple unmanned aerial vehicles (UAV) systems. However, it is a challenge considering the dynamics of environments and the partial observability of UAVs. In this article, the problem of multi-UAV target assignment and path planning is formulated as a partially observable Markov decision process (POMDP), and a novel deep reinforcement learning (DRL)-based algorithm is proposed to address it. Specifically, a target assignment network is introduced into the twin-delayed deep deterministic policy gradient (TD3) algorithm to solve the target assignment problem and path planning problem simultaneously. The target assignment network executes target assignment for each step of UAVs, while the TD3 guides UAVs to plan paths for this step based on the assignment result and provides training labels for the optimization of the target assignment network. Experimental results demonstrate that the proposed approach can ensure an optimal complete target allocation and achieve a collision-free path for each UAV in three-dimensional (3D) dynamic multiple-obstacle environments, and present a superior performance in target completion and a better adaptability to complex environments compared with existing methods.

Optimal Investment with a Noisy Signal of Future Stock Prices

We consider an investor who is dynamically informed about the future evolution of one of the independent Brownian motions driving a stock’s price fluctuations. With linear temporary price impact the resulting optimal investment problem with exponential utility turns out to be not only well posed, but it even allows for a closed-form solution. We describe this solution and the resulting problem value for this stochastic control problem with partial observation by solving its convex-analytic dual problem.

A model-free method based on RDPG for fiber diameter steady control

Optical fibers have made outstanding contributions to optical instruments such as micro-channel plates (MCPs) and micro-pore optics (MPOs). As the stricter performance requirements for the MCP and MPO, traditional diameter control methods are incapable. Additionally, the partial observation issue of fiber drawing process further limits the outcome of the existing control approaches. Innovative control algorithms are vital for the diameter precision enhancement. We investigate the significance of recurrent deterministic policy gradient (RDPG) on diameter control. RDPG consists of deep deterministic policy gradient (DDPG) and recurrent neural network (RNN) where DDPG is a model-free reinforcement learning algorithm for continuous control and RNN for addressing partial observation. The impact of RNN variants selection on control performance is further discussed. Results illustrate that the application of RNN substantially solves the partial observation and varying RNN variants play a positive role in diameter control, but bidirectional gated recurrent unit (BiGRU) reaches the best accuracy. Comparison experiment results indicate that RDPG is able to learn reasonable control strategies and achieves significantly more accurate and robust diameter control in contrast with the existing methods.

PHORTEX: Physically-Informed Operational Robotic Trajectories for Scientific Expeditions

Mobile robots are increasingly used to collect valuable in situ samples during scientific expeditions. However, many phenomena of scientific interest—deep-sea hydrothermal plumes, algal blooms, warm-core eddies, and lava flows—are spatiotemporal distributions that evolve on spatial and temporal scales that complicate sample collection. Here, we consider the problem of charting the space-time dynamics of deep-sea hydrothermal plumes with the state-of-the-art autonomous underwater vehicle (AUV) Sentry. In the hydrothermal plume charting problem, the plume state is driven by complicated and unobserved dynamics in the deep sea. To effectively sample the moving plume, an autonomy system must infer plume dynamics from sparse, point observations, while respecting operational constraints of AUV Sentry that restrict the set of possible trajectories to nonadaptive, uniform-coverage patterns. We frame the plume charting problem as a sequential decision-making problem and formulate a mission planner PHORTEX (PHysically-informed Operational Robotic Trajectories for EXpeditions) that strategically designs full mission trajectories for Sentry, where each mission plan is informed by the observations of the last. PHORTEX is composed of a trajectory optimizer, which maximizes expected samples collected within a moving plume, and PHUMES (PHysically-informed Uncertainty Models for Environment Spatiotemporality), a modeling framework that leverages an embedded simulator of idealized plume physics as an inductive bias to enable dynamics learning from extreme partial observations and a few Sentry deployments. In both simulation and in field trials at a hydrothermal site in the Gulf of California, we demonstrate that Sentry using PHORTEX learns to track a moving hydrothermal plume and gather samples that significantly improve upon baseline spatial and temporal diversity for use in downstream science tasks.

Risk-Aware Markov Decision Process Contingency Management Autonomy for Uncrewed Aircraft Systems

Uncrewed aircraft systems (UASs) are being increasingly adopted for a variety of applications. The risk UAS poses to people and property must be kept to acceptable levels. This paper proposes risk-aware contingency management autonomy to prevent an accident in the event of component malfunction, specifically propulsion unit failure and/or battery degradation. The proposed autonomy is modeled as a Markov decision process (MDP), whose solution is a contingency management policy that appropriately executes emergency landing, flight termination, or continuation of planned flight actions. Motivated by the potential for errors in fault/failure indicators, the partial observability of the MDP state space is investigated. The performance of optimal policies is analyzed over varying observability conditions in a high-fidelity simulator. Results indicate that both partially observable MDP and maximum a posteriori MDP policies had similar performance over different state observability criteria, given the nearly deterministic state transition model.

The second-order maximum principle for partially observed optimal controls

In this paper, the stochastic control system under partial observation for singular optimal controls in the classical sense is studied. Note that both the system and the observation contain noise, in which the control variable is allowed to enter into all the coefficient terms. By adding some regularity assumptions, the pointwise second-order maximum principle under partial observation can be obtained by Malliavin calculus. Eventually, an example is given to demonstrate how to find the optimal control.

Condition-Based Maintenance scheduling of an aircraft fleet under partial observability: A Deep Reinforcement Learning approach

Condition-Based Maintenance scheduling of an aircraft fleet under partial observability: A Deep Reinforcement Learning approach

Noisy Tensor Completion via Low-Rank Tensor Ring.

Tensor completion is a fundamental tool for incomplete data analysis, where the goal is to predict missing entries from partial observations. However, existing methods often make the explicit or implicit assumption that the observed entries are noise-free to provide a theoretical guarantee of exact recovery of missing entries, which is quite restrictive in practice. To remedy such drawback, this article proposes a novel noisy tensor completion model, which complements the incompetence of existing works in handling the degeneration of high-order and noisy observations. Specifically, the tensor ring nuclear norm (TRNN) and least-squares estimator are adopted to regularize the underlying tensor and the observed entries, respectively. In addition, a nonasymptotic upper bound of estimation error is provided to depict the statistical performance of the proposed estimator. Two efficient algorithms are developed to solve the optimization problem with convergence guarantee, one of which is specially tailored to handle large-scale tensors by replacing the minimization of TRNN of the original tensor equivalently with that of a much smaller one in a heterogeneous tensor decomposition framework. Experimental results on both synthetic and real-world data demonstrate the effectiveness and efficiency of the proposed model in recovering noisy incomplete tensor data compared with state-of-the-art tensor completion models.

Probabilistic methods for flotation circuit mass balance estimation

Flotation control and supervision allow flotation circuits to maintain target concentrate grades and plant recoveries. The decision-making required for effective supervision of flotation control systems (e.g., selection of targets, setpoints, and setpoint limits) is enhanced by up-to-date information on total and component mass flows throughout the circuit. Limited instrumentation in flotation circuits typically yields a partial, noisy, and infrequently updated view of flotation conditions. Data reconciliation approaches have traditionally been used in conjunction with conservation balances to provide more consistent estimates of mass flows, although unmeasured and unobservable variables are either not fully characterized or at all. In this work, a probabilistic approach to mass balance estimation is presented, which can generalize different use cases (full redundancy, full observability, and partial observability with additional process assumptions) – using maximum a posteriori and marginal density estimation methods. This probabilistic approach is demonstrated on two case studies (a simple illustration and the Brunswick Mining flotation circuit).

Machine Learning Under Partial Observability: The Effect Of The SEC's Labeling Bias On Detecting Financial Misconduct

Machine Learning Under Partial Observability: The Effect Of The SEC's Labeling Bias On Detecting Financial Misconduct

Bayesian High-Rank Hankel Matrix Completion for Nonlinear Synchrophasor Data Recovery

Phasor measurement units (PMUs) provide high temporal-resolution synchrophasor measurements for power system monitoring and control. The frequent data quality issues, such as missing and bad data, prevent the incorporation of synchrophasor data in real-time operations. Most existing data-driven data recovery methods assume the power system dynamics can be approximated by a linear dynamical system, and the recovery performance degrades significantly when the power system is experiencing nonlinear dynamics during significant events. This paper proposes a data-driven Bayesian nonlinear synchrophasor data recovery method (Ba-NSDR) that can recover a consecutive time period of simultaneous data losses or errors across all channels, even when the underlying system is highly nonlinear. The idea is to lift the Hankel matrix of the spatial-temporal synchrophasor data to a higher dimension such that the lifted Hankel matrix is low-rank in that space and can be processed with the kernel trick. Our proposed Bayesian method then infers the probabilistic distributions of synchrophasor from the partial observations. Some distinctive features of Ba-NSDR include an uncertainty index to measure the accuracy of the recovery result and the robustness to parameter selections. Our method is verified on both synthetic and recorded event datasets.