Continuous Decision Spaces Research Articles

Online Markov Decision Processes Configuration with Continuous Decision Space

In this paper, we investigate the optimal online configuration of episodic Markov decision processes when the space of the possible configurations is continuous. Specifically, we study the interaction between a learner (referred to as the configurator) and an agent with a fixed, unknown policy, when the learner aims to minimize her losses by choosing transition functions in online fashion. The losses may be unrelated to the agent's rewards. This problem applies to many real-world scenarios where the learner seeks to manipulate the Markov decision process to her advantage. We study both deterministic and stochastic settings, where the losses are either fixed or sampled from an unknown probability distribution. We design two algorithms whose peculiarity is to rely on occupancy measures to explore with optimism the continuous space of transition functions, achieving constant regret in deterministic settings and sublinear regret in stochastic settings, respectively. Moreover, we prove that the regret bound is tight with respect to any constant factor in deterministic settings. Finally, we compare the empiric performance of our algorithms with a baseline in synthetic experiments.

Conflict in moral and nonmoral decision making: an empirical study coupled with a computational model.

While moral psychology research has extensively studied decision making using moral dilemmas, such high-conflict situations may not fully represent all moral decisions. Moreover, most studies on the effect of conflict have focused on nonmoral decisions, and it is unclear how it applies to the moral realm. The present mixed-method research investigates how conflict impacts moral compared to nonmoral decision making. In a preregistered empirical study ( ), participants made moral and nonmoral decisions with varying levels of conflict while their mouse trajectories were recorded. Results indicate that moral decisions were more stable in the presence of conflict, while still seeking compromise. In addition, decisions were more affected when conflict got higher. Mouse-tracking data further indicate that some factors are impacting the decision process earlier than others, supporting the relevance of tracing methods to dig into finer-grained decision dynamics. We also present a computational model that aims to capture decision mechanisms and how conflict and morality influence decision making. The model uses dynamic neural fields coupled with sensorimotor control to map a continuous decision space. Two model versions were compared: one with greater perceptual weight for moral information, and another with earlier processing of moral versus nonmoral information. The simulated data more successfully reproduced empirical patterns for the second version, thus providing insights into the underlying decision processes for both moral and nonmoral decisions, in the presence of conflict or not.

Novel Data-Driven decentralized coordination model for electric vehicle aggregator and energy hub entities in multi-energy system using an improved multi-agent DRL approach

Energy hub (EH) is an independent entity that benefits to the efficiency, flexibility, and reliability of integrated energy systems (IESs). On the other hand, the rapid emerging of electric vehicles (EVs) drives the EV aggregator (EVAGG) as another independent entity to facilitate the electricity exchange with the grid. However, due to privacy consideration for different owners, it is challenging to investigate the optimal coordinated strategies for such interconnected entities only by exchanging the information of electrical energy. Besides, the existence of parameter uncertainties (load demands, EVs’ charging behaviors, wind power and photovoltaic generation), continuous decision space, dynamic energy flows, and non-convex multi-objective function is difficult to solve. To this end, this paper proposes a novel model-free multi-agent deep reinforcement learning (MADRL) -based decentralized coordination model to minimize the energy costs of EH entities and maximize profits of EVAGGs. First, a long short-term memory (LSTM) module is used to capture the future trend of uncertainties. Then, the coordination problem is formulated as Markov games and solved by the attention enabled MADRL algorithm, where the EH or EVAGG entity is modeled as an adaptive agent. An attention mechanism makes each agent only focus on state information related to the reward. The proposed MADRL adopts the forms of offline centralized training to learn the optimal coordinated control strategy, and decentralized execution to enable agents’ online decisions to only require local measurements. A safety network is employed to cope with equality constraints (demand–supply balance). Simulation results illustrate that the proposed method achieves similar results compared to the traditional model-based method with perfect knowledge of system models, and the computation performance is at least two orders of magnitudes shorter than the traditional method. The testing results of the proposed method are better than those of the Concurrent and other MADRL method, with 10.79%/3.06% lower energy cost and 17.11%/6.82% higher profits of aggregator. Besides, the electric equality constraint of the proposed method is only 0.25 MW averaged per day, which is a small and acceptable violation.

Multi-agent deep reinforcement learning-based coordination control for grid-aware multi-buildings

In commercial buildings, Heat, Ventilation, and Air Conditioning (HVAC) systems account for about 40–50 % of total electricity usage, contributing to an economic burden on building operators. Furthermore, increasing amounts of distributed generation can bring challenges and opportunities for voltage regulation in low voltage distribution networks. In this paper, we tend to develop intelligent management to save the electricity bills of HVAC systems in multiple multi-zone buildings while relieving the stress of voltage regulation across the network. However, it is challenging to achieve the above aims due to the existence of parameter uncertainties (e.g., electricity load, outdoor temperature, photovoltaic generation, etc.), a sizeable continuous decision space, unknown thermal dynamics model, and distribution network topology, and a non-convex multi-objective function. In this context, a novel model-free multi-agent deep reinforcement learning (MADRL)-based multi-building control algorithm is proposed to achieve building-side and grid-level objectives. The proposed method adopts a centralized training and decentralized execution framework while integrating an attention mechanism to ease training and preserve privacy. This also enables the agent to achieve control purposes based only on local measurements, reducing communication cost. Simulation results based on real-world data verify that the proposed method can achieve real-time physical-model-free control of multi-buildings to tackle fast fluctuations of voltage and temperature caused by the uncertain external factors while being advantageous over other two MADRL methods. Additionally, comparison analysis on untouched datasets illustrates that the proposed method achieves similar results and better computation performance with the perfect physical-model-based approach.

Effects of Subjective Biases on Strategic Information Transmission

In this paper, we study the effects of subjective biases on strategic information transmission (SIT) within a Stackelberg game setting, where a human transmitter (leader) communicates an encoded source message to a human receiver (follower) so that the receiver decodes back a desired version of the original source signal. We model human decisions using Rieger–Wang’s prospect theory, which is an extension of traditional prospect theory to continuous decision spaces. Having found a closed-form expression for the receiver’s best response strategy under any general setting, we consider two settings: Gaussian SIT games and exponential SIT games. While the Gaussian SIT games result in strategies that are independent of subjective biases of both the transmitter and the receiver, we show that the equilibrium strategies in exponential SIT games depend on the subjective biases of both the transmitter and the receiver. Numerical results are presented to illustrate results in both Gaussian and exponential settings.

An efficient memory-based electromagnetism-like mechanism for the redundancy allocation problem

An efficient MBEM is proposed to solve the redundancy allocation problem.MBEM employs a memory matrix in local search to save the features of good solutions.Various test problems and benchmarks are used to evaluate the performance of MBEM.Experimental results show that optimal solutions of all benchmark instances are obtained.The computer execution times of the algorithm on all large-scale instances are reasonable. Meta-heuristic algorithms have been successfully applied to solve the redundancy allocation problem in recent years. Among these algorithms, the electromagnetism-like mechanism (EM) is a powerful population-based algorithm designed for continuous decision spaces. This paper presents an efficient memory-based electromagnetism-like mechanism called MBEM to solve the redundancy allocation problem. The proposed algorithm employs a memory matrix in local search to save the features of good solutions and feed it back to the algorithm. This would make the search process more efficient. To verify the good performance of MBEM, various test problems, especially the 33 well-known benchmark instances in the literature, are examined. The experimental results show that not only optimal solutions of all benchmark instances are obtained within a reasonable computer execution time, but also MBEM outperforms EM in terms of the quality of the solutions obtained, even for large-size problems.

The entropic basis of collective behaviour.

We identify a unique viewpoint on the collective behaviour of intelligent agents. We first develop a highly general abstract model for the possible future lives these agents may encounter as a result of their decisions. In the context of these possibilities, we show that the causal entropic principle, whereby agents follow behavioural rules that maximize their entropy over all paths through the future, predicts many of the observed features of social interactions among both human and animal groups. Our results indicate that agents are often able to maximize their future path entropy by remaining cohesive as a group and that this cohesion leads to collectively intelligent outcomes that depend strongly on the distribution of the number of possible future paths. We derive social interaction rules that are consistent with maximum entropy group behaviour for both discrete and continuous decision spaces. Our analysis further predicts that social interactions are likely to be fundamentally based on Weber's law of response to proportional stimuli, supporting many studies that find a neurological basis for this stimulus–response mechanism and providing a novel basis for the common assumption of linearly additive ‘social forces’ in simulation studies of collective behaviour.

Annealing-based particle swarm optimization to solve the redundant reliability problem with multiple component choices

In this paper, the performance of a particle swarm optimization (PSO) algorithm named Annealing-based PSO (APSO) is investigated to solve the redundant reliability problem with multiple component choices (RRP-MCC). This problem aims to choose an optimal combination of components and redundancy levels for a system with a series–parallel configuration that maximizes the overall system reliability. PSO is a population-based meta-heuristic algorithm inspired by the social behavior of the biological swarms that is designed for continuous decision spaces. As a local search engine (LSE), the proposed APSO employs the Metropolis-Hastings strategy, the key idea behind the simulated annealing (SA) algorithm. In APSO, the best position among all particles in each iteration is dynamically improved using the inner loop of the SA (i.e., equilibrium loop) while the temperature is updated in the main loop of the PSO algorithm. The well-known benchmarks are used to verify the performance of the proposed APSO. Even though APSO fails to outperform the best solution obtained in the literature, the contribution of this paper is comprised of the implementation of APSO as a hybrid meta-heuristic as well as the effect of Metropolis-Hastings strategy on the performance of the classical PSO.

METAL: A framework for mixture-of-experts task and attention learning

Rapid increase in the size and complexity of sensory systems demands for attention control in real world robotic tasks. However, attention control and the task are often highly interlaced which demands for interactive learning. In this paper, a framework called METAL mixture-of-experts task and attention learning is proposed to cope with this complex learning problem. METAL consists of three consecutive learning phases, where the first two phases provide an initial knowledge about the task, while in the third phase the attention control is learned concurrently with the task. The mind of the robot is composed of a set of tiny agents learning and acting in parallel in addition to an attention control learning ACL agent. Each tiny agent provides the ACL agent with some partial knowledge about the task in the form of its decision preference-called policy as well. The ACL agent in the third phase learns how to make the final decision by attending the least possible number of tiny agents. It acts on a continuous decision space which gives METAL the ability to integrate different sources of knowledge with ease. A Bayesian continuous RL method is utilized at both levels of learning on perceptual and decision spaces. Implementation of METAL on an E-puck robot in a miniature highway driving task along with farther simulation studies in Webots™ environment verify the applicability and effectiveness of the proposed framework, where a smooth driving behavior is shaped. It is also shown that even though the robot has learned to discard some sensory data, probability of raising aliasing in the decision space is very low, which means that the robot can learn the task as well as attention control simultaneously.

A probability model of system downtime with implications for optimal warranty design

AbstractTraditional approaches toward modeling the availability of a system often do not formally take into account uncertainty over the parameter values of the model. Such models are then frequently criticized because the observed reliability of a system does not match that predicted by the model. This paper extends a recently published segregated failures model so that, rather than providing a single figure for the availability of a system, uncertainty over model parameter values is incorporated and a predictive probability distribution is given. This predictive distribution is generated in a practical way by displaying the uncertainties and dependencies of the parameters of the model through a Bayesian network (BN). Permitting uncertainty in the reliability model then allows the user to determine whether the predicted reliability was incorrect due to inherent variability in the system under study, or due to the use of an inappropriate model. Furthermore, it is demonstrated how the predictive distribution can be used when reliability predictions are employed within a formal decision‐theoretic framework. Use of the model is illustrated with the example of a high‐availability computer system with multiple recovery procedures. An BN is produced to display the relations between parameters of the model in this case and to generate a predictive probability distribution of the system's availability. This predictive distribution is then used to make two decisions under uncertainty concerning the offered warranty policies on the system: a qualitative decision and an optimization over a continuous decision space. Copyright © 2009 John Wiley & Sons, Ltd.

An Improved Continuous Ant Algorithm for Optimization of Water Resources Problems

Ant colony optimization was initially proposed for discrete search spaces while in continuous domains, discretization of the search space has been widely practiced. Attempts for direct extension of ant algorithms to continuous decision spaces are rapidly growing. This paper briefly reviews the central idea and mathematical representation of a recently proposed algorithm for continuous domains followed by further improvements in order to make the algorithm adaptive and more efficient in locating near optimal solutions. Performance of the proposed improved algorithm has been tested on few well-known benchmark problems as well as a real-world water resource optimization problem. The comparison of the results obtained by the present method with those of other ant-based algorithms emphasizes the robustness of the proposed algorithm in searching the continuous space more efficiently as locating the closest, among other ant methods, to the global optimal solution.

Minimax filters for microphone arrays

Conventionally, minimax spatial/spectral filters for microphone arrays are designed by first discretizing the spatial and frequency domains into a finite number of grid points and then performing the optimization over these points. The drawback with this approach is that the response of the spatial/spectral filters in between the grid points can be poor. More recently, an approach that performs the minimax design over the continuum of points in the decision space has appeared in the literature. In this brief, we describe an approach to solving this continuous decision space design problem that is numerically more elegant and efficient. The effectiveness of the new method is illustrated by a numerical example.

An algorithm for the mixed-integer nonlinear bilevel programming problem

The bilevel programming problem (BLPP) is a two-person nonzero sum game in which play is sequential and cooperation is not permitted. In this paper, we examine a class of BLPPs where the leader controls a set of continuous and discrete variables and tries to minimize a convex nonlinear objective function. The follower's objective function is a convex quadratic in a continuous decision space. All constraints are assumed to be linear. A branch and bound algorithm is developed that finds global optima. The main purpose of this paper is to identify efficient branching rules, and to determine the computational burden of the numeric procedures. Extensive test results are reported. We close by showing that it is not readily possible to extend the algorithm to the more general case involving integer follower variables.