Deterministic Policies Research Articles

Using deep reinforcement learning (DRL) for minimizing power consumption in video-on-demand (VoD) storage systems

As video streaming services such as Netflix become popular, resolving the problem of high power consumption arising from both large data size and high bandwidth in video storage systems has become important. However, because various factors, such as the power characteristics of heterogeneous storage devices, variable workloads, and disk array models, influence storage power consumption, reducing power consumption with deterministic policies is ineffective. To address this, we present a new deep reinforcement learning (DRL)-based file placement algorithm for replication-based video storage systems, which aims to minimize overall storage power consumption. We first model the video storage system with time-varying streaming workloads as the DRL environment, in which the agent aims to find power-efficient file placement. We then propose a proximal policy optimization (PPO) algorithm, consisting of (1) an action space that determines the placement of each file; (2) an observation space that allows the agent to learn a power-efficient placement based on the current I/O bandwidth utilization; (3) a reward model that assigns a greater penalty for increased power consumption for each action; and (4) an action masking model that supports effective learning by preventing agents from selecting unnecessary actions. Extensive simulations were performed to evaluate the proposed scheme under various solid-state disk (SSD) models and replication configurations. Results show that our scheme reduces storage power consumption by 5% to 25.8% (average 12%) compared to existing benchmark methods known to be effective for file placement.

Relaxed Equilibria for Time-Inconsistent Markov Decision Processes

This paper considers an infinite-horizon Markov decision process (MDP) that allows for general nonexponential discount functions in both discrete and continuous time. Because of the inherent time inconsistency, we look for a randomized equilibrium policy (i.e., relaxed equilibrium) in an intrapersonal game between an agent’s current and future selves. When we modify the MDP by entropy regularization, a relaxed equilibrium is shown to exist by a nontrivial entropy estimate. As the degree of regularization diminishes, the entropy-regularized MDPs approximate the original MDP, which gives the general existence of a relaxed equilibrium in the limit by weak convergence arguments. As opposed to prior studies that consider only deterministic policies, our existence of an equilibrium does not require any convexity (or concavity) of the controlled transition probabilities and reward function. Interestingly, this benefit of considering randomized policies is unique to the time-inconsistent case.

Policy Compression for Intelligent Continuous Control on Low-Power Edge Devices.

Interest in deploying deep reinforcement learning (DRL) models on low-power edge devices, such as Autonomous Mobile Robots (AMRs) and Internet of Things (IoT) devices, has seen a significant rise due to the potential of performing real-time inference by eliminating the latency and reliability issues incurred from wireless communication and the privacy benefits of processing data locally. Deploying such energy-intensive models on power-constrained devices is not always feasible, however, which has led to the development of model compression techniques that can reduce the size and computational complexity of DRL policies. Policy distillation, the most popular of these methods, can be used to first lower the number of network parameters by transferring the behavior of a large teacher network to a smaller student model before deploying these students at the edge. This works well with deterministic policies that operate using discrete actions. However, many real-world tasks that are power constrained, such as in the field of robotics, are formulated using continuous action spaces, which are not supported. In this work, we improve the policy distillation method to support the compression of DRL models designed to solve these continuous control tasks, with an emphasis on maintaining the stochastic nature of continuous DRL algorithms. Experiments show that our methods can be used effectively to compress such policies up to 750% while maintaining or even exceeding their teacher's performance by up to 41% in solving two popular continuous control tasks.

Optimal Decision Making Under Strategic Behavior

We are witnessing an increasing use of data-driven predictive models to inform decisions in high-stakes situations, from lending and hiring to university admissions. As decisions have implications for individuals and society, there is increasing pressure on decision makers to be transparent about their decision policies. At the same time, individuals may use knowledge, gained by transparency, to invest effort strategically in order to maximize their chances of receiving a beneficial decision. Our goal is to find decision policies that are optimal in terms of utility in such a strategic setting. First, we characterize how strategic investment of effort by individuals leads to a change in the feature distribution. Using this characterization, we show that, in general, optimal decision policies are hard to find in polynomial time, and there are cases in which deterministic policies are suboptimal. Then, we demonstrate that, if the cost individuals pay to change their features satisfies a natural monotonicity assumption, we can narrow down the search for the optimal policy to a particular family of decision policies with a set of desirable properties, which allow for a highly effective polynomial time-heuristic search algorithm using dynamic programming. Finally, under no assumptions on the cost, we develop an iterative search algorithm that is guaranteed to converge to locally optimal decision policies. Experiments on synthetic and real credit card data illustrate our theoretical findings and show that the decision policies found by our algorithms achieve higher utility than those that do not account for strategic behavior. This paper was accepted by Vivek Farias, data science. Funding: This work was supported by Machine Learning Cluster of Excellence [EXC #2064/1, Project #390727645], H2020 European Research Council [Grant 945719], and Bundesministerium für Bildung und Forschung [Grant 01IS18039B]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2021.02567

Fully Spiking Actor Network With Intralayer Connections for Reinforcement Learning.

With the help of special neuromorphic hardware, spiking neural networks (SNNs) are expected to realize artificial intelligence (AI) with less energy consumption. It provides a promising energy-efficient way for realistic control tasks by combining SNNs with deep reinforcement learning (DRL). In this article, we focus on the task where the agent needs to learn multidimensional deterministic policies to control, which is very common in real scenarios. Recently, the surrogate gradient method has been utilized for training multilayer SNNs, which allows SNNs to achieve comparable performance with the corresponding deep networks in this task. Most existing spike-based reinforcement learning (RL) methods take the firing rate as the output of SNNs, and convert it to represent continuous action space (i.e., the deterministic policy) through a fully connected (FC) layer. However, the decimal characteristic of the firing rate brings the floating-point matrix operations to the FC layer, making the whole SNN unable to deploy on the neuromorphic hardware directly. To develop a fully spiking actor network (SAN) without any floating-point matrix operations, we draw inspiration from the nonspiking interneurons found in insects and employ the membrane voltage of the nonspiking neurons to represent the action. Before the nonspiking neurons, multiple population neurons are introduced to decode different dimensions of actions. Since each population is used to decode a dimension of action, we argue that the neurons in each population should be connected in time domain and space domain. Hence, the intralayer connections are used in output populations to enhance the representation capacity. This mechanism exists extensively in animals and has been demonstrated effectively. Finally, we propose a fully SAN with intralayer connections (ILC-SAN). Extensive experimental results demonstrate that the proposed method outperforms the state-of-the-art performance on continuous control tasks from OpenAI gym. Moreover, we estimate the theoretical energy consumption when deploying ILC-SAN on neuromorphic chips to illustrate its high energy efficiency.

Multi-Agent Deep Reinforced Co-Dispatch of Energy and Hydrogen Storage in Low-Carbon Building Clusters

Low-carbon communities integrate various energy producers and consumers and have been forming an important architecture for future multi-energy system to tackle challenges in improving energy utilization efficiency, alleviating environmental pollution, and achieving reliability of energy supply systems. This paper proposes a multi-building network model containing hydrogen energy storage systems, and verifies the applicability of the multi-agent deep deterministic policy gradient (MADDPG) algorithm in solving the multi-energy coordinated dispatch of a multi-building network by comparing various energy dispatch methods. First, the model of hydrogen storage system based on liquid organic hydrogen carrier (LOHC) is completed by unifying the process of electrolysis, storage and release of energy from hydrogen storage system, and an energy dispatch method is further constructed by utilizing the hydrogen storage system, heat storage system, and renewable energy generation equipment in the building for coordinated energy supply to the load. Secondly, the observation space and action space of grid-interactive building multi-energy dispatch in the network are respectively established by considering the ability of MADDPG algorithm to learn deterministic policies for continuous actions. Finally, the advantage of grid-interactive buildings containing hydrogen storage systems to reduce operating costs is validated in a numerical simulation based on three dispatch scenarios for six buildings. Four deep reinforcement learning (DRL) algorithms are used to compare the operating costs with model predictive control (MPC) algorithms in a network model containing six grid-interactive buildings, and validate the advantages of MADDPG in optimizing the cost of co-dispatching multi-energy sources in networked grid-interactive buildings.

Diverse Imitation Learning via Self-Organizing Generative Models.

Imitation learning (IL) is a well-known problem in the field of Markov decision process (MDP), where one is given multiple demonstration trajectories generated by expert(s), and the goal is to replicate the hidden expert-policies so that when the MDP is run independently, it generates trajectories close to the demonstrated ones. IL is one of the most useful tools used in building versatile robots that can learn from examples. This task becomes particularly challenging when the expert exhibits a mixture of behavior modes. Prior work has introduced latent variables to model variations of the expert policy. However, our experiments show that the existing works do not exhibit appropriate imitation of individual modes. To tackle this problem, we first draw inspiration from the well-known classical technique of self-organizing maps (SOMs) and introduce an encoder-free generative model-referred to as the self-organizing generative (SOG) model-for learning multimodal data distributions from samples. We then apply SOG for behavior cloning (BC)-a framework that learns deterministic policies without considering the environment-to accurately distinguish and imitate different modes. Then, we integrate it with generative adversarial IL (GAIL)-a framework that learns policies while considering the environment-to make the learning robust toward compounding errors at unseen states. We show that our method significantly outperforms the state of the art across multiple experiments within the MuJoCo simulator, including locomotion and robotic manipulation tasks.

Risk‐sensitive Markov decision processes with long‐run CVaR criterion

CVaR (Conditional value at risk) is a risk metric widely used in finance. However, dynamically optimizing CVaR is difficult, because it is not a standard Markov decision process (MDP) and the principle of dynamic programming fails. In this paper, we study the infinite‐horizon discrete‐time MDP with a long‐run CVaR criterion, from the view of sensitivity‐based optimization. By introducing a pseudo‐CVaR metric, we reformulate the problem as a bilevel MDP model and derive a CVaR difference formula that quantifies the difference of long‐run CVaR under any two policies. The optimality of deterministic policies is derived. We obtain a so‐called Bellman local optimality equation for CVaR, which is a necessary and sufficient condition for locally optimal policies and only necessary for globally optimal policies. A CVaR derivative formula is also derived for providing more sensitivity information. Then we develop a policy iteration type algorithm to efficiently optimize CVaR, which is shown to converge to a local optimum in mixed policy space. Furthermore, based on the sensitivity analysis of our bilevel MDP formulation and critical points, we develop a globally optimal algorithm. The piecewise linearity and segment convexity of the optimal pseudo‐CVaR function are also established. Our main results and algorithms are further extended to optimize the mean and CVaR simultaneously. Finally, we conduct numerical experiments relating to portfolio management to demonstrate the main results. Our work sheds light on dynamically optimizing CVaR from a sensitivity viewpoint.

Age-Optimal Low-Power Status Update Over Time-Correlated Fading Channel

In this paper, we consider transmission scheduling in a status update system, where updates are generated periodically and transmitted over a Gilbert-Elliott fading channel. The goal is to minimize the long-run average age of information (AoI) under a long-run average energy constraint. We consider two practical cases to obtain channel state information (CSI): (i) without channel sensing and (ii) with delayed channel sensing. For (i), CSI is revealed by the feedback (ACK/NACK) of a transmission, but when no transmission occurs, CSI is not revealed. Thus, we have to balance tradeoffs across energy, AoI, channel exploration, and channel exploitation. The problem is formulated as a constrained partially observable Markov decision process (POMDP). We show that the optimal policy is a randomized mixture of no more than two stationary deterministic policies each of which is of a threshold-type in the belief on the channel. For (ii), (delayed) CSI is available via channel sensing. Then, the tradeoff is only between the AoI and energy. The problem is formulated as a constrained MDP. The optimal policy is shown to have a similar structure as in (i) but with an AoI associated threshold. With these, we develop an optimal structure-aware algorithm for each case.

Approximability and efficient algorithms for constrained fixed-horizon POMDPs with durative actions

Partially Observable Markov Decision Process (POMDP) is a fundamental model for probabilistic planning in stochastic domains. More recently, constrained POMDP and chance-constrained POMDP extend the model allowing constraints to be specified on some aspects of the policy in addition to the objective function. Despite their expressive power, these models assume all actions take a fixed duration, which poses a limitation in modeling real-world planning problems. In this work, we propose a unified model for durative POMDP and its constrained extensions. First, we convert these extensions into an Integer Linear Programming (ILP) formulation, which can be solved using existing solvers in the ILP literature. Second, a heuristic search approach is provided that can efficiently prune the search space, guided by solving successive partial ILP programs. Third, we give a theoretical analysis of the problem: unlike short-horizon POMDPs, with policies of a constant depth, which can be solved in polynomial time, the constrained extensions are NP-Hard even with a planning horizon of two and non-negative rewards. To alleviate that, we propose a Fully Polynomial Time Approximation Scheme (FPTAS) that computes (near) optimal deterministic policies in polynomial time. The FPTAS is among the best achievable in theory in terms of approximation ratio. Finally, evaluation results show that our approach is empirically superior to the state-of-the-art fixed-horizon chance-constrained POMDP solver.

ACC-RL: Adaptive Congestion Control Based on Reinforcement Learning in Power Distribution Networks with Data Centers

Modern data center power distribution networks place greater demands on the stability and reliability of power supply. Growing network computing demands and complex network environments can cause network congestion, which in turn leads to network traffic overload and power supply equipment overload. Therefore, network congestion is one of the most important problems faced by data center power distribution networks. In this paper, we propose an approach called ACC-RL based on reinforcement learning (RL), which can effectively avoid network congestion and improve energy performance. ACC-RL models the congestion control task as a Partially Observable Markov Decision Process (POMDP). It is independent of the estimated value function and supports deterministic policies. It also sets the reward value function using real-time network information such as the transmission rate, RTT, and switch queue length, with the target transmission rate as the target equilibrium point. ACC-RL is highly general, can be trained on datasets running in different network environments, and generates a robust congestion control policy. The experimental results show that ACC-RL can solve the congestion problem without any predefined scenarios in different network environments. It can control the network traffic well, thus ensuring the stability and reliability of the power supply in the distribution network. We conduct network simulation experiments through NS-3. We set up different scenarios for experiments and data analysis in many-to-one, all-to-all, and long–short network environments. Compared with the popular rule-based congestion control algorithms such as TIMELY, DCQCN, and HPCC, ACC-RL shows different degrees of energy performance advantages in network metrics such as fairness, link utilization, and throughput.

Composable energy policies for reactive motion generation and reinforcement learning

In this work, we introduce composable energy policies (CEP), a novel framework for multi-objective motion generation. We frame the problem of composing multiple policy components from a probabilistic view. We consider a set of stochastic policies represented in arbitrary task spaces, where each policy represents a distribution of the actions to solve a particular task. Then, we aim to find the action in the configuration space that optimally satisfies all the policy components. The presented framework allows the fusion of motion generators from different sources: optimal control, data-driven policies, motion planning, and handcrafted policies. Classically, the problem of multi-objective motion generation is solved by the composition of a set of deterministic policies, rather than stochastic policies. However, there are common situations where different policy components have conflicting behaviors, leading to oscillations or the robot getting stuck in an undesirable state. While our approach is not directly able to solve the conflicting policies problem, we claim that modeling each policy as a stochastic policy allows more expressive representations for each component in contrast with the classical reactive motion generation approaches. In some tasks, such as reaching a target in a cluttered environment, we show experimentally that CEP additional expressivity allows us to model policies that reduce these conflicting behaviors. A field that benefits from these reactive motion generators is the one of robot reinforcement learning. Integrating these policy architectures with reinforcement learning allows us to include a set of inductive biases in the learning problem. These inductive biases guide the reinforcement learning agent towards informative regions or improve collision safety while exploring. In our work, we show how to integrate our proposed reactive motion generator as a structured policy for reinforcement learning. Combining the reinforcement learning agent exploration with the prior-based CEP, we can improve the learning performance and explore safer.

Deterministic optimal control for discrete-time systems with multiplicative noises and random coefficients

This paper studies a linear-quadratic optimal control problem for discrete-time systems with multiplicative noise and random coefficients. Motivated by fiscal policy planning problems, where deterministic policies are required, this paper aims to find a deterministic optimal controller. The challenge is to seek a solution to forward and backward stochastic difference equations with structural inconsistency caused by random and deterministic terms. We derive necessary and sufficient conditions to solve this optimal control problem by employing a decoupling method. We also present an explicit expression of the optimal controller based on coupled stochastic Riccati-type difference equations.

Multi-objective deep reinforcement learning for optimal design of wind turbine blade

The design of a wind turbine blade is a typical complex multi-objective optimization problem, mostly solved by evolutionary algorithms. However, these methods are not effective due to limitations such as inaccurate solutions on Pareto fronts for high-dimensional problems, numerous iterations and low adaptability to problems with similar conditions. To address these issues, two multi-objective deep reinforcement learning models are introduced in this paper from an entirely different perspective. The first model, namely the multi-objective deep deterministic policy gradient (MO-DDPG), extends the existing popular reinforcement learning algorithm DDPG to multi-objective optimization problems by integrating various techniques including modeling of constraints on high-dimensional spaces and generation of Pareto solutions. The second model, namely the multi-objective deep stochastic policy gradient (MO-DSPG), further improves the MO-DDPG by incorporating a random neural network called restricted Boltzmann machine (RBM). An adaptive random agent is trained to transform multiple deterministic policies into an optimal stochastic policy. In addition, neighborhood-based parameter transfer strategy is applied to MO-DSPG in the model training phase to reduce the computation time. Experiments showed that the aerodynamic performance of the blades is improved by both the MO-DDPG and the MO-DSPG models with the hypervolume increasing an average of 6.67% and 9.25% respectively, compared with the state-of-art models. The computational efficiency of MO-DSPG is improved by using the parameter transfer strategy, with its runtime reduced to 72.52% compared with state-of-art models.

An actor-critic framework based on deep reinforcement learning for addressing flexible job shop scheduling problems.

With the rise of Industry 4.0, manufacturing is shifting towards customization and flexibility, presenting new challenges to meet rapidly evolving market and customer needs. To address these challenges, this paper suggests a novel approach to address flexible job shop scheduling problems (FJSPs) through reinforcement learning (RL). This method utilizes an actor-critic architecture that merges value-based and policy-based approaches. The actor generates deterministic policies, while the critic evaluates policies and guides the actor to achieve the most optimal policy. To construct the Markov decision process, a comprehensive feature set was utilized to accurately represent the system's state, and eight sets of actions were designed, inspired by traditional scheduling rules. The formulation of rewards indirectly measures the effectiveness of actions, promoting strategies that minimize job completion times and enhance adherence to scheduling constraints. The experimental evaluation conducted a thorough assessment of the proposed reinforcement learning framework through simulations on standard FJSP benchmarks, comparing the proposed method against several well-known heuristic scheduling rules, related RL algorithms and intelligent algorithms. The results indicate that the proposed method consistently outperforms traditional approaches and exhibits exceptional adaptability and efficiency, particularly in large-scale datasets.

An anytime algorithm for constrained stochastic shortest path problems with deterministic policies

Sequential decision-making problems arise in every arena of daily life and pose unique challenges for research in decision-theoretic planning. Although there has been a wide variety of research in this field, most of the studies have largely focused on single objective problem without constraints. In many real-world applications, however, it is often desirable to bound certain costs or resources under some predefined level. Constrained stochastic shortest path problem (C-SSP), one of the most well-known mathematical frameworks for stochastic decision-making problems with constraints, can formally model such problems, by incorporating constraints in the model formulation. However, it remains an open challenge to produce a deterministic optimal policy with desirable computation time due to its intrinsic complexity.In this paper, we propose a method that produces an optimal and deterministic policy for a C-SSP based on the Lagrangian duality theory and the heuristic forward search method. To address the intrinsic complexity of C-SSP, the proposed method is designed to have an anytime property. In other words, the proposed algorithm tries to find a feasible but decent solution quickly, then improves the solution incrementally until it converges to a true optimal solution. An extensive experimental evaluation on three problem domains shows that the proposed method outperforms the state-of-the-art methods in terms of the near-optimal solution with an optimality gap of less than 0.1%.

Autonomous driving planning and decision making based on game theory and reinforcement learning

AbstractAutonomous driving technology is one of the important methods that avoid the hidden dangers of traffic safety. Although the existing autonomous driving technology can meet some needs of real traffic scenarios, the ability of multi‐agents to effectively generate autonomous driving strategy in complex traffic environments remains to be improved. Aiming at this problem, the automatic drive model based on game theory and reinforcement learning is proposed by combining these two technologies and applying them in multi‐agent cooperative driving, which enables multi‐agents to carry out strategic reasoning with negotiation in traffic scenarios by extending the game description language, and puts forward the constrained multi‐agents deep deterministic policies gradient algorithm. Finally, the related experiments are conducted through the autonomous driving simulation platform, and the experimental results show that the proposed model can effectively generate driving strategies for multi‐agents in complex traffic environments, which verifies the validity and feasibility of the proposed model, and provides the general research basis for multi‐agents autonomous driving.

OM-TCN: A dynamic and agile opponent modeling approach for competitive games

The non-stationarity of the environment is a crucial challenge for competitive Multi-Agent Reinforcement Learning (MARL) due to the constantly changing opponent policy. Existing schemes are challenging to make the protagonist agent that agilely responds to the opponent’s changes and the resulting non-stationarity, which may inevitably limit their applicability. To address the dynamic opponent policy and adapt to the non-stationary environment continuously, we propose a Temporal Convolutional Network (TCN) model for modeling and predicting opponent behaviors called OM-TCN, and apply it to the widely-used Multi-Agent Deep Deterministic Policies Gradient (MADDPG) algorithm of competitive MARL. In this work, we collect the opponent’s behavior data observed by the protagonist agent and serialize it in granularity of episodes. Then we input the time-series data into OM-TCN for sequence modeling. The OM-TCN learns the historical behaviors of the opponent instead of overfitting to a specific opponent policy, and can make predictions about the opponent’s future actions. Finally, we use predictions of opponent actions in place of the history sampled from the playback buffer, and apply the OM-TCN model to the MADDPG framework for decentralized training. We use the competitive scenario of Multi-agent Particle Environment (MPE) to evaluate the proposed method. Simulation results show that the protagonist agent is able to learn more efficient and stable policy and converge easier than other baselines.

Convex analytic method revisited: Further optimality results and performance of deterministic policies in average cost stochastic control

The convex analytic method has proved to be a very versatile method for the study of infinite horizon average cost optimal stochastic control problems. In this paper, we revisit the convex analytic method and make three primary contributions: (i) We present an existence result for controlled Markov models that lack weak continuity of the transition kernel but are strongly continuous in the action variable for every fixed state variable. (ii) For average cost stochastic control problems in standard Borel spaces, while existing results establish the optimality of stationary (possibly randomized) policies, few results are available on the optimality of deterministic policies. We review existing results and present further conditions under which an average cost optimal stochastic control problem admits optimal solutions that are deterministic stationary. (iii) We establish conditions under which the performance under stationary deterministic (and also quantized) policies is dense in the set of performance values under randomized stationary policies.

Stochastic RWA and Lightpath Rerouting in WDM Networks

In a telecommunication network, routing and wavelength assignment (RWA) is the problem of finding lightpaths for incoming connection requests. When facing a dynamic traffic, greedy assignment of lightpaths to incoming requests based on predefined deterministic policies leads to a fragmented network that cannot make use of its full capacity because of stranded bandwidth. At this point, service providers try to recover the capacity via a defragmentation process. We study this setting from two perspectives: (i) while granting the connection requests via the RWA problem and (ii) during the defragmentation process by lightpath rerouting. For both problems, we present the first two-stage stochastic integer programming model incorporating incoming request uncertainty to maximize the expected grade of service. We develop a decomposition-based solution approach, which uses various relaxations of the problem and a newly developed problem-specific cut family. Simulation of two-stage policies for a variety of instances in a rolling-horizon framework of 52 stages shows that our stochastic models provide high-quality solutions when compared with traditionally used deterministic ones. Specifically, the proposed provisioning policies yield improvements of up to 19% in overall grade of service and 20% in spectrum saving, while the stochastic lightpath rerouting policies grant up to 36% more requests, using up to just 4% more bandwidth spectrum.Summary of Contribution: For handling the intrinsic uncertainty of demand in the telecommunications industry, this paper proposes novel stochastic models and solution methodology for two fundamental problems in telecommunications at operational level: (i) routing and wavelength assignment (RWA) and (ii) lightpath rerouting problem. Despite the vast literature on the RWA problem, stochastic optimization has not been considered as a viable solution for resource allocation in optical networks. We propose two-stage stochastic programming models for both problems and design efficient decomposition-based solution methods that use various relaxations of the models and a new family of cutting planes. Our extensive and rigorous numerical experiments show the significant merit of incorporating uncertainty into decision making, as well as the effectiveness of the decomposition framework and our newly designed family of cuts in enhancing the solvability of both models. This work opens new avenues to explore where the powerful stochastic programming literature can be leveraged to make operational decisions in telecommunications problems, a field that currently relies mostly on deterministic and heuristic solution methods.