Greedy Policy Research Articles

Decentralized Scheduling for Cooperative Localization With Deep Reinforcement Learning

Cooperative localization is a promising solution to the vehicular high-accuracy localization problem. Despite its high potential, exhaustive measurement and information exchange between all adjacent vehicles are expensive and impractical for applications with limited resources. Greedy policies or hand-engineering heuristics may not be able to meet the requirement of complicated use cases. In this paper, we formulate a scheduling problem to improve the localization accuracy (measured through the Cramer-Rao lower bound) of every vehicle up to a given threshold using the minimum number of measurements. The problem is cast as a partially observable Markov decision process and solved using decentralized scheduling algorithms with deep reinforcement learning, which allow vehicles to optimize the scheduling (i.e., the instants to execute measurement and information exchange with each adjacent vehicle) in a distributed manner without a central controlling unit. Simulation results show that the proposed algorithms have a significant advantage over random and greedy policies in terms of both required numbers of measurements to localize all nodes and achievable localization precision with limited numbers of measurements.

Average Age of Information in Wireless Powered Sensor Networks

In this letter, we deal with the age of information (AoI) for a sensor network with wireless power transfer (WPT) capabilities. Specifically, we study a simple network topology, where a sensor node harvests energy from radio frequency signals (transmitted by a dedicated energy source) to transmit real-time status updates. The sensor node generates an update when its capacitor/battery becomes fully charged and transmits by using all the available energy without further energy management. The average AoI performance of the considered greedy policy is derived in closed form and is a function of the capacitor’s size. The optimal value of the capacitor that maximizes the freshness of the information, corresponds to a simple optimization problem requiring a 1-D search. The derived theoretical results provide useful performance bounds for practical WPT networks.

Guaranteed satisficing and finite regret: Analysis of a cognitive satisficing value function

As reinforcement learning algorithms are being applied to increasingly complicated and realistic tasks, it is becoming increasingly difficult to solve such problems within a practical time frame. Hence, we focus on a satisficing strategy that looks for an action whose value is above the aspiration level (analogous to the break-even point), rather than the optimal action. In this paper, we introduce a simple mathematical model called risk-sensitive satisficing (RS) that implements a satisficing strategy by integrating risk-averse and risk-prone attitudes under the greedy policy. We apply the proposed model to the K-armed bandit problems, which constitute the most basic class of reinforcement learning tasks, and prove two propositions. The first is that RS is guaranteed to find an action whose value is above the aspiration level. The second is that the regret (expected loss) of RS is upper bounded by a finite value, given that the aspiration level is set to an “optimal level” so that satisficing implies optimizing. We confirm the results through numerical simulations and compare the performance of RS with that of other representative algorithms for the K-armed bandit problems.

Joint Optimization of Data Offloading and Resource Allocation With Renewable Energy Aware for IoT Devices: A Deep Reinforcement Learning Approach

A large number of connected sensors and devices in Internet of Things (IoT) can generate large amounts of computing data and increase massive energy consumption. Real-time states monitoring and data processing of IoT nodes are of great significance, but the processing power of IoT devices is limited. Using the emerging mobile edge computing (MEC), IoT devices can offload computing tasks to an MEC server associated with small or macro base stations. Moreover, the use of renewable energy harvesting capabilities in base stations or IoT nodes may reduce energy consumption. As wireless channel conditions vary with time and the arrival rates of renewable energy, computing tasks are stochastic, and data offloading and renewable energy aware for IoT devices under a dynamic and unknown environment are major challenges. In this work, we design a data offloading and renewable energy aware model considering an MEC server performing multiple stochastic computing tasks and involving time-varied wireless channels. To optimize data transmission delay, energy consumption, and bandwidth allocation jointly, and to avoid the curse of dimensionality caused by the complexity of the action space, we propose a joint optimization method for data offloading, renewable energy aware, and bandwidth allocation for IoT devices based on deep reinforcement learning (JODRBRL), which can handle the continuous action space. JODRBRL can minimize the total system cost(including data buffer delay cost, energy consumption cost, and bandwidth cost) and obtain an efficient solution by adaptively learning from the dynamic IoT environment. The numerical results demonstrate that JODRBRL can effectively learn the optimal policy, which outperforms Dueling DQN, Double DQN (DDQN), and greedy policy in stochastic environments.

Barrier-Certified Adaptive Reinforcement Learning With Applications to Brushbot Navigation

This paper presents a safe learning framework that employs an adaptive model learning algorithm together with barrier certificates for systems with possibly nonstationary agent dynamics. To extract the dynamic structure of the model, we use a sparse optimization technique. We use the learned model in combination with control barrier certificates which constrain policies (feedback controllers) in order to maintain safety, which refers to avoiding particular undesirable regions of the state space. Under certain conditions, recovery of safety in the sense of Lyapunov stability after violations of safety due to the nonstationarity is guaranteed. In addition, we reformulate an action-value function approximation to make any kernel-based nonlinear function estimation method applicable to our adaptive learning framework. Lastly, solutions to the barrier-certified policy optimization are guaranteed to be globally optimal, ensuring the greedy policy improvement under mild conditions. The resulting framework is validated via simulations of a quadrotor, which has previously been used under stationarity assumptions in the safe learnings literature, and is then tested on a real robot, the brushbot, whose dynamics is unknown, highly complex and nonstationary.

An Optimal Greedy Algorithm for the Single Access Contention Resolution Problem

We present the greedy optimal algorithm for contention resolution (GOAL-CR), a greedy algorithm that solves a variant of the standard contention resolution problem where a set of nodes want to access a shared resource only once, and the objective is to minimize the time it takes for all the nodes to access the resource successfully. These assumptions hold, for instance, in event reporting applications or in the cluster formation phase of wireless sensor networks. We formally prove that the GOAL-CR computes access policies that minimize the expected contention resolution time. We also show, numerically, that the performance of the greedy policies is close to that of a protocol with complete information about the exact number of nodes that have not yet accessed the resource; this latter assumption is hard to fulfill in practice but allows the derivation of a lower bound for the problem. In addition, we show how to adapt the algorithm to scenarios where there is uncertainty in the initial number of nodes and to scenarios where nodes have very limited memory. Finally, we use simulations to show the robustness of the GOAL-CR against asynchronous starts.

Multi Pseudo Q-Learning-Based Deterministic Policy Gradient for Tracking Control of Autonomous Underwater Vehicles.

This paper investigates trajectory tracking problem for a class of underactuated autonomous underwater vehicles (AUVs) with unknown dynamics and constrained inputs. Different from existing policy gradient methods which employ single actor critic but cannot realize satisfactory tracking control accuracy and stable learning, our proposed algorithm can achieve high-level tracking control accuracy of AUVs and stable learning by applying a hybrid actors-critics architecture, where multiple actors and critics are trained to learn a deterministic policy and action-value function, respectively. Specifically, for the critics, the expected absolute Bellman error-based updating rule is used to choose the worst critic to be updated in each time step. Subsequently, to calculate the loss function with more accurate target value for the chosen critic, Pseudo Q-learning, which uses subgreedy policy to replace the greedy policy in Q-learning, is developed for continuous action spaces, and Multi Pseudo Q-learning (MPQ) is proposed to reduce the overestimation of action-value function and to stabilize the learning. As for the actors, deterministic policy gradient is applied to update the weights, and the final learned policy is defined as the average of all actors to avoid large but bad updates. Moreover, the stability analysis of the learning is given qualitatively. The effectiveness and generality of the proposed MPQ-based deterministic policy gradient (MPQ-DPG) algorithm are verified by the application on AUV with two different reference trajectories. In addition, the results demonstrate high-level tracking control accuracy and stable learning of MPQ-DPG. Besides, the results also validate that increasing the number of the actors and critics will further improve the performance.

Optimal Policies and Heuristics To Match Supply With Demand For Online Seasonal Sales

Problem definition: We consider an online retailer selling multiple products to multiple zones over a single period. The retailer orders the products from a single supplier and stores them at multiple warehouses. At the start of the selling period, the retailer determines the order quantities of the products and their storage quantities at each warehouse subject to its capacity constraint. At the end of the period, after knowing the demands, the retailer determines the retrieval quantities from each warehouse to fulfill the demands. The retailer's objective is to maximize her expected profit. \\ Methodology/results: For the single-zone case, we solve the problem optimally. The optimal retrieval policy is a greedy policy. We design a polynomial-time algorithm to determine the optimal storage policy, which preserves a nested property: Among all non-empty warehouses, a smaller-index warehouse contains all the products stored in a larger-index warehouse. The optimal ordering policy is a newsvendor-type policy. The problem becomes intractable analytically if there are multiple zones and we propose an efficient heuristic to solve it. This heuristic involves a non-trivial hybrid approximation of the second-stage expected profit. The heuristic is data driven, which uses demand samples as inputs to solve the problem without knowing the true demand distributions. Our numerical experiments suggest that this heuristic achieves a larger profit in a much shorter time compared to state-of-the-art approaches. Managerial implications: The advantage of our heuristic becomes more obvious as the tail of the demand distribution becomes fatter or as the problem size becomes larger, clearly showing the heuristic's efficiency. A case study based on data from a major fashion online retailer in Asia further confirms the superiority of the heuristic. With flexible fulfillment, our heuristic improves the efficiency by 28% on average compared to a dedicated policy adopted by the retailer.

Global Bandits.

Multiarmed bandits (MABs) model sequential decision-making problems, in which a learner sequentially chooses arms with unknown reward distributions in order to maximize its cumulative reward. Most of the prior works on MAB assume that the reward distributions of each arm are independent. But in a wide variety of decision problems-from drug dosage to dynamic pricing-the expected rewards of different arms are correlated, so that selecting one arm provides information about the expected rewards of other arms as well. We propose and analyze a class of models of such decision problems, which we call global bandits (GB). In the case in which rewards of all arms are deterministic functions of a single unknown parameter, we construct a greedy policy that achieves bounded regret, with a bound that depends on the single true parameter of the problem. Hence, this policy selects suboptimal arms only finitely many times with probability one. For this case, we also obtain a bound on regret that is independent of the true parameter; this bound is sublinear, with an exponent that depends on the informativeness of the arms. We also propose a variant of the greedy policy that achieves worst case and parameter-dependent regret. Finally, we perform experiments on dynamic pricing and show that the proposed algorithms achieve significant gains with respect to the well-known benchmarks.

Scheduling Policies for Minimizing Age of Information in Broadcast Wireless Networks

We consider a wireless broadcast network with a base station sending time-sensitive information to a number of clients through unreliable channels. The Age of Information (AoI), namely the amount of time that elapsed since the most recently delivered packet was generated, captures the freshness of the information. We formulate a discrete-time decision problem to find a transmission scheduling policy that minimizes the expected weighted sum AoI of the clients in the network. We first show that in symmetric networks a Greedy policy, which transmits the packet with highest current age, is optimal. For general networks, we develop three low-complexity scheduling policies: a randomized policy, a Max-Weight policy and a Whittle's Index policy, and derive performance guarantees as a function of the network configuration. To the best of our knowledge, this is the first work to derive performance guarantees for scheduling policies that attempt to minimize AoI in wireless networks with unreliable channels. Numerical results show that both Max-Weight and Whittle's Index policies outperform the other scheduling policies in every configuration simulated, and achieve near optimal performance.

Learning algorithms for scheduling in wireless networks with unknown channel statistics

We study the problem of learning channel statistics to efficiently schedule transmissions in wireless networks subject to interference constraints. We propose an algorithm that uses greedily-constructed schedules in order to learn the channels’ transmission rates, while simultaneously exploiting previous observations to obtain high throughput. Comparison to the offline solution shows our algorithm to have good performance that scales well with the number of links in the network. We then turn our attention to the stochastic setting where packets randomly arrive to the network and await transmission in queues at the nodes. We develop a queue-length-based scheduling policy that uses the channel learning algorithm as a component. We analyze our method in time-varying environments and show that it achieves the same stability region as that of a greedy policy with full channel knowledge.

Putting the Next 500 VM Placement Algorithms to the Acid Test: The Infrastructure Provider Viewpoint

Most current infrastructures for cloud computing leverage static and greedy policies for the placement of virtual machines. Such policies impede the optimal allocation of resources from the infrastructure provider viewpoint. Over the last decade, more dynamic and often more efficient policies based, e.g., on consolidation and load balancing techniques, have been developed. Due to the underlying complexity of cloud infrastructures, these policies are evaluated either using limited scale testbeds/in-vivo experiments or ad-hoc simulators. These validation methodologies are unsatisfactory for two important reasons: they (i) do not model precisely enough real production platforms (size, workload variations, failure, etc.) and (ii) do not enable the fair comparison of different approaches. More generally, new placement algorithms are thus continuously being proposed without actually identifying their benefits with respect to the state of the art. In this article, we show how VMPlaceS, a dedicated simulation framework enables researchers (i) to study and compare VM placement algorithms from the infrastructure perspective, (ii) to detect possible limitations at large scale and (iii) to easily investigate different design choices. Built on top of the SimGrid simulation platform, VMPlaceS provides programming support to ease the implementation of placement algorithms and runtime support dedicated to load injection and execution trace analysis. To illustrate the relevance of VMPlaceS, we first discuss a few experiments that enabled us to study in details three well known VM placement strategies. Diving into details, we also identify several modifications that can significantly increase their performance in terms of reactivity. Second, we complete this overall presentation of VMPlaceS by focusing on the energy efficiency of the well-know FFD strategy. We believe that VMPlaceS will allow researchers to validate the benefits of new placement algorithms, thus accelerating placement research and favouring the transfer of results to IaaS production platforms.

Cooperative Q-learning based channel selection for cognitive radio networks

This paper deals with the jamming attack which may hinder the cognitive radio from efficiently exploiting the spectrum. We model the problem of channel selection as a Markov decision process. We propose a real-time reinforcement learning algorithm based on Q-learning to pro-actively avoid jammed channels. The proposed algorithm is based on wideband spectrum sensing and a greedy policy to learn an efficient real-time strategy. The learning approach is enhanced through cooperation with the receiving CR node based on its sensing results. The algorithm is evaluated through simulations and real measurements with software defined radio equipment. Both simulations and radio measurements reveal that the presented solution achieves a higher packet success rate compared to the classical fixed channel selection and best channel selection without learning. Results are given for various scenarios and diverse jamming strategies.

Spatial Resource Allocation for Emerging Epidemics: A Comparison of Greedy, Myopic, and Dynamic Policies

Rapidly evolving infectious disease epidemics, such as the 2014 West African Ebola outbreak, pose significant health threats and present challenges to the global health community because of their heterogeneous geographic spread. Policy makers must allocate limited intervention resources quickly, in anticipation of where the outbreak is moving next. We develop a two-stage model for optimizing when and where to assign Ebola treatment units across geographic regions during the outbreak’s early phases. The first stage employs a novel dynamic transmission model to forecast the occurrence of new cases at the region level, capturing connectivity among regions. We introduce an empirically estimated coefficient for behavioral adaptation to changing epidemic conditions. The second stage compares four approaches to allocate units across affected regions: (i) a heuristic based on observed cases, (ii) a greedy policy that prioritizes regions based on the reproductive number, (iii) a myopic linear program that allocates resources in the next period based on an iterative estimation–optimization approach coupled with the underlying epidemic model, and (iv) an approximate dynamic programming algorithm that optimizes over all future periods. After testing the allocation schemes under different budgets and time periods, we find that the myopic policy performs best, even when limited data are available. Our methodology could be generalized to other disease outbreaks, including the Zika virus, and other interventions. The online appendix is available at https://doi.org/10.1287/msom.2017.0681 .

Energy Harvesting Communications Without Transmitter Channel State Information Using Layered Coding

In this paper, we optimize an energy harvesting transmitter, communicating over a slow fading channel, using layered coding. The transmitter has access to the channel statistics, but does not know the exact channel state. In layered coding, the codewords are first designed for a given finite set of channel states at different rates, and then the codewords are either time-multiplexed or superimposed before the transmission, leading to two transmission strategies. The receiver then decodes the information adaptively based on the realized channel state. The transmitter is equipped with a finite-capacity battery having non-zero internal resistance. In each of the transmission strategies, we first formulate and study an average rate maximization problem with non-causal knowledge of the harvested power variations. We also highlight the structural properties of the optimal solutions. Further, assuming statistical knowledge and causal information of the harvested power variations, we propose a sub-optimal algorithm, and compare it with the stochastic dynamic programming-based solution and a greedy policy. By numerical simulations, we demonstrate that the average rate decreases by approximately 20% when the internal resistance of the battery is increased from 0 ${\Omega }$ to 50 ${\Omega }$ in all the policies.

Social Learning from Online Reviews with Product Choice

This paper studies product ranking mechanisms of a monopolistic online platform in the presence of social learning. The products' quality is initially unknown, but consumers can sequentially learn it as online reviews accumulate. A salient aspect of our problem is that consumers, who want to purchase a product from a list of items displayed by the platform, incur a search cost while scrolling down the list. In this setting, the social learning dynamics, and hence the demand, is affected by the interplay of two unique features: substitution and ranking effects. The platform can influence the social learning dynamics by adjusting the ranking of the products to ultimately maximize the revenue collected from commission fees for sold items. To formulate the problem in a tractable form, we use a large-market (fluid) approximation and show that consumers eventually learn the products' quality and characterize the speed of learning. Armed with this backing, we formulate the platform's ranking problem in the fluid setting, where we assume the perspective of an uninformed platform that does not know the true quality vector but rather learns it through consumers' review process. We compare different ranking policies based on the worst-case regret with respect to a fully-informed platform benchmark. Our analysis yields three main insights. First, a greedy policy that maximizes immediate revenue by displaying products based on current ratings may incur highly suboptimal worst-case regret, as it may relegate the most profitable products to the lowest positions in the ranking if their current rating is not high enough. Second, a simple variant of the greedy policy can sufficiently alleviate the regret by balancing the trade-off between exploration and exploitation. Third, we characterize the critical level of search cost for which the regret does not grow exponentially with the number of products.

A large data processing algorithm for energy efficiency in a heterogeneous cluster

It is reportedi that the electricity cost to operate a cluster may well exceed its acquisition cost, and the processing of big data requires large scale cluster and long period. Therefore, energy efficient processing of big data is essential for the data owners and users. In this paper, we propose a novel algorithm MinBalance to processing I/O intensive big data tasks energy efficiently in heterogeneous cluster. In the former step, four greedy policies are used to select the proper nodes considering heterogeneity of the cluster. While in the latter step, the workloads of the selected nodes will be well balanced to avoid the energy wastes caused by waiting. MinBalance is a universal algorithm and cannot be affected by the data storage strategies. Experimental results indicate that MinBalance can achieve over 60% energy reduction for large sets over the traditional methods of powering down partial nodes.

Optimal Attack Strategies Against Predictors - Learning From Expert Advice

Motivated by many real-world examples, such as recommendation systems or sensor fusion, and aiming to capture the influence of malicious experts who intentionally degrade the performance of learning systems, we analyze optimal adversarial strategies against the weighted average prediction algorithm in the learning with expert advice framework. All but one expert is honest and the malicious expert’s goal is to sabotage the performance of the algorithm by strategically providing dishonest recommendations. We formulate the problem as a Markov decision process and analyze it under various settings. For the logarithmic loss, somewhat surprisingly, we prove that the optimal strategy for the adversary is the greedy policy, i.e., lying at every step. For the absolute loss, in the 2-experts, discounted cost setting, we prove that the optimal strategy is a threshold policy, where the malicious expert tells the truth until he earns enough weight and then lies afterwards. We extend the results to the infinite horizon problem and find the exact thresholds for the stationary optimal policy. Finally, we use a mean field approach in the $N$ -experts setting to find the optimal strategy when the predictions of the honest experts are independent and identically distributed. We justify our results using simulations throughout this paper.

Matching in Dynamic Imbalanced Markets

We study dynamic matching in exchange markets with easy- and hard-to-match agents. A greedy policy, which attempts to match agents upon arrival, ignores the positive externality that waiting agents generate by facilitating future matchings. We prove that this trade-off between a ``thicker'' market and faster matching vanishes in large markets; A greedy policy leads to shorter waiting times, and more agents matched than any other policy. We empirically confirm these findings in data from the National Kidney Registry. Greedy matching achieves as many transplants as commonly-used policies (1.6\% more than monthly-batching), and shorter patient waiting times.

Interpretable Optimal Stopping

Optimal stopping is the problem of deciding when to stop a stochastic system to obtain the greatest reward, arising in numerous application areas such as finance, healthcare and marketing. State-of-theart methods for high-dimensional optimal stopping involve approximating the value function or the continuation value, and then using that approximation within a greedy policy. Although such policies can perform very well, they are generally not guaranteed to be interpretable; that is, a decision maker may not be able to easily see the link between the current system state and the policy's action. In this paper, we propose a new approach to optimal stopping, wherein the policy is represented as a binary tree, in the spirit of naturally interpretable tree models commonly used in machine learning. We show that the class of tree policies is rich enough to approximate the optimal policy. We formulate the problem of learning such policies from observed trajectories of the stochastic system as a sample average approximation (SAA) problem. We prove that the SAA problem converges under mild conditions as the sample size increases, but that computationally even immediate simplifications of the SAA problem are theoretically intractable. We thus propose a tractable heuristic for approximately solving the SAA problem, by greedily constructing the tree from the top down. We demonstrate the value of our approach by applying it to the canonical problem of option pricing, using both synthetic instances and instances using real S&P-500 data. Our method obtains policies that (1) outperform state-of-the-art noninterpretable methods, based on simulation-regression and martingale duality, and (2) possess a remarkably simple and intuitive structure.