Greedy Exploration Research Articles

Fairness of Interaction in Ranking under Position, Selection, and Trust Bias

Ranking algorithms in online platforms serve not only users on the demand side, but also items on the supply side. While ranking has traditionally presented items in an order that maximizes their utility to users, the uneven interactions that different items receive as a result of such a ranking can pose item fairness concerns. Moreover, interaction is affected by various forms of bias, two of which have received considerable attention: position bias and selection bias. Position bias occurs due to lower likelihood of observation for items in lower ranked positions. Selection bias occurs because interaction is not possible with items below an arbitrary cutoff position chosen by the front-end application at deployment time (i.e., showing only the top- k items). A less studied, third form of bias, trust bias, is equally important, as it makes interaction dependent on rank even after observation, by influencing the item’s perceived relevance. To capture interaction disparity in the presence of all three biases, in this paper we introduce a flexible fairness metric. Using this metric, we develop a post-processing algorithm that optimizes fairness in ranking through greedy exploration and allows a tradeoff between fairness and utility. Our algorithm outperforms state-of-the-art fair ranking algorithms on several datasets.

Sampling Efficient Deep Reinforcement Learning Through Preference-Guided Stochastic Exploration.

Stochastic exploration is the key to the success of the deep Q -network (DQN) algorithm. However, most existing stochastic exploration approaches either explore actions heuristically regardless of their Q values or couple the sampling with Q values, which inevitably introduce bias into the learning process. In this article, we propose a novel preference-guided ϵ -greedy exploration algorithm that can efficiently facilitate exploration for DQN without introducing additional bias. Specifically, we design a dual architecture consisting of two branches, one of which is a copy of DQN, namely, the Q branch. The other branch, which we call the preference branch, learns the action preference that the DQN implicitly follows. We theoretically prove that the policy improvement theorem holds for the preference-guided ϵ -greedy policy and experimentally show that the inferred action preference distribution aligns with the landscape of corresponding Q values. Intuitively, the preference-guided ϵ -greedy exploration motivates the DQN agent to take diverse actions, so that actions with larger Q values can be sampled more frequently, and those with smaller Q values still have a chance to be explored, thus encouraging the exploration. We comprehensively evaluate the proposed method by benchmarking it with well-known DQN variants in nine different environments. Extensive results confirm the superiority of our proposed method in terms of performance and convergence speed.

Efficient and Private Scheduling of Wireless Electric Vehicles Charging Using Reinforcement Learning

Future vehicle-to-grid (V2G) systems require more flexible scheduling to adjust and flatten the peak energy demand. For efficient scheduling and energy trading, the utility provider (UP) needs to keep track of the state of charge (SoC) of vehicle batteries (VBs). However, sharing of SoC of VBs from electric vehicles (EVs) to UP may compromise owner privacy by analyzing the electricity usage in EVs. Therefore, we propose Reinforcement learning (RL)-based demand-side energy management using a rechargeable battery (RB) for enhanced cost-friendly privacy of EVs, efficient scheduling, and accurate billing. With existing Q-Learning-based RL (using <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\epsilon $ </tex-math></inline-formula> -greedy exploration and exploitation), we find that the reward maximization of efficient and private scheduling is often sluggish and incurs convergence issues. Therefore, we develop a genetic algorithm (GA)-based exploration and exploitation, which solves the convergence problems. We develop theoretical analysis and implement numerical results to demonstrate that the proposed GA-based RL framework accelerates convergence and enhances cost-friendly privacy considerably.

Network slice reconfiguration with deep reinforcement learning under variable number of service function chains

Network slicing has been attracting attention as a technology for providing flexible services while satisfying a wide variety of service requirements. On the network slice, individual services are virtually embedded in the form of service function chain (SFC). The embedded SFCs lose the efficiency of their placement due to traffic fluctuations, so they need to be reconfigured to the placement that allows for low-cost network operation at the right time. There are several works that take a deep reinforcement learning (DRL) approach to address such reconfiguration problems. However, these approaches have a fixed number of SFCs to be reconfigured at a time, and do not assume a situation where the number of SFCs varies due to service arrival or termination. This paper proposes Deep reinforcement learning model for Network Slice Reconfiguration with Dummy and Partial greedy exploration (DNSR-DP), an online learning model that addresses the problem of SFC reconfiguration on a specific network slice under the situation where the number of SFCs varies. By introducing dummy state/action, DNSR-DP keeps the state/action representation. Therefore, DNSR-DP can continue to learn one model regardless of the variation of the number of SFCs in the middle of operation. In addition, we introduce an exploration policy for online learning, the partial-ϵ-greedy policy, which enables DNSR-DP to avoid unnecessary exploration and reduce the cost of exploration. Numerical results show that dummy state/action and the partial-ϵ-greedy policy work effectively, and that DNSR-DP can continue stable learning under the situation where the number of SFCs varies, keeping the total cost of reconfiguration low.

Minimax Persistent Monitoring of a network system

We investigate the problem of optimally observing a finite set of targets using a mobile agent over an infinite time horizon. The agent is tasked to move in a network-constrained structure to gather information so as to minimize the worst-case uncertainty about the internal states of the targets. To do this, the agent has to decide its sequence of target-visits and the corresponding dwell-times at each visited target. For a given visiting sequence, we prove that in an optimal dwelling time allocation the peak uncertainty is the same among all the targets. This allows us to formulate the optimization of dwelling times as a resource allocation problem and to solve it using a novel efficient algorithm. Next, we optimize the visiting sequence using a greedy exploration process, using heuristics inspired by others developed in the context of the traveling salesman problem. Numerical results are included to illustrate the contributions.

The reinforcement learning model with heterogeneous learning rate in activity-driven networks

Agent’s learning behavior usually presents biased judgments influenced by many internal and external reasons, we incorporate an improved [Formula: see text]-learning algorithm in the reinforcement learning which is examined with the prisoner’s dilemma game in an activity-driven networks. The heterogeneous learning rate and [Formula: see text]-greedy exploration mechanism are taken into account while modeling decision-making of agents. Simulation results show the proposed reinforcement learning mechanism is conducive to the emergence of defective behavior, i.e. it could maximize one’s expected payoff regardless of its neighbors’ strategy. In addition, we find the temptation gain, vision level and the number of connected edges of activated agents are proportional to the density of defectors. Interestingly, when the inherent learning rate is small, the increase of exploration rate can demote the appearance of defectors, and the decrease of defectors is insignificant by increasing of exploration rate conversely.

Large deviations for the greedy exploration process on configuration models

Large deviations for the greedy exploration process on configuration models

Markov-based genetic algorithm with [formula omitted]-greedy exploration for Indian classical music composition

Recent technological advancements have made it possible to automate some of the intricate activities involved in music production. In this paper, a heuristic method is proposed for automatic composition of Indian Classical Music (ICM). Composing ICM necessitates consideration of raga constraints which distinguishes it from other genres of music. The raga-based music cannot be generated through random exploration of search space. More specifically, certain disordering in note combinations can violate the raga constraints. To address the aforesaid issue, the paper proposes a novel method by combining genetic algorithm with Markov chain model attuned to ICM sequences. Moreover, the proposed method uses ϵ-greedy strategy to balance the exploration and exploitation of musical search space. In order to preserve the musical characteristics, domain-specific genetic operators (i.e., crossover and mutation) are defined. The evaluation function based on music theory is used to direct the search to favorable tracks. The Kullback Liebler (KL) divergence metric is used to analyze the probability distribution of output and sample data sequence. The results indicate that by regulating the exploration rate, the method is capable of generating melodic sequences while maintaining the desired musical note distribution. Moreover, the method converges faster than the conventional genetic algorithm.

Large Deviation Principle for the Greedy Exploration Algorithm over Erdös-Rényi Graphs

We prove a large deviation principle (LDP) for a greedy exploration process on an Erdos-Renyi (ER) graph when the number of nodes goes to this http URL prove our main result we use the general strategy for the study of large deviations of processes proposed by Feng and Kurtz (2006), which is based on the convergence of non-linear semigroups. The rate function can be expressed in a closed form formula and associated optimization problems can be solved explicitly providing the trajectory of the large deviation. In addition we derive a LDP for the size of the maximum independent set discovered by such algorithm and analyze the probability that it exceeds known bounds for the maximal independent set. We also analyze the link between these results and the landscape complexity of the independent set and the exploration dynamic.

A Routing Framework for Heterogeneous Multi-Robot Teams in Exploration Tasks

This letter proposes a routing framework for heterogeneous multi-robot teams in exploration tasks. The proposed framework deals with a combinatorial optimization problem, and provides a new solving algorithm, for Generalized Team Orienteering Problem (GTOP). In this letter, a route optimization problem is formulated for a heterogeneous multi-robot system. A novel problem solver is also proposed based on self-organizing map. The proposed framework has a strong advantage in its scalability because the processing time is independent from the number of robots, and the heterogeneity of the team. The validity of the proposed framework is evaluated in the exploration, and mapping tasks by heterogeneous robot team with overlapping abilities. The simulation results show the effectiveness of the proposed framework, and how it outperforms the conventional greedy exploration scheme.

Latent Structure Matching for Knowledge Transfer in Reinforcement Learning

Reinforcement learning algorithms usually require a large number of empirical samples and give rise to a slow convergence in practical applications. One solution is to introduce transfer learning: Knowledge from well-learned source tasks can be reused to reduce sample request and accelerate the learning of target tasks. However, if an unmatched source task is selected, it will slow down or even disrupt the learning procedure. Therefore, it is very important for knowledge transfer to select appropriate source tasks that have a high degree of matching with target tasks. In this paper, a novel task matching algorithm is proposed to derive the latent structures of value functions of tasks, and align the structures for similarity estimation. Through the latent structure matching, the highly-matched source tasks are selected effectively, from which knowledge is then transferred to give action advice, and improve exploration strategies of the target tasks. Experiments are conducted on the simulated navigation environment and the mountain car environment. The results illustrate the significant performance gain of the improved exploration strategy, compared with traditional ϵ -greedy exploration strategy. A theoretical proof is also given to verify the improvement of the exploration strategy based on latent structure matching.

Deep Evolutionary Networks with Expedited Genetic Algorithms for Medical Image Denoising.

Deep convolutional neural networks offer state-of-the-art performance for medical image analysis. However, their architectures are manually designed for particular problems. On the one hand, a manual designing process requires many trials to tune a large number of hyperparameters andis thus quite a time-consuming task. On the other hand, the fittest hyperparameters that can adapt to source data properties (e.g., sparsity, noisy features) are not able to be quickly identified for target data properties. For instance, the realistic noise in medical images is usually mixed and complicated, and sometimes unknown, leading to challenges in applying existing methods directly and creating effective denoising neural networks easily. In this paper, we present a Genetic Algorithm (GA)-based network evolution approach to search for the fittest genes to optimize network structures automatically. We expedite the evolutionary process through an experience-based greedy exploration strategy and transfer learning. Our evolutionary algorithm procedure has flexibility, which allows taking advantage of current state-of-the-art modules (e.g., residual blocks) to search for promising neural networks. We evaluate our framework on a classic medical image analysis task: denoising. The experimental results on computed tomography perfusion (CTP) image denoising demonstrate the capability of the method to select the fittest genes for building high-performance networks, named EvoNets. Our results outperform state-of-the-art methods consistently at various noise levels.

Multi-Agent Deep Reinforcement Learning-Based Cooperative Spectrum Sensing With Upper Confidence Bound Exploration

In this paper, a multi-agent deep reinforcement learning method was adopted to realize cooperative spectrum sensing in cognitive radio networks. Each secondary user learns an efficient sensing strategy from the sensing results of some of the selected spectra to avoid interference to the primary users and to coordinate with other secondary users. It is necessary to balance exploration and exploitation in the learning process when using deep reinforcement learning methods, helping explain that upper confidence bound with Hoeffding-style bonus has been adopted in this paper to improve the efficiency of exploration. The simulation results verify that the proposed algorithm, when compared with the conventional reinforcement learning methods with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\varepsilon $ </tex-math></inline-formula> -greedy exploration, is much easier to achieve faster convergence speed and better reward performance.

Adaptive Object Tracking via Multi-Angle Analysis Collaboration

Although tracking research has achieved excellent performance in mathematical angles, it is still meaningful to analyze tracking problems from multiple perspectives. This motivation not only promotes the independence of tracking research but also increases the flexibility of practical applications. This paper presents a significant tracking framework based on the multi-dimensional state–action space reinforcement learning, termed as multi-angle analysis collaboration tracking (MACT). MACT is comprised of a basic tracking framework and a strategic framework which assists the former. Especially, the strategic framework is extensible and currently includes feature selection strategy (FSS) and movement trend strategy (MTS). These strategies are abstracted from the multi-angle analysis of tracking problems (observer’s attention and object’s motion). The content of the analysis corresponds to the specific actions in the multidimensional action space. Concretely, the tracker, regarded as an agent, is trained with Q-learning algorithm and -greedy exploration strategy, where we adopt a customized rewarding function to encourage robust object tracking. Numerous contrast experimental evaluations on the OTB50 benchmark demonstrate the effectiveness of the strategies and improvement in speed and accuracy of MACT tracker.

Gaussian processes autonomous mapping and exploration for range-sensing mobile robots

Most of the existing robotic exploration schemes use occupancy grid representations and geometric targets known as frontiers. The occupancy grid representation relies on the assumption of independence between grid cells and ignores structural correlations present in the environment. We develop a Gaussian Processes (GPs) occupancy mapping technique that is computationally tractable for online map building due to its incremental formulation and provides a continuous model of uncertainty over the map spatial coordinates. The standard way to represent geometric frontiers extracted from occupancy maps is to assign binary values to each grid cell. We extend this notion to novel probabilistic frontier maps computed efficiently using the gradient of the GP occupancy map. We also propose a mutual information-based greedy exploration technique built on that representation that takes into account all possible future observations. A major advantage of high-dimensional map inference is the fact that such techniques require fewer observations, leading to a faster map entropy reduction during exploration for map building scenarios. Evaluations using the publicly available datasets show the effectiveness of the proposed framework for robotic mapping and exploration tasks.

Active Autonomous Aerial Exploration for Ground Robot Path Planning

We address the problem of planning a path for a ground robot through unknown terrain, using observations from a flying robot. In search and rescue missions, which are our target scenarios, the time from arrival at the disaster site to the delivery of aid is critically important. Previous works required exhaustive exploration before path planning, which is time-consuming but eventually leads to an optimal path for the ground robot. Instead, we propose active exploration of the environment, where the flying robot chooses regions to map in a way that optimizes the overall response time of the system, which is the combined time for the air and ground robots to execute their missions. In our approach, we estimate terrain classes throughout our terrain map, and we also add elevation information in areas where the active exploration algorithm has chosen to perform 3-D reconstruction. This terrain information is used to estimate feasible and efficient paths for the ground robot. By exploring the environment actively, we achieve superior response times compared to both exhaustive and greedy exploration strategies. We demonstrate the performance and capabilities of the proposed system in simulated and real-world outdoor experiments. To the best of our knowledge, this is the first work to address ground robot path planning using active aerial exploration.

Speeding-Up Robot Exploration by Exploiting Background Information

The ability to autonomously learn a model of an environment is an important capability of a mobile robot. In this paper, we investigate the problem of exploring a scene given background information in form of a topo-metric graph of the environment. Our method is relevant for several real-world applications in which the rough structure of the environment is known beforehand. We present an approach that exploits such background information and enables a robot to cover the environment with its sensors faster compared to a greedy exploration system without this information. We implemented our exploration system in ROS and evaluated it in different environments. As the experimental results demonstrate, our proposed method significantly reduces the overall trajectory length needed to cover the environment with the robot’s sensors and thus yields a more efficient exploration strategy compared to state-of-the-art greedy exploration, if the additional information is available.

Learning to soar: Resource-constrained exploration in reinforcement learning

This paper examines temporal difference reinforcement learning with adaptive and directed exploration for resource-limited missions. The scenario considered is that of an unpowered aerial glider learning to perform energy-gaining flight trajectories in a thermal updraft. The presented algorithm, eGP-SARSA(λ), uses a Gaussian process regression model to estimate the value function in a reinforcement learning framework. The Gaussian process also provides a variance on these estimates that is used to measure the contribution of future observations to the Gaussian process value function model in terms of information gain. To avoid myopic exploration we developed a resource-weighted objective function that combines an estimate of the future information gain using an action rollout with the estimated value function to generate directed explorative action sequences. A number of modifications and computational speed-ups to the algorithm are presented along with a standard GP-SARSA(λ) implementation with -greedy exploration to compare the respective learning performances. The results show that under this objective function, the learning agent is able to continue exploring for better state-action trajectories when platform energy is high and follow conservative energy-gaining trajectories when platform energy is low.

A lightweight dynamic optimization methodology and application metrics estimation model for wireless sensor networks

Technological advancements in embedded systems due to Moore's law have led to the proliferation of wireless sensor networks (WSNs) in different application domains (e.g., defense, health care, surveillance systems) with different application requirements (e.g., lifetime, reliability). Many commercial-off-the-shelf (COTS) sensor nodes can be specialized to meet these requirements using tunable parameters (e.g., processor voltage and frequency) to specialize the operating state. Since a sensor node's performance depends greatly on environmental stimuli, dynamic optimizations enable sensor nodes to automatically determine their operating state in situ. However, dynamic optimization methodology development given a large design space and resource constraints (memory and computational) is an extremely challenging task. In this paper, we propose a lightweight dynamic optimization methodology that intelligently selects initial tunable parameter values to produce a high-quality initial operating state in one-shot for time-critical or highly constrained applications. Further operating state improvements are made using an efficient greedy exploration algorithm, achieving optimal or near-optimal operating states while exploring only 0.04% of the design space on average. We also propose an application metrics estimation model, which is leveraged by our dynamic optimization methodology, to estimate high-level application metrics (e.g., lifetime, throughput) from sensor node tunable parameters and hardware specific internals.