Regret Learning Research Articles

An ECA Regret Learning Game for Cross-Tier Computation Offloading Against Swarm Attacks in Sensor Edge Cloud

The distributed nature of multi-tier swarm attacks renders it more difficult for a single-tier intrusion detection system (IDS) to secure cross-tier computation offloading in multi-tier sensor edge cloud (SEC). To perceive and prevent such attacks, we model IDSs in different layers as an IDS federation network (IDFN) and present a generic framework to prevent cooperative attacks and reconfigure the defense strategy of IDFN across the three-tier SEC. The framework provides single-tier, two-tier, and three-tier dynamic awareness models based on the susceptible-infected-susceptible (SIS) dynamical equations to characterize the update process of message states to obtain the equilibrium solution between alarm messages and normal messages captured by IDSs. For swarm attack events from multi-tier SEC, we model the cross-tier cooperative interactions between IDSs and swarm attackers as an event-condition-action (ECA) regret learning game (ERLG) to achieve a distributed IDS reconfiguration to reduce the overall SEC alarm messages while ensuring the equilibrium of message states with the cooperation of IDSs. Simulation results demonstrate that our proposed scheme is superior to other reconfiguration mechanisms under swarm attacks in three-tier SEC.

Sparse Bandit Learning Based Location Management for Space-Ground Integrated Networks

The Space-Ground Integrated Network (SGIN) concept constitutes a promising solution for providing seamless global coverage. However, the mobility of satellites and wireless terminals imposes unprecedented challenges on the location management in SGIN. We tackle this challenge by conceiving a split identifier (ID) and Network Address (NA) based design for providing natural mobility support, and characterize the ID-NA mapping allocation problem by exploiting the storage capacity of both Geostationary Earth Orbit Satellites (GEOSs) and Low Earth Orbiting Satellites (LEOSs) to form a spatially distributed binding resolution system and optimize the caching reward in each LEOS. By considering the large quantity of ID-NA mapping and the sparsity of popular mapping having positive mean caching rewards, we formulate the mapping allocation problem as a sparse Multi-Armed Bandit (MAB) learning procedure, where the mappings are treated as the arms and the LEOSs act as the players. A distributed learning algorithm, namely the Sparse Upper confidence bound based Learning aided Caching algorithm (SULC), is proposed for estimating the mean caching rewards of mappings and selecting the optimal mappings for caching. Moreover, we derive a sub-linear upper bound of the cumulative learning regret to prove the learning efficiency of the proposed SULC. Extensive simulations have been conducted to show that the proposed SULC can quickly identify the popular mappings and provide near-optimal content hit rates. In contrast with the existing solutions, SULC has higher caching rewards and can significantly reduce the cumulative regret after a short period of learning.

A Learning-Based Approach for Vehicle-to-Vehicle Computation Offloading

Vehicle-to-vehicle (V2V) computation offloading has emerged as a promising solution to facilitate computing-intensive vehicular task processing, where task vehicles (i.e., TaVs) will be requested to offload computing-intensive tasks to server vehicles (i.e., SeVs) in order to keep task delay low. However, it is challenging for TaVs to obtain the optimal V2V computation offloading decisions (i.e., realizing the minimal task delay) due to the constraints, including: 1) incomplete offloading information; 2) degraded Quality-of-Service (QoS) of SeVs; and 3) privacy leakage risks. In this article, we develop a learning-based V2V computation offloading algorithm enhanced by SeV’s ability & trustfulness awareness to solve these problems. We emphasize that the proposed algorithm learns the offloading performance of candidate SeVs based on history offloading selections, without requiring the complete offloading information in advance. Additionally, both the QoS of SeVs and safe V2V computation offloading are enhanced in the proposed learning-based algorithm. Furthermore, we conduct extensive simulation experiments to validate the proposed algorithm. The results demonstrate that the proposed algorithm reduces the average task delay by 35% and 40%, and at the same time decreases the learning regret by 39% and 41%, compared to the algorithms without SeV’s ability and trustfulness awareness.

MASS: Mobility-Aware Sensor Scheduling of Cooperative Perception for Connected Automated Driving

Timely and reliable environment perception is fundamental to safe and efficient automated driving. However, the perception of standalone intelligence inevitably suffers from occlusions. A new paradigm, Cooperative Perception (CP), comes to the rescue by sharing sensor data from another perspective, i.e., from a cooperative vehicle (CoV). Due to the limited communication bandwidth, it is essential to schedule the most beneficial CoV, considering both the viewpoints and communication quality. Existing methods rely on the exchange of meta-information, such as visibility maps, to predict the perception gains from nearby vehicles, which induces extra communication and processing overhead. In this paper, we propose a new approach, learning while scheduling, for distributed scheduling of CP. The solution enables CoVs to predict the perception gains using past observations, leveraging the temporal continuity of perception gains. Specifically, we design a mobility-aware sensor scheduling (MASS) algorithm based on the restless multi-armed bandit (RMAB) theory to maximize the expected average perception gain. An upper bound on the expected average learning regret is proved, which matches the lower bound of any online algorithm up to a logarithmic factor. Extensive simulations are carried out on realistic traffic traces. The results show that the proposed MASS algorithm achieves the best average perception gain and improves recall by up to 3.1 percentage points compared to other learning-based algorithms. Finally, a case study on a trace of LiDAR frames qualitatively demonstrates the superiority of adaptive exploration, the key element of the MASS algorithm.

Learning Based Energy Efficient Task Offloading for Vehicular Collaborative Edge Computing

Extensive delay-sensitive and computation-intensive tasks are involved in emerging vehicular applications. These tasks can hardly be all processed by the resource constrained vehicle alone, nor fully offloaded to edge facilities (like road side units) due to their incomplete coverage. To this end, we refer to the new paradigm of vehicular collaborative edge computing (VCEC) and make the best use of vehicles’ idle and redundant resources for energy consumption reduction within the VCEC system. To realize this target, we are faced with several nontrivial challenges, including short-term decision making coupled with long-term queue delay constraints, information uncertainty, and task offloading conflicts. Accordingly, we apply Lyapunov optimization to decouple the original problem into three sub-problems and then tackle them one by one: the first sub-problem is resolved by Lagrange multiplier method; the second is handled by UCB learning-matching approach; the third is addressed by a carefully designed greedy method. Scenarios without volatility and real-world road topology with realistic vehicular traffics are utilized to evaluate the proposed solution. Results from extensive numerical simulations demonstrate that our solution can achieve superior performances compared with the benchmark methods, in terms of energy consumption, learning regret, task backlog, and end-to-end delay.

Adaptive-Learning-Based Vehicle-to-Vehicle Opportunistic Resource-Sharing Framework

With an ever-increasing number of connected devices on roads, it becomes unsustainable to provide nearby specialized execution resources (compute and storage) for servicing innovative applications. Moreover, the vehicular environment being inherently ad hoc and opportunistic, not to mention highly mobile, makes it unsuitable to use traditional cloud computing due to delayed and interrupted services. Thus, there is a possibility to introduce potential collaboration among nearby connected vehicles. However, the underlying decision model for the selection of the most suitable vehicle for task offloading is challenging in such a dynamic environment. In this study, we propose a collaborative vehicular computing framework that adopts online learning for efficient task assignment between local and neighboring computing resources. The underlying workload adaptive task offloading intends to balance out the workload across neighboring vehicles. The framework is compared against three techniques including two adaptive learning techniques in terms of service delay, efficiency, task delivery rate, task failures, and learning regret. The results demonstrate the effectiveness of the proposed resource-sharing network, improving service quality and throughput for servicing innovative intelligent transportation applications.

Privacy-Preserving Communication-Efficient Federated Multi-Armed Bandits

Communication bottleneck and data privacy are two critical concerns in federated multi-armed bandit (MAB) problems, such as situations in decision-making and recommendations of connected vehicles via wireless. In this paper, we design the privacy-preserving communication-efficient algorithm in such problems and study the interactions among privacy, communication and learning performance in terms of the regret. To be specific, we design privacy-preserving learning algorithms and communication protocols and derive the learning regret when networked private agents are performing online bandit learning in a master-worker, a decentralized and a hybrid structure. Our bandit learning algorithms are based on epoch-wise sub-optimal arm eliminations at each agent and agents exchange learning knowledge with the server/each other at the end of each epoch. Furthermore, we adopt the differential privacy (DP) approach to protect the data privacy at each agent when exchanging information; and we curtail communication costs by making less frequent communications with fewer agents participation. By analyzing the regret of our proposed algorithmic framework in the master-worker, decentralized and hybrid structures, we theoretically show trade-offs between regret and communication costs/privacy. Finally, we empirically show these trade-offs which are consistent with our theoretical analysis.

Contextual User-Centric Task Offloading for Mobile Edge Computing in Ultra-Dense Network

Integrating mobile edge computing (MEC) in the ultra-dense network (UDN) is a key enabler to meet the service demand by allowing smart devices to perform uninterrupted task offloading via densely deployed MEC servers. In most cases, the smart devices randomly move around the whole network. Consequently, the popular ‘`MEC-centralized decision’' offloading approach could be inapplicable, as joint decision-making among multiple MEC servers becomes difficult due to time synchronization and information exchange overhead. In this paper, we take a user-centric approach to minimize a long-term delay for a given task duration under a price budget constraint. To address this problem, we develop a novel contextual sleeping bandit learning (CSBL) algorithm, which integrates contextual information and sleeping characteristic to accelerate the learning convergence and leverage Lyapunov optimization to deal with the price budget constraint. Furthermore, we extend to a multiple offloading scenario where multiple MEC servers can be selected in each offloading round and propose a CSBL-multiple (CSBL-M) algorithm to address the exponential increase of the offloading selections. For both CSBL and CSBL-M, we derive the upper bounds of learning regret and provide rigorous proofs that they asymptotically approach the Oracle algorithm within bounded deviations for finite task duration.

Online-Learning-Based Defense Against Jamming Attacks in Multichannel Wireless CPS

We study security of remote state estimation in wireless cyber-physical systems (CPS) where a sensor sends its measurements to the remote state estimator over a multichannel wireless link in presence of a jamming attacker. Most of the existing works study the sensor's defense scheme by adopting optimization-based methods and rely on the prior knowledge of the attacker's attack policy. To relax this constraint, we propose a novel online-learning-based policy called joint channel and power selection (J-CAP) for the sensor to dynamically choose transmission channel and power. The proposed method assumes no prior knowledge of the attacker's attack policy, nor of the channel state information. J-CAP jointly optimizes sensor's channel selection and power consumption, and guarantees the estimator's asymptotic stability. We theoretically prove that J-CAP achieves a sublinear learning regret bound. We also show J-CAP's optimality by deriving and matching its regret lower and upper bound orders. Compared with the solution that directly applies the baseline solution, J-CAP improves the regret upper bound by a factor of √{K+L}, where K and L denote the number of channels and number of power levels, respectively. Numerical evaluations validate the analytical results under various CPS parameters, and compare the J-CAP's performance with the state-of-the-art solutions.

Reinforcement Learning-Based Routing Protocol to Minimize Channel Switching and Interference for Cognitive Radio Networks

In the existing network-layered architectural stack of Cognitive Radio Ad Hoc Network (CRAHN), channel selection is performed at the Medium Access Control (MAC) layer. However, routing is done on the network layer. Due to this limitation, the Secondary/Unlicensed Users (SUs) need to access the channel information from the MAC layer whenever the channel switching event occurred during the data transmission. This issue delayed the channel selection process during the immediate routing decision for the channel switching event to continue the transmission. In this paper, a protocol is proposed to implement the channel selection decisions at the network layer during the routing process. The decision is based on past and expected future routing decisions of Primary Users (PUs). A learning agent operating in the cross-layer mode of the network-layered architectural stack is implemented in the spectrum mobility manager to pass the channel information to the network layer. This information is originated at the MAC layer. The channel selection is performed on the basis of reinforcement learning algorithms such as No-External Regret Learning, Q-Learning, and Learning Automata. This leads to minimizing the channel switching events and user interferences in the Reinforcement Learning- (RL-) based routing protocol. Simulations are conducted using Cognitive Radio Cognitive Network simulator based on Network Simulator (NS-2). The simulation results showed that the proposed routing protocol performed better than all the other comparative routing protocols in terms of number of channel switching events, average data rate, packet collision, packet loss, and end-to-end delay. The proposed routing protocol implies the improved Quality of Service (QoS) of the delay sensitive and real-time networks such as Cellular and Tele Vision (TV) networks.

Cognitive-Caching: Cognitive Wireless Mobile Caching by Learning Fine-Grained Caching-Aware Indicators

Caching content on mobile devices reduces not only the transmission of backhaul links but also the latency of content acquisition. However, in the existing caching schemes, the joint effect of the caching-aware indicator of users and content dimensions on the caching strategy is not considered. Thus, how to learn fine-grained caching- aware indicators and design caching schemes are still challenging issues. In order to solve these problems, in this article, we first propose the cognitive caching architecture and give the fine-grained caching-aware indicators metric. Then we design a cognitive caching scheme that includes what to do in cognitive caching and how to do cognitive caching. Finally, experimental results show that the proposed cognitive caching scheme is superior to other caching schemes in terms of learning regret and caching cost.

Modeling Settlement Bargaining with Algorithmic Game Theory

Past computational models of settlement bargaining have lacked explicit game theoretic foundations. Algorithmic game theory, however, offers techniques that can nd perfect Bayesian equilibria even where closed-form mathematical solutions may be intractable. Some recent mathematical models tackle two-sided asymmetric information, including evidentiary signals models, in which the judgment is a sum of both shared and independent private information, and correlated signals models, in which both parties receive noisy signals about the same information. To relax assumptions inherent in these models, this paper employs several progressively more complicated techniques, including iterative elimination of dominated alternatives, no regret learning, and counterfactual regret minimization. Although these algorithms are not guaranteed to produce Nash equilibria in general-sum games like litigation, they nonetheless succeed in producing either exact or close approximate equilibria on discrete versions of the corresponding mathematical models. A single algorithmic game theory model can incorporate a number of features that state-of-the-art mathematical models cannot handle simultaneously, such as two-sided correlated signals of both liability and damages, risk aversion, and options to concede.

Online Learning Enabled Task Offloading for Vehicular Edge Computing

Vehicular edge computing pushes the cloud computing capability to the distributed network edge nodes, enabling computation-intensive and latency-sensitive computing services for smart vehicles through task offloading. However, the inherent mobility introduces fast variation of network structure, which are usually unknown a priori . In this letter, we formulate the vehicular task offloading as a mortal multi-armed bandit problem, and develop a new online algorithm to enable distributed decision making on the node selection. The key is to exploit the contextual information of edge nodes and transform the infinite exploration space to a finite one. Theoretically, we prove that the proposed algorithm has a sublinear learning regret. Simulation results verify its effectiveness.

Adaptive Learning-Based Task Offloading for Vehicular Edge Computing Systems

The vehicular edge computing (VEC) system integrates the computing resources of vehicles, and provides computing services for other vehicles and pedestrians with task offloading. However, the vehicular task offloading environment is dynamic and uncertain, with fast varying network topologies, wireless channel states and computing workloads. These uncertainties bring extra challenges to task offloading. In this work, we consider the task offloading among vehicles, and propose a solution that enables vehicles to learn the offloading delay performance of their neighboring vehicles while offloading computation tasks. We design an adaptive learning-based task offloading (ALTO) algorithm based on the multi-armed bandit (MAB) theory, in order to minimize the average offloading delay. ALTO works in a distributed manner without requiring frequent state exchange, and is augmented with input-awareness and occurrence-awareness to adapt to the dynamic environment. The proposed algorithm is proved to have a sublinear learning regret. Extensive simulations are carried out under both synthetic scenario and realistic highway scenario, and results illustrate that the proposed algorithm achieves low delay performance, and decreases the average delay up to 30% compared with the existing upper confidence bound based learning algorithm.

Online Learning Schemes for Power Allocation in Energy Harvesting Communications

We consider the problem of power allocation over one or more time-varying channels with unknown distributions in energy harvesting communications. In the single-channel case, the transmitter chooses the transmit power based on the amount of stored energy in its battery with the goal of maximizing the average rate over time. We model this problem as a Markov decision process (MDP) with transmitter as the agent, battery status as the state, transmits power as the action and rate as the reward. The average reward maximization problem can be modeled by a linear program (LP) that uses the transition probabilities for the state-action pairs and their reward values to select a power allocation policy. This problem is challenging because the uncertainty in channels implies that the mean rewards associated with the state-action pairs are unknown. We therefore propose two online learning algorithms: linear program of sample means (LPSM) and Epoch-LPSM that learn these rewards and adapt their policies over time. For both algorithms, we prove that their regret is upper-bounded by a constant. To our knowledge this is the first result showing constant regret learning algorithms for MDPs with unknown mean rewards. We also prove an even stronger result about LPSM: that its policy matches the optimal policy exactly in finite expected time. Epoch-LPSM incurs a higher regret compared with the LPSM, while reducing the computational requirements substantially. We further consider a multi-channel scenario, where the agent also chooses a channel in each slot, and present our multi-channel LPSM (MC-LPSM) algorithm that explores different channels and uses that information to solve the LP during exploitation. MC-LPSM incurs a regret that scales logarithmically in time and linearly in the number of channels. Through a matching lower bound on the regret of any algorithm, we also prove the asymptotic order optimality of MC-LPSM.

Learning-Based Caching in Cloud-Aided Wireless Networks

This letter studies content caching in cloud-aided wireless networks, where small cell base stations with limited storage are connected to the cloud via limited capacity fronthaul links. By formulating a utility (inverse of service delay) maximization problem, we propose a cache update algorithm based on spatio-temporal traffic demands. To account for the large number of contents, we propose a content clustering algorithm to group similar contents. Subsequently, with the aid of regret learning at small cell base stations and the cloud, each base station caches contents based on the learned content popularity subject to its storage constraints. The performance of the proposed caching algorithm is evaluated for sparse and dense environments, while investigating the tradeoff between global and local class popularity. Simulation results show 15% and 40% gains in the proposed method compared to various baselines.

Sex with no regrets: How sexual reproduction uses a no regret learning algorithm for evolutionary advantage

The question of ‘why sex’ has long been a puzzle. The randomness of recombination, which potentially produces low fitness progeny, contradicts notions of fitness landscape hill climbing. We use the concept of evolution as an algorithm for learning unpredictable environments to provide a possible answer. While sex and asex both implement similar machine learning no-regret algorithms in the context of random samples that are small relative to a vast genotype space, the algorithm of sex constitutes a more efficient goal-directed walk through this space. Simulations indicate this gives sex an evolutionary advantage, even in stable, unchanging environments. Asexual populations rapidly reach a fitness plateau, but the learning aspect of the no-regret algorithm most often eventually boosts the fitness of sexual populations past the maximal viability of corresponding asexual populations. In this light, the randomness of sexual recombination is not a hindrance but a crucial component of the ‘sampling for learning’ algorithm of sexual reproduction.

Learning Based Content Caching and Sharing for Wireless Networks

Content caching at base stations (BSs) is a promising technique for future wireless networks by reducing network traffic and alleviating server bottleneck. However, in practice, the content popularity distribution may change with spatio-temporal variation but be unknown for BSs, which is an intractable obstacle for efficient caching strategy design. In this paper, considering unknown popularity distribution, we explore the content caching problem by jointly optimizing the content caching in cooperative BSs, content sharing among BSs, and cost of content retrieving. We tackle the problem from a multi-armed bandit learning perspective, where the learning of the popularity distribution is incorporated with the content caching and sharing process. Specifically, we first propose a centralized algorithm by employing a semidefinite relaxation approach, and we prove that this centralized algorithm learns efficient caching by deriving a sub-linear learning regret bound. To further reduce computational complexity, we propose a distributed algorithm based on alternating direction method of multipliers, where each BS only solves their own problems by exchanging local information with neighbor BSs. Extensive simulation results show the effectiveness of the proposed algorithms in terms of learning content popularity distributions of individual BSs, offloading traffic from the content server, and reducing cost of content retrieving.

Trend-Aware Video Caching Through Online Learning

This paper presents Trend-Caching, a novel cache replacement method that optimizes cache performance according to the trends of video content. Trend-Caching explicitly learns the popularity trend of video content and uses it to determine which video it should store and which it should evict from the cache. Popularity is learned in an online fashion and requires no training phase, hence it is more responsive to continuously changing trends of videos. We prove that the learning regret of Trend-Caching (i.e., the gap between the hit rate achieved by Trend-Caching and that by the optimal caching policy with hindsight) is sublinear in the number of video requests, thereby guaranteeing both fast convergence and asymptotically optimal cache hit rate. We further validate the effectiveness of Trend-Caching by applying it to a movie.douban.com dataset that contains over 38 million requests. Our results show significant cache hit rate lift compared to existing algorithms, and the improvements can exceed 40% when the cache capacity is limited. Furthermore, Trend-Caching has low complexity.

Scheduling in Wireless Networks with Rayleigh-Fading Interference

We study approximation algorithms for optimization of wireless spectrum access with n communication requests when interference conditions are given by the Rayleigh-fading model. This model extends the deterministic interference model based on the signal-to-interference-plus-noise ratio (SINR) using stochastic propagation to address fading effects observed in reality. We consider worst-case approximation guarantees for the two standard problems of capacity maximization and latency minimization. Our main result is a generic reduction of Rayleigh fading to the deterministic non-fading model. It allows to apply existing algorithms for the non-fading model in the Rayleigh-fading scenario while losing only a factor of O(log* n) in the approximation guarantee. This way, we obtain the first approximation guarantees for Rayleigh fading and, more fundamentally, show that non-trivial stochastic fading effects can be successfully handled using existing and future techniques for the non-fading model. We generalize these results in two ways. First, the same results apply for capacity maximization with variable data rates, when links obtain (non-binary) utility depending on the achieved SINR. Second, for binary utilities, we use a more detailed argument to obtain similar results even for distributed and game-theoretic approaches. Our analytical treatment is supported by simulations illustrating the performance of regret learning and, more generally, the relationship between both models.