Cooperative Learning Algorithm Research Articles

Cooperative Online Learning for Multiagent System Control via Gaussian Processes With Event-Triggered Mechanism.

In the realm of the cooperative control of multiagent systems (MASs) with unknown dynamics, Gaussian process (GP) regression is widely used to infer the uncertainties due to its modeling flexibility of nonlinear functions and the existence of a theoretical prediction error bound. Online learning, which involves incorporating newly acquired training data into GP models, promises to improve control performance by enhancing predictions during the operation. Therefore, this article investigates the online cooperative learning algorithm for MAS control. Moreover, an event-triggered data selection mechanism, inspired by the analysis of a centralized event-trigger (CET), is introduced to reduce the model update frequency and enhance the data efficiency. With the proposed learning-based control, the practical convergence of the MAS is validated with guaranteed tracking performance via the Lyapunov theory. Furthermore, the exclusion of the Zeno behavior for individual agents is shown. Finally, the effectiveness of the proposed event-triggered online learning method is demonstrated in simulations.

Joint Resource Allocation and Drones Relay Selection for Large-Scale D2D Communication Underlaying Hybrid VLC/RF IoT Systems

Relay-aided Device-to-Device (D2D) communication combining visible light communication (VLC) with radio frequency (RF) is a promising paradigm in the internet of things (IoT). Static relay limits the flexibility and maintaining connectivity of relays in Hybrid VLC/RF IoT systems. By using a drone as a relay station, it is possible to avoid obstacles such as buildings and to communicate in a line-of-sight (LoS) environment, which naturally aligns with the requirement of VLC Systems. To further support the application of VLC in the IoT, subject to the challenges imposed by the constrained coverage, the lack of flexibility, poor reliability, and connectivity, drone relay-aided D2D communication appears on the horizon and can be cost-effectively deployed for the large-scale IoT. This paper proposes a joint resource allocation and drones relay selection scheme, aiming to maximize the D2D system sum rate while ensuring the quality of service (QoS) requirements for cellular users (CUs) and D2D users (DUs). First, we construct a two-phase coalitional game to tackle the resource allocation problem, which exploits the combination of VLC and RF, as well as incorporates a greedy strategy. After that, a distributed cooperative multi-agent reinforcement learning (MARL) algorithm, called WoLF policy hill-climbing (WoLF-PHC), is proposed to address the drones relay selection problem. Moreover, to further reduce the computational complexity, we propose a lightweight neighbor–agent-based WoLF-PHC algorithm, which only utilizes historical information of neighboring DUs. Finally, we provide an in-depth theoretical analysis of the proposed schemes in terms of complexity and signaling overhead. Simulation results illustrate that the proposed schemes can effectively improve the system performance in terms of the sum rate and outage probability with respect to other outstanding algorithms.

On Jointly Optimizing Partial Offloading and SFC Mapping: A Cooperative Dual-Agent Deep Reinforcement Learning Approach

Multi-access edge computing (MEC) and network function virtualization (NFV) are promising technologies to support emerging IoT applications, especially those computation-intensive. In NFV-enabled MEC environment, service function chain (SFC), i.e., a set of ordered virtual network functions (VNFs), can be mapped on MEC servers. Mobile devices (MDs) can offload computation-intensive applications, which can be represented by SFCs, fully or partially to MEC servers for remote execution. This paper studies the partial offloading and SFC mapping joint optimization (POSMJO) problem in an NFV-enabled MEC system, where the data from an incoming task is partitioned into two parts, with one part executed locally and the other offloaded to the edge infrastructure for execution. These two parts are independent of each other, but both need to be processed by the same SFC. The objective is to minimize the average cost in the long term which is a combination of execution delay, MD's energy consumption, and usage charge for edge computing. This problem consists of two closely related decision-making steps, namely task partition and VNF placement, which is highly complex and quite challenging. To address this, we propose a cooperative dual-agent deep reinforcement learning (CDADRL) algorithm, where two agents interact with each other. Simulation results show that the proposed algorithm outperforms three combinations of deep reinforcement learning algorithms with respect to cumulative reward and it overweighs a number of baseline algorithms in terms of execution delay, energy consumption, and usage charge.

Task Computation Offloading for Multi-Access Edge Computing via Attention Communication Deep Reinforcement Learning

This paper investigates how to enhance the Multi-access Edge Computing (MEC) systems performance with the aid of device-to-device (D2D) communication computation offloading. By adequately exploiting a novel computation offloading mechanism based on D2D collaboration, users can efficiently share computational resources with each other. However, it is challenging to distinguish valuable information that truly promotes a collaborative decision, as worthless information can hinder collaboration among users. In addition, the transmission of large volumes of information requires high bandwidth and incurs significant latency and computational complexity, resulting in unacceptable costs. In this paper, we propose an efficient D2D-assisted MEC computation offloading framework based on Attention Communication Deep Reinforcement Learning (ACDRL), which simulates the interactions between related entities, including device-to-device collaboration in the horizontal and device-to-edge offloading in the vertical. Secondly, we developed a distributed cooperative reinforcement learning algorithm that includes an attention mechanism that skews computational resources towards active users to avoid unnecessary resource wastage in large-scale MEC systems. Finally, to improve the effectiveness and rationality of cooperation among users, we introduce a communication channel to integrate information from all users in a communication group, thus facilitating cooperative decision-making. The proposed framework is benchmarked, and the experimental results show that the proposed framework can effectively reduce latency and provide valuable insights for practical design compared to other baseline approaches.

UAV Assisted Cooperative Caching on Network Edge Using Multi-Agent Actor-Critic Reinforcement Learning

In recent times, caching at edge nodes is a well-known technique to overcome the limitation of strict latency, which simultaneously improves users' Quality of Experience (QoE). However, choosing an appropriate caching policy and content placement poses another significant issue that has been acknowledged in this research. Conventional caching policies that are experimented with at the edge do not consider the dynamic and stochastic characteristics of edge caching. As a result, we have proposed a cooperative deep reinforcement learning algorithm that deals with the dynamic nature of content demand. It also ensures efficient use of storage through the cooperation between nodes. In addition, previous works on cooperative caching have assumed the users to be static and didn't consider the mobile nature of users. Therefore, we have proposed UAVs as aerial Base Stations (UAV-BS) to assist in peak hours where a ground base station is insufficient to support the surge in user requests. In this novel research, we have demonstrated the cooperation between aerial and Ground Base Stations (GBS) and aimed at maximizing the global cache hit ratio. Simulations have shown that our proposed Cooperative Multi-Agent Actor-Critic algorithm outperforms conventional and reinforcement learning based caching methods and achieves a state-of-the-art global cache hit ratio when there is a surge in user requests. Thus, it opens the door for further research on cooperative caching in joint air and ground architecture.

Cooperative Learning of Multi-Agent Systems Via Reinforcement Learning

In many specific scenarios, accurateand practical cooperative learning is a commonly encountered challenge in multi-agent systems. Thus, the current investigation focuses on cooperative learning algorithms for multi-agent systems and underpins an alternate data-based neural network reinforcement learning framework. To achieve the data-based learning optimization, the proposed cooperative learning framework, which comprises two layers, introduces a virtual learning objective. The followers learn the behaviors of the virtual objects in the first layer based on the adaptive neural networks (NNs). Specifically, the actor and critic NNs are applied to acquire cooperative behaviors and assess this layer's long-term utility function. Then another layer realizes the tracking performance between the virtual objects and the leader by introducing the local data-based performance index. Then, we formulate a resulting deterministic optimization problem and resolve it effectively with the policy iteration algorithm. This intuitive cooperative learning algorithm also preserves good robustness properties and eliminates the dependence on the prior knowledge of the multi-agent system model in the solution process. Finally, a multi-robot formation system demonstrates this promising development's practical appeal and highly effective outcome.

Cooperative Multiagent Deep Reinforcement Learning for Computation Offloading: A Mobile Network Operator Perspective

Computation offloading decisions play a crucial role in implementing mobile-edge computing (MEC) technology in the Internet of Things (IoT) services. Mobile network operators (MNOs) can employ computation offloading techniques to reduce task completion delay and improve the Quality of Service (QoS) for users by optimizing the system’s processing delay and energy consumption. However, different IoT applications (e.g., entertainment and autonomous driving) generate different delay tolerances and benefits for computational tasks from the MNO perspective. Therefore, simply minimizing the delay of all tasks does not satisfy the QoS of each user. The system architecture design should consider the significance of users and the heterogeneity of tasks. Unfortunately, rare work has been done to discuss this practical issue. In this article, from the perspective of MNO, we investigate the computation offloading optimization problem of multiuser delay-sensitive tasks. First, we propose a new optimization model, which designs different optimization objectives for the cost and revenue of tasks. Then, we transform the problem into a Markov decision processes problem, which leads to designing a multiagent iterative optimization framework. For the strategic optimization of each agent, we further propose a cooperative multiagent deep reinforcement learning (CMDRL) algorithm to optimize two different objectives at the same time. Two agents are integrated into the CMDRL framework to enable agents to collaborate and converge to the global optimum in a distributed manner. At the same time, the priority experience replay method is introduced to improve the utilization rate of effective samples and the learning efficiency of the algorithm. The experimental results show that our proposed method can effectively achieve a significantly higher profit than the alternative state-of-the-art method and exhibit a more favorable computational performance than benchmark deep reinforcement learning methods.

Structured Cooperative Reinforcement Learning With Time-Varying Composite Action Space.

In recent years, reinforcement learning has achieved excellent results in low-dimensional static action spaces such as games and simple robotics. However, the action space is usually composite, composed of multiple sub-action with different functions, and time-varying for practical tasks. The existing sub-actions might be temporarily invalid due to the external environment, while unseen sub-actions can be added to the current system. To solve the robustness and transferability problems in time-varying composite action spaces, we propose a structured cooperative reinforcement learning algorithm based on the centralized critic and decentralized actor framework, called SCORE. We model the single-agent problem with composite action space as a fully cooperative partially observable stochastic game and further employ a graph attention network to capture the dependencies between heterogeneous sub-actions. To promote tighter cooperation between the decomposed heterogeneous agents, SCORE introduces a hierarchical variational autoencoder, which maps the heterogeneous sub-action space into a common latent action space. We also incorporate an implicit credit assignment structure into the SCORE to overcome the multi-agent credit assignment problem in the fully cooperative partially observable stochastic game. Performance experiments on the proof-of-concept task and precision agriculture task show that SCORE has significant advantages in robustness and transferability for time-varying composite action space.

MARL-Based Cooperative Multi-AGV Control in Warehouse Systems

Automated guided vehicles (AGVs) are essential components for the automation of fulfillment centers, a type of warehouse, where goods are stored on shelves and carried by AGVs. To increase the productivity in inventory management, a well-organized cooperative path control is required to transport goods to the designated picking stations. In this paper, we propose a QMIX-based scheme for the cooperative path control of multiple AGVs. Although QMIX is the one of popular cooperative multi-agent reinforcement learning algorithms, we find that QMIX alone was not enough to increase productivity in warehouse systems. So, we develop two novel techniques that can be used with QMIX: 1) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sequential action masking</i> that eliminates all the collision cases and 2) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">additional local loss</i> that improves collaboration of individual AGVs. They help to encourage the AGVs to cooperate more for high productivity. By extensive simulations, we present the superiority of the proposed scheme on several layouts in fulfillment centers. The effect of cooperation among AGVs in the proposed scheme is verified through the comparison study with the existing algorithms. Additionally, we show the generalization performance by investigating the reusability of the model trained with the proposed scheme.

Symmetrical lattice generative adversarial network for remote sensing images compression

Image compression usually includes two important operations: compression and decompression. The compression process includes the operation of discarding information, while the decompression process is to retrieve the lost information. In order to make the decompressed image more similar to the original image, the classic compression methods generally adopt the process of approximately reversible compression and decompression. Inspired by the symmetric structure in classic compression methods, we propose a new symmetrical lattice generating adversarial network (SLGAN) for the remote sensing images (RSIs) compression in this paper. Several pairs of symmetrical encoder-decoder lattices are constructed to build a generator to first generate deep representative codes of images and then decode them. For each pair of encoded lattice and decoded lattice, one discriminator is constructed to perform adversarial learning with the generator. When multiple discriminators are used for all the lattices, a cooperative learning algorithm is proposed to train jointly pairs of symmetric lattices in the generator. Moreover, to enhance edges, contours, and textures in the decomposed RSIs, an enhanced Laplacian of gaussian (ELoG) loss is designed as a regularizer to train the SLGAN. Experimental results on the panchromatic images from GF2 satellite show that SLGAN outperforms other existing state-of-the-art methods.

SRAGL-AWCL: A two-step multi-view clustering via sparse representation and adaptive weighted cooperative learning

Sparse representation and cooperative learning are two representative technologies in the field of multi-view spectral clustering. The former can effectively extract features of multiple views by the removal of redundant information contained in each view. The latter can incorporate the diversity of each view. However, traditional sparse representation and cooperative learning algorithms are inadequate in preserving the internal geometric features of data by manifold regularization. In fact, general approaches rarely consider the similarities between the internal graph structures of individual views. Moreover, to achieve the optimal global feature learning, we present a novel two-step multi-view spectral clustering strategy, which combines the proposed sparse representation by adaptive graph learning with adaptive weighted cooperative learning. In the first step, the proposed matrix factorization by manifold regularization can strengthen the sparse features clustering discrimination of samples of each view. Specifically, the synchronization optimization method by introducing adaptive graph learning can better retain its internal complete structure of each view. This ensures the structure correlation of views through the usage of the sparse matrix and the optimal graph similarity matrix. In the second step, the adaptive weighted cooperative learning is performed on each view to get a global optimized matrix. In order to ensure that the global matrix is associated with various view features, graph learning is also performed on the global matrix. Experiment results on several multi-view datasets and single-view datasets show that the proposed method significantly outperformed the state-of-the-art algorithms.

A Cooperative Online Learning-Based Load Balancing Scheme for Maximizing QoS Satisfaction in Dense HetNets

This paper proposes a cooperative multi-agent online reinforcement learning-based (COMORL) bias offset (BO) control scheme for cell range expansion (CRE) in dense heterogeneous networks (HetNets). The proposed COMORL scheme controls BOs for CRE to maximize the number of user equipments (UEs) that satisfy their quality of service (QoS) requirements, especially in terms of delay and data rates. For this purpose, we developed a QoS satisfaction indicator that measures a violation of delay requirements by considering both QoS requirements and signal-to-interference-plus-noise ratio (SINR). In addition, we formulated a Markov decision process (MDP) model that is solved with a cooperative multi-agent online reinforcement learning algorithm. The proposed COMORL scheme maximizes the global utility for load-coupled base stations. Our simulation results verify the proposed COMORL scheme’s effectiveness in terms of throughput, delay satisfaction ratio, and fairness. Specifically, we verify that the proposed COMORL scheme achieves a maximum of approximately 27% and 30% improvement of the delay satisfaction ratio, which is how many UEs satisfy their delay requirement among all of the UEs in a serving BS under medium and full traffic loads, respectively, in a dynamic scenario in comparison to the max-SINR scheme.

Access and Radio Resource Management for IAB Networks Using Deep Reinforcement Learning

Congestion in dense traffic networks is a prominent obstacle towards realizing the performance requirements of 5G new radio. Since traditional adaptive traffic signal control cannot resolve this type of congestion, realizing context in the network and adapting resource allocation based on real-time parameters is an attractive approach. This article proposes a radio resource management solution for congestion avoidance on the access side of an integrated access and backhaul (IAB) network using deep reinforcement learning (DRL). The objective of this article is to obtain an optimal policy under which the transmission throughput of all UEs is maximized under the dictates of environmental pressures such as traffic load and transmission power. Here, the resource management problem was converted into a constrained problem using Markov decision processes and dynamic power management, where a deep neural network was trained for optimal power allocation. By initializing a power control parameter, $\theta _{t}$ , with zero-mean normal distribution, the DRL algorithm adopts a learning policy that aims to achieve logical allocation of resources by placing more emphasis on congestion control and user satisfaction. The performance of the proposed DRL algorithm was evaluated using two learning schemes, i.e., individual learning and nearest neighbor cooperative learning, and this was compared with the performance of a baseline algorithm. The simulation results indicate that the proposed algorithms give better overall performance when compared to the baseline algorithm. From the simulation results, there is a subtle, but critically important insight that brings into focus the fundamental connection between learning rate and the two proposed algorithms. The nearest neighbor cooperative learning algorithm is suitable for IAB networks because its throughput has a good correlation with the congestion rate.

Classification and Recognition of Electronic Components Based on Improved Cooperative Semi-supervised Learning Algorithm

Background: As known that the semi-supervised algorithm is a classical algorithm in semi-supervised learning algorithm. Methods: In the paper, it proposed improved cooperative semi-supervised learning algorithm, and the algorithm process is presented in detailed, and it is adopted to predict unlabeled electronic components image. Results: In the experiments of classification and recognition of electronic components, it show that through the method the accuracy the proposed algorithm in electron device image recognition can be significantly improved, the improved algorithm can be used in the actual recognition process . Conclusion: With the continuous development of science and technology, machine vision and deep learning will play a more important role in people's life in the future. The subject research based on the identification of the number of components is bound to develop towards the direction of high precision and multi-dimension, which will greatly improve the production efficiency of electronic components industry.

Reinforcement Learning for Efficient and Fair Coexistence Between LTE-LAA and Wi-Fi

Long-Term Evolution (LTE) over unlicensed spectrum extends LTE technology to the spacious unlicensed spectrum with readily available bandwidth. The provided capacity surge makes it one of the most high-profile technologies to meet the explosive growth of mobile traffic demand. Among its different variants, Licensed Assisted Access (LAA) is considered as a promising global solution attributed to its mandatory listen-before-talk (LBT) procedure. Nevertheless, although LBT effectively maintains transmission fairness between LTE and other unlicensed systems (e.g., Wi-Fi), the current LAA protocol specified in 3GPP Release 13 is far from perfect to achieve harmony coexistence. To this end, in this paper, we first develop an analytical model to evaluate the throughput performance of Category 4 (Cat 4) algorithm agreed in 3GPP release 13. Subject to the system fairness constraint, the aggregate throughput of LTE-LAA and Wi-Fi networks is maximized based on a semi branch and bound algorithm. To make the complex optimization tractable, reinforcement learning techniques are introduced to intelligently tune the contention window size for both LTE-LAA and Wi-Fi nodes. Specifically, a cooperative learning algorithm is developed assuming that the information between different systems is exchangeable. A non-cooperative version is subsequently developed to remove the previous assumption for better practicability. Extensive simulations are conducted to demonstrate the performance of the proposed learning algorithms in contrast to the analytical upper bound under various conditions. It is shown that both proposed learning algorithms can significantly improve the total throughput performance while satisfying the fairness constraints. Particularly, the proposed cooperative learning algorithm can closely approach the analytical bound.

Cooperative Multitask Learning for Sparsity-Driven SAR Imagery and Nonsystematic Error Autocalibration

Conventional sparsity-driven synthetic aperture radar (SAR) imagery often encounters the sensitivity of nonsystematic errors and highly computational load. In this article, a cooperative multitask learning algorithm is proposed based on an autocalibrated alternating direction method of multipliers (AutoCal-ADMM) framework, by which the sparse feature of the scenes/targets-of-interests can be enhanced, and simultaneously the nonmodeled motion errors of either airborne platform or moving target can be autocalibrated in a synergistic manner. By leveraging the entropy and sparsity regularizers in the AutoCal-ADMM framework, the proposed algorithm is particularly tailored to obtain focused SAR images with enhanced sparsity. A reasonable surrogate function is designed for a convex objective function, so that an analytical proximal mapping of the entropy regularizer can be derived. Both nonsystematic range cell migration (NsRCM) and azimuthal phase errors (APEs) are concerned and coherently compensated. A linear and complex soft-thresholding operator is introduced for the sparse solution. The proposed algorithm is capable of greatly alleviating “error propagation” between multiple tasks, where an optima balance between the sparse and focusing features can be achieved. Superior performance in terms of convergence and efficiency can be guaranteed. Both raw SAR and canonical ground moving target imaging (GMTIm) data sets are processed and comparisons with conventions are performed, where the effectiveness and superiority of the proposed AutoCal-ADMM algorithm are validated.

Modeling Communication Processes in the Human Connectome through Cooperative Learning

Communication processes within the human brain at different cognitive states are neither well understood nor completely characterized. We assess communication processes in the human connectome using ant colony-inspired cooperative learning algorithm, starting from a source with no a priori information about the network topology, and cooperatively searching for the target through a pheromone-inspired model. This framework relies on two parameters, namely pheromone perception and edge perception , to define the cognizance and subsequent behaviour of the ants on the network and, overall, the communication processes happening between source and target nodes. Simulations obtained through different configurations allow the identification of path-ensembles that are involved in the communication between node pairs. These path-ensembles may contain different number of paths depending on the perception parameters and the node pair. In order to assess the different communication regimes displayed on the simulations and their associations with functional connectivity, we introduce two network measurements, effective path-length and arrival rate. These communication features are tested as individual as well as combined descriptors of functional connectivity during different tasks. Finally, different communication regimes are found in different specialized functional networks. Overall, this framework may be used as a test-bed for different communication regimes on top of an underlaying topology.

Distributed cooperative learning over time-varying random networks using a gossip-based communication protocol

Motivated by applications of distributed estimation and distributed decision making in wireless sensor networks (WSNs) and unmanned aerial vehicle (UAV) networks, we study a distributed learning problem over time-varying undirected random networks. Using a gossip-based communication protocol, a novel distributed cooperative learning (DCL) algorithm, termed the gossip-based DCL (GBDCL) algorithm, is presented to solve the problem by training the raw data distributed and blocked throughout different nodes. Exploiting the robustness of the gossip-based protocol, each node is guaranteed to build the same learning model in theory against random disconnections and communication route variations in the network topology. It is proved that the GBDCL algorithm converges to the optimal consensus asymptotically. The correctness and effectiveness of the presented GBDCL algorithm are verified in the theoretical analysis and simulations.

Event-Triggered Distributed Cooperative Learning Algorithms over Networks via Wavelet Approximation

This paper investigates the problem of event-triggered distributed cooperative learning (DCL) over networks based on wavelet approximation theory, where each node only has access to local data which are produced by the same and unknown pattern (map or function). All nodes cooperatively learn this unknown pattern by exchanging learned information with their neighboring nodes under event-triggered strategy in order to remove unnecessary communications, so as to avoid the waste of network resources. For the above problem, two novel event-triggered continuous-time and discrete-time DCL algorithms are proposed to approximate the unknown pattern by using wavelet basis function. The proposed event-triggered DCL algorithms are used to train the optimal weight coefficient matrix of wavelet series. Moreover, the convergence of the proposed algorithms are presented by using the Lyapunov method, and the Zeno behavior is excluded as well by the strictly positive sampling interval. The illustrative examples are presented to show the efficiency and convergence of the proposed algorithms.

A Generic Human-Machine Annotation Framework Based on Dynamic Cooperative Learning.

The task of obtaining meaningful annotations is a tedious work, incurring considerable costs and time consumption. Dynamic active learning and cooperative learning are recently proposed approaches to reduce human effort of annotating data with subjective phenomena. In this paper, we introduce a novel generic annotation framework, with the aim to achieve the optimal tradeoff between label reliability and cost reduction by making efficient use of human and machine work force. To this end, we use dropout to assess model uncertainty and thereby to decide which instances can be automatically labeled by the machine and which ones require human inspection. In addition, we propose an early stopping criterion based on inter-rater agreement in order to focus human resources on those ambiguous instances that are difficult to label. In contrast to the existing algorithms, the new confidence measures are not only applicable to binary classification tasks but also regression problems. The proposed method is evaluated on the benchmark datasets for non-native English prosody estimation, provided in the Interspeech computational paralinguistics challenge. In the result, the novel dynamic cooperative learning algorithm yields 0.424 Spearman's correlation coefficient compared to 0.413 with passive learning, while reducing the amount of human annotations by 74%.