Distributed Online Optimization Research Articles

Distributed online optimization for integrated energy systems: A multi‐agent system consensus approach

SummaryThe integration of multi‐energy within distribution networks has escalated the need for efficient operation and control of integrated energy systems (IES). Addressing the complexities of real‐time scheduling and low‐carbon optimization, we propose a novel artificial intelligence driven multi‐agent system (MAS) approach for modeling the interactions and operations within the multi‐agent integrated energy systems (MA‐IES) framework. In this framework, distinct components such as electric, gas, and heat networks are conceptualized as autonomous agents, each responsible for managing its domain while interacting with other agents to achieve system‐wide efficiency and economical goals. The agents communicate and coordinate through a distributed online optimization framework, utilizing the alternating direction multiplier method (ADMM) to ensure effective consensus despite the inherent nontransparency of information exchange. This MAS based approach allows for dynamic adaptation of strategies based on local data and global objectives, significantly enhancing the responsiveness and adaptability of MA‐IES. We further integrate an objective function reliant on a tiered carbon pricing mechanism to assess and minimize the environmental impact of operations. Enhanced by adaptive penalty coefficients within the ADMM, our MA‐IES framework demonstrates improved convergence rates and robustness in operational scenarios. Empirical validation through detailed case studies confirms the superior performance of our MAS‐based model, demonstrating its potential to realize an efficient and economical low‐carbon operation of MA‐IES.

Distributed Online Optimization Under Dynamic Adaptive Quantization

Distributed Online Optimization Under Dynamic Adaptive Quantization

Distributed Online Constrained Optimization With Feedback Delays.

We investigate multiagent distributed online constrained convex optimization problems with feedback delays, where agents make sequential decisions before being aware of the cost and constraint functions. The main purpose of the distributed online constrained convex optimization problem is to cooperatively minimize the sum of time-varying local cost functions subject to time-varying coupled inequality constraints. The feedback information of the distributed online optimization problem is revealed to agents with time delays, which is common in practice. Every node in the system can interact with neighbors through a time-varying sequence of directed communication topologies, which is uniformly strongly connected. The distributed online primal-dual bandit push-sum algorithm that generates primal and dual variables with delayed feedback is used for the presented problem. Expected regret and expected constraint violation are proposed for measuring the performance of the algorithm, and both of them are shown to be sublinear with respect to the total iteration span T in this article. In the end, the optimization problem for the power grid is simulated to justify the proposed theoretical results.

Differentially private distributed online optimization via push-sum one-point bandit dual averaging

This paper focuses on the distributed online optimization problem in multi-agent systems considering privacy preservation. Each agent exchanges local information with neighboring agents on the strongly connected time-varying directed graphs. Since the process of information transmission is prone to information leakage, a distributed push-sum dual averaging algorithm based on the differential privacy mechanism is proposed to protect the privacy of the data. In addition, to handle situations where the gradient information of the node cost function is unknown, the one-point gradient estimation is designed to calculate the true gradient information and guide the update of the decision variables. With the appropriate choice of the stepsizes and the exploration parameters, the algorithm can effectively protect the privacy of agents while achieving sublinear regret with the convergence rate O(T34). Furthermore, this paper also explores the effect of one-point estimation parameters on the regret in the online setting and investigates the relation between the convergence effect of individual regret and differential privacy levels. Finally, several federated learning experiments were conducted to verify the efficacy of the algorithm.

Distributed online primal-dual subgradient method on unbalanced directed networks

In this paper, we investigate a distributed online optimization method on multiagent communication networks. We consider a distributed prime-dual subgradient algorithm for an online convex optimization problem with time-varying coupled constraints. Each agent updates the estimations for the primal and dual optimizers by a consensus-based online algorithm. In the proposed algorithm, the gradient direction is scaled by estimating the left eigenvector of a weight matrix associated with the directed communication network. The scaling procedure enables agents in the network to estimate the time-varying optimal solutions by counterbalancing the unbalanced communication flows. The performance of the proposed algorithm is examined by a dynamic regret and a fit, which evaluate the cumulative error against the time-varying optimal cost function and the constraint function, respectively. We provide a sufficient condition under which both the dynamic regret and the fit are sublinear. A numerical example of an online economic dispatch problem confirms the validity of the proposed method.

Distributed online optimization for heterogeneous linear multi-agent systems with coupled constraints

This paper studies a class of distributed online convex optimization problems for heterogeneous linear multi-agent systems. Agents in a network, knowing only their own outputs, need to minimize the time-varying costs through neighboring interaction subject to time-varying coupled inequality constraints. Based on the saddle-point technique, we design a continuous-time distributed controller which is shown to achieve constant regret bound and sublinear fit bound, matching those of the standard centralized online method. We further extend the control law to the event-triggered communication mechanism and show that the constant regret bound and sublinear fit bound are still achieved while reducing the communication frequency. Additionally, we study the situation of communication noise, i.e., the agent’s measurement of the relative states of its neighbors is disturbed by a noise. It is shown that, if the noise is not excessive, the regret and fit bounds are unaffected, which indicates the controller’s noise-tolerance capability to some extent. Finally, a numerical simulation is provided to support the theoretical conclusions.

Privacy preserving distributed online projected residual feedback optimization over unbalanced directed graphs

The distributed online optimization (DOO) problem with privacy guarantees over an unbalanced directed graph is considered in this paper, where the cost function is explicitly unknown. To solve this problem, a distributed online one-point residual feedback optimization algorithm based on differential privacy is designed. For the objective function explicitly unknown, this algorithm employs the residual feedback to estimate the true gradient information. In addition, only the row stochastic adjacency matrix utilized releases the requirement of double stochastic weighting matrices, making the algorithm easier to implement on directed graphs. Theoretical results show that the algorithm not only protects the privacy information of the nodes but also achieves the same sublinear rate regret as the DOO algorithm based on two-point feedback. Moreover, our regret bound depends on weaker assumptions than the traditional DOO algorithm based on one-point feedback. Finally, the simulation results verify the effectiveness of the algorithm.

Distributed mirror descent algorithm over unbalanced digraphs based on gradient weighting technique

This paper studies the mirror descent algorithm for distributed optimization, where the underlying digraph is assumed to be weight-unbalanced. Within this framework, a novel distributed mirror descent algorithm based on gradient weighting technique is developed. Theoretically, different from the existing works, which prove that the function value corresponding to the estimates converge to the optimal value of the optimization problem, this paper proves that (1) the proposed algorithm can achieve exact convergence of the estimates to the solution of the optimization problem; and (2) the algorithm has a convergence rate O(1T) with a given time horizon T. Further, taking into account the fact that the cost functions in many significant optimization problems are dynamic, the distributed online optimization based on the proposed algorithm is studied. Especially, it is shown that the individual regret of the proposed algorithm is bounded by O(T). Finally, the theoretical results are verified through some simulation examples.

Privacy‐preserving distributed projected one‐point bandit online optimization over directed graphs

AbstractThe distributed online optimization (DOO) problem with privacy‐preserving properties over multiple agents is considered in this paper, where the network model is built by a strongly connected directed graph. To solve this problem, a stochastic bandit DOO algorithm based on differential privacy is proposed. This algorithm uses row‐ and column‐stochastic matrix as the weighting matrices, the requirement of the double random weighting matrix is released. To handle the unknown objective function, the one‐point bandit is used to estimate the true gradient information, and the estimated gradient information is used to update of decision variables. Different from the existing DOO algorithms that ignore privacy issues, this algorithm successfully protects the privacy of nodes through a differential privacy policy. Theoretical results show that the algorithm can not only achieve sublinear regret bounds but also protect the privacy of nodes. Finally, simulation results verify the effectiveness of the algorithm.

Time-Varying Optimization of Networked Systems With Human Preferences

This article considers a time-varying optimization problem associated with a network of systems, with each of the systems shared by (and affecting) a number of individuals. The objective is to minimize cost functions associated with the individuals' preferences, which are unknown, subject to time-varying constraints that capture physical or operational limits of the network. To this end, this article develops a distributed online optimization algorithm with concurrent learning of the cost functions. The cost functions are learned on-the-fly based on the users' feedback (provided at irregular intervals) by leveraging tools from the shape-constrained Gaussian process. The online algorithm is based on a primal–dual method and acts effectively in a closed-loop fashion where: First, users' feedback is utilized to estimate the cost, and second, measurements from the network are utilized in the algorithmic steps to bypass the need for sensing of (unknown) exogenous inputs of the network. The performance of the algorithm is analyzed in terms of dynamic network regret and constraint violation. Numerical examples are presented in the context of real-time optimization of distributed energy resources.

Distributed Online Optimization With Dynamic Coupling Constraints Under Time-Varying Communication Delays

Distributed Online Optimization With Dynamic Coupling Constraints Under Time-Varying Communication Delays

A Distributed Online Newton Step Algorithm for Multi-Agent Systems

Most of the current algorithms for solving distributed online optimization problems are based on the first-order method, which are simple in computation but slow in convergence. Newton’s algorithm with fast convergence speed needs to calculate the Hessian matrix and its inverse, leading to computationally complex. A distributed online optimization algorithm based on Newton’s step is proposed in this paper, which constructs a positive definite matrix by using the first-order information of the objective function to replace the inverse of the Hessian matrix in Newton’s method. The convergence of the algorithm is proved theoretically and the regret bound of the algorithm is obtained. Finally, numerical experiments are used to verify the feasibility and efficiency of the proposed algorithm. The experimental results show that the proposed algorithm has an efficient performance on practical problems, compared to several existing gradient descent algorithms.

Distributed online adaptive subgradient optimization with dynamic bound of learning rate over time‐varying networks

Adaptive online optimization algorithms, such as Adam, RMSprop, and AdaBound, have recently been tremendously popular as they have been widely applied to address the issues in the field of deep learning. Despite their prevalence and prosperity, however, it is rare to investigate the distributed versions of these adaptive online algorithms. To fill the gap, a distributed online adaptive subgradient learning algorithm over time-varying networks, called DAdaxBound, which exponentially accumulates long-term past gradient information and possesses dynamic bounds of learning rates under learning rate clipping is developed. Then, the dynamic regret bound of DAdaxBound on convex and potentially nonsmooth objective functions is theoretically analysed. Finally, numerical experiments are carried out to assess the effectiveness of DAdaxBound on different datasets. The experimental results demonstrate that DAdaxBound compares favourably to other competing distributed online optimization algorithms.

Distributed online bandit optimization under random quantization

This paper considers the problem of solving distributed online optimization over a network that consists of multiple interacting nodes. Each node in the network is endowed with a sequence of loss functions, each of which is revealed to the node after a decision has been committed. The goal of the network is to minimize the cumulative loss functions of nodes in a distributed fashion, while subject to two types of information constraints, namely, message quantization and bandit feedback. To this end, a quantized distributed online bandit optimization algorithm is proposed by adopting random quantization operation and one-point gradient estimator. We show the convergence of the algorithm by establishing an O(dT3/4) regret bound, where d is the dimension of states and T is the total number of rounds. Finally, an online distributed quadratic programming problem is investigated to validate the theoretical findings of the paper.

Optimal Bi-Level Bidding and Dispatching Strategy Between Active Distribution Network and Virtual Alliances Using Distributed Robust Multi-Agent Deep Reinforcement Learning

The deregulated active distribution network (ADN) would incorporate numerous autonomous stakeholders, including some emerging distributed virtual alliances (DVAs) like virtual microgrids and virtual power plants. Those DVAs would autonomously participate in energy market trading through bidding among themselves and dispatching conducted by the ADN. In this paper, the optimal bidding and dispatching model for DVAs and ADN in the day-ahead market is first developed as a stochastic dynamic programming model with the risk of misconduct considered, and then re-formulated as a Markov Decision Process to be solved by a new Distributed Robust Multi-Agent Deep Deterministic Policy Gradient algorithm based on the concept of robust Nash equilibrium (RNE). This algorithm is a fully distributed online optimization that would converge to RNE. It is an effective risk-averse method to obtain the optimal bidding strategies of DVAs and the optimal dispatching decisions of distribution system operator (DSO). Its high computational performance is demonstrated in the case studies, and the strategic decisions of DVAs and DSO are thoroughly analyzed.

MIMO architecture for fast convergence of distributed online optimization in smart grids

This paper proposes an architectural solution to enhance the resilience of distributed decision-making and control in microgrids and smart grids with respect to communication delays. The paper develops a resource allocation framework to optimally place the multiple-input multiple-output (MIMO) communication technology on critical channels to reduce the communication bottle neck and expedite the convergence speed of distributed online optimization algorithms, which require convergence in the cyber network before the solution can be implemented on the physical grid. The paper shows that optimal placement of the MIMO technology can decrease communication transmission delays, which in turn improves the reliability of power system operational protocols, such as distributed frequency regulation. Illustrative case studies using data from real-world power systems show that only upgrading the critical communication channels is sufficient to achieve system-wide performance improvement.

Cooperative and Competitive Multi-Agent Systems: From Optimization to Games

Multi-agent systems can solve scientific issues related to complex systems that are difficult or impossible for a single agent to solve through mutual collaboration and cooperation optimization. In a multi-agent system, agents with a certain degree of autonomy generate complex interactions due to the correlation and coordination, which is manifested as cooperative/competitive behavior. This survey focuses on multi-agent cooperative optimization and cooperative/non-cooperative games. Starting from cooperative optimization, the studies on distributed optimization and federated optimization are summarized. The survey mainly focuses on distributed online optimization and its application in privacy protection, and overviews federated optimization from the perspective of privacy protection mechanisms. Then, cooperative games and non-cooperative games are introduced to expand the cooperative optimization problems from two aspects of minimizing global costs and minimizing individual costs, respectively. Multi-agent cooperative and non-cooperative behaviors are modeled by games from both static and dynamic aspects, according to whether each player can make decisions based on the information of other players. Finally, future directions for cooperative optimization, cooperative/non-cooperative games, and their applications are discussed.

Distributed Online Optimization in Time-Varying Unbalanced Networks Without Explicit Subgradients

This paper studies a distributed online constrained optimization problem over time-varying unbalanced digraphs without explicit subgradients. In sharp contrast to the existing algorithms, we design a novel consensus-based distributed online algorithm with a local randomized zeroth-order oracle and then rescale the oracle by constructing row-stochastic matrices, which aims to address the unbalancedness of time-varying digraphs. Under mild conditions, the average dynamic regret over a time horizon is shown to asymptotically converge at a sublinear rate provided that the accumulated variation grows sublinearly with a specific order. Moreover, the counterpart of the proposed algorithm when subgradients are available is also provided, along with its dynamic regret bound, which reflects that the convergence of our algorithm is essentially not affected by the zeroth-order oracle. Simulations on distributed targets tracking problem and dynamic sparse signal recovery problem in sensor networks are employed to demonstrate the effectiveness of the proposed algorithm.

Distributed Online Optimization of Edge Computing With Mixed Power Supply of Renewable Energy and Smart Grid

Edge infrastructures, including edge computing servers, are increasingly powered by renewable energy and smart grid combined. Two-way energy trading allows the surplus or shortfall of renewable energy to be traded between a server and the smart grid, but is non-trivial due to randomly varying computation demands and renewables. This paper proposes a new online policy, namely, distributed online resource allocation and load management (DORL), which enables such an edge server and its serving devices to minimize their energy cost and energy consumption, respectively, in a fully distributed manner. The key idea is that we employ the stochastic dual-subgradient method to interpret the battery of the server as a virtual queue. Based on the virtual queue and task queues, the CPU frequencies of the devices and the edge server, the offloading transmit rates of the devices (to the server) and the energy trading decisions of the server (with the smart grid) are decoupled over time and among devices, and optimized on an ongoing basis. Furthermore, we prove that the DORL yields a feasible and asymptotically optimal solution with a cost-backlog tradeoff of <inline-formula> <tex-math notation="LaTeX">$[\eta, 1/\eta]$ </tex-math></inline-formula>. Simulations show that the DORL reduces the system cost by nearly 50&#x0025;, as compared to existing benchmarks.

An accelerated distributed online gradient push‐sum algorithm on time‐varying directed networks

AbstractThis paper investigates a distributed online optimization problem with convex objective functions on time‐varying directed networks, where each agent holds its own convex cost function and the goal is to cooperatively minimize the sum of the local cost functions. To tackle such optimization problems, an accelerated distributed online gradient push‐sum algorithm is firstly proposed, which combines the momentum acceleration technique and the push‐sum strategy. Then, we specifically analyze the regret for the proposed algorithm. The theoretical result shows that the individual regret of the proposed algorithm achieves a sublinear regret with order of , where T is the time horizon. Moreover, we implement the proposed algorithm in sensor networks for solving the distributed online estimation problem, and the results illustrate the effectiveness of the proposed algorithm.