Gradient Algorithm Research Articles

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

As renewable energy becomes more prevalent in the power grid, energy storage systems (ESSs) are playing an ever-increasingly crucial role in mitigating short-term supply–demand imbalances. However, the operation and control of ESS are not straightforward, given the ever-changing electricity prices in the market environment and the stochastic and intermittent nature of renewable energy generations, which respond to real-time load variations. In this paper, we propose a deep reinforcement learning (DRL) approach to address the electricity arbitrage problem associated with optimal ESS management. First, we analyze the structure of the optimal offline ESS control problem using the mixed-integer linear programming (MILP) formulation. This formulation identifies optimal control actions to absorb excess renewable energy and perform price arbitrage strategies. To tackle the uncertainties inherent in the prediction data, we then recast the online ESS control problem into a Markov Decision Process (MDP) framework and develop the DRL approach, which involves integrating the optimal offline control solution obtained from the training data into the training process and introducing noise to the state transitions. Unlike typical offline DRL training over a long time interval, we employ the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) algorithms with smaller neural networks training over a short time interval. Numerical studies demonstrate the promising potential of the proposed DRL-enabled approach for achieving better online control performance than the model predictive control (MPC) method under different price errors. This highlights the sample efficiency and robustness of our DRL approaches in managing ESS for electricity arbitrage.

Cloud-edge collaborative computing offloading and resource allocation optimization in LEO satellite network

Satellite edge computing can provide communication and computing services to areas lacking ground network coverage, enhancing the computational capabilities of terrestrial users. However, due to the relatively limited computational resources available on satellites, cloud-edge collaborative computing in Low Earth Orbit (LEO) satellite communication scenarios emerges as a more optimal solution. In this paper, a cloud-edge collaborative offloading algorithm based on Deep Deterministic Policy Gradient (DDPG) algorithm is proposed to solve the offloading decision and resource allocation problems. The optimization problem is modeled as Markov decision process, and the comprehensive delay and energy consumption cost of computing tasks completed by ground users are optimized. Simulation results validate that the proposed algorithm has good convergence properties and lower comprehensive cost than processing all tasks locally and offloading all tasks to LEO satellites.

Cooperative sensing, communication and computation resource allocation in mobile edge computing-enabled vehicular networks

The combination of integrated sensing and communication (ISAC) with mobile edge computing (MEC) enhances the overall safety and efficiency for vehicle to everything (V2X) system. However, existing works have not considered the potential impacts on base station (BS) sensing performance when users offload their computational tasks via uplink. This could leave insufficient resources allocated to the sensing tasks, resulting in low sensing performance. To address this issue, we propose a cooperative power, bandwidth and computation resource allocation (RA) scheme in this paper, maximizing the overall utility of Cramér-Rao bound (CRB) for sensing accuracy, computation latency for processing sensing information, and communication and computation latency for computational tasks. To solve the RA problem, a twin delayed deep deterministic policy gradient (TD3) algorithm is adopted to explore and obtain the effective solution of the RA problem. Furthermore, we investigate the performance tradeoff between sensing accuracy and summation of communication latency and computation latency for computational tasks, as well as the relationship between computation latency for processing sensing information and that of computational tasks by numerical simulations. Simulation demonstrates that compared to other benchmark methods, TD3 achieves an average utility improvement of 97.11% and 27.90% in terms of the maximum summation of communication latency and computation latency for computational tasks and improves 3.60 and 1.04 times regarding the maximum computation latency for processing sensing information.

DUCD: Deep Unfolding Convolutional-Dictionary network for pansharpening remote sensing image

The goal of pansharpening methods is to complement the spectral and spatial information contained in Multi-spectral (MS) and panchromatic (PAN) images to obtain the desired High-resolution multispectral (HRMS) image. The existing majority of pansharpening methods either extract feature information separately from the MS image and PAN image, or extract feature information after concatenating the MS image and PAN image. However, the entire extraction process lacks the utilization of complementary information and tends to generate redundant information, thereby leading to the loss of certain important information during the extraction process, which in turn affects the overall performance. In order to better utilize the complementary information between the MS image and PAN image and enhance the interpretability of the network, we propose the Deep Unfolding Convolutional-Dictionary Network (DUCD) for pansharpening in this paper. This network fully integrates complementary information between the MS image and PAN image to generate the final fused image. The entire network structure consists of two parts: The encoder and the decoder. In the encoder part of the network, we clarify the common and unique feature information between MS and PAN images by constructing an observation model. Simultaneously, we use the approximate gradient algorithm to continuously optimize the model and iteratively unfold it into a deep network structure. In the decoder part of the network, we concatenate the obtained common and specific information from MS and PAN images and pass them through convolutional and activation layers. Subsequently, they are input into the introduced Frequency Domain-based Transformer (FDT) module and an information-lossless inversible neural network(INN). This provides a more efficient method for establishing long-range dependency relationships between feature extraction and feature fusion. To demonstrate the effectiveness of our proposed method, we conduct extensive experiments on three benchmark datasets QB, GF2 and WV3. Experimental results show that our method outperforms the current SOTA Pansharpening methods in terms of performance.

Time-aware MADDPG with LSTM for multi-agent obstacle avoidance: a comparative study

AbstractIntelligent agents and multi-agent systems are increasingly used in complex scenarios, such as controlling groups of drones and non-player characters in video games. In these applications, multi-agent navigation and obstacle avoidance are foundational functions. However, problems become more challenging with the increased complexity of the environment and the dynamic decision-making interactions among agents. The Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm is a classical multi-agent reinforcement learning algorithm successfully used to improve agents’ performance. However, it ignores the temporal message hidden in agents’ interaction with the environment and needs to be more efficient in scenarios with many agents due to its training technique. To address the limitations of MADDPG, we propose to explore modified algorithms of MADDPG for multi-agent navigation and obstacle avoidance. By combining MADDPG with Long Short-Term Memory (LSTM), we obtain the MADDPG-LSTMactor algorithm, which leverages continuous observations over time as input for the policy network, enabling the LSTM layer to capture hidden temporal patterns. Moreover, by simplifying the input of the critic network, we obtain the MADDPG-L algorithm for efficiency improvement in scenarios with many agents. Experimental results demonstrate that these algorithms outperform existing networks in the OpenAI multi-agent particle environment. We also conducted a comparative study of the LSTM-based approach with Transformer and self-attention models in the task of multi-agent navigation and obstacle avoidance. The results reveal that Transformer and self-attention do not consistently outperform LSTM. The LSTM-based model exhibits a favorable tradeoff across varying sequence lengths. Overall, this work addresses the limitations of MADDPG in multi-agent navigation and obstacle avoidance tasks, providing insights for developing intelligent agents and multi-agent systems.

Signal Compensation Control With DDPG-Based Tuning Strategy for Electronic Throttle System

The electronic throttle system (ETS) is one of the important components of an automobile engine that adjust the engine air intake to directly affect the combustion power of the engine. In practical scenarios, Proportional-Integral-Derivative (PID) is a common and effective method. However, the traditional PID is difficult to satisfy the ETS performance requirements, e.g., the system response overshoot is not allowed to exist. Moreover, the controller gains are also difficult to tune manually. To tackle the above-mentioned issues, we propose a signal compensation control framework with an adaptive gain tuning strategy for the electronic throttle system, which consists of an upper-level to tune controller gains and a lower-level signal compensation controller to eliminate the nonlinear terms. The main feature of the signal compensation controller relies on two nonlinear compensators to remove the strong nonlinearity and uncertainty in the actual throttle system. Furthermore, we develop an adaptive gain tunning strategy based on Deep Deterministic Policy Gradient (DDPG) algorithm, and we analyze the corresponding stability and convergence properties of the proposed method. Finally, we illustrate the proposed algorithm on benchmarks and the tracking control problem using the real-world experiment platform of the ETS to show the effectiveness of the proposed algorithm.

Energy Management Strategy for Diesel–Electric Hybrid Ship Considering Sailing Route Division Based on DDPG

The continuous evolution of the shipping industry has resulted in increasingly severe environmental pollution. Hybrid ships can be a promising solution. However, for certain aspects such as the complex power structure of hybrid ships and the different characteristics of energy sources, it is difficult to control the energy conversion and transmission process and to reasonably allocate the power output of each energy unit. In this study, an improved K-means++ algorithm was newly adopted to divide the ship’s route. Subsequently, an energy management strategy (EMS) based on the Deep Deterministic Policy Gradient (DDPG) algorithm for diesel–electric hybrid ships was established. Based on the current navigation environment, the EMS can allocate the output power of each energy unit, and optimize the working space of the diesel engine in real time. The simulation results show that the DDPG-based EMS can reduce fuel consumption by 2.3%, 5.5%, 18.1%, and 20.1% with respect to the results of the Deep Q Network (DQN), Q Learning, HAAR, and rule-based algorithms, respectively.

A Large-Scale Multi-Agent Deep Reinforcement Learning Method for Cooperative Output Voltage Control of PEMFCs

To increase the output voltage stability and improve the operating efficiency of proton exchange membrane fuel cells (PEMFCs), a data-driven cooperative method for controlling the PEMFC output voltage is proposed in this paper. The proposed method adapts centralized learning and decentralized implementation, which can nonlinearly and adaptively realize optimal coordinated control over the hydrogen valve and the DC/DC converter. Additionally, a champion multiagent double delay deep deterministic policy gradient (CMA-4DPG) algorithm is proposed in this method, the design of which incorporates the policies of the champion selection mechanism, cooperative exploration, imitation learning guidance and curriculum guidance to improve the robustness of the cooperative method. The method cooperates with multiple controller to prevent output voltage fluctuation. The experimental results show that by simultaneously regulating the hydrogen flow through the hydrogen valve and the duty ratio of the DC/DC converter, the proposed method achieves better robustness and can improve the tracking performance of the PEMFC output voltage.

Convergence Analysis of Distributed Gradient Descent Algorithms With One and Two Momentum Terms

For the centralized optimization, it is well known that adding one momentum term (also called the heavy-ball method) can obtain a faster convergence rate than the gradient method. However, for the distributed counterpart, there is quite few results about the effect of added momentum terms on the convergence rate. This article is aimed at studying the issue in the distributed setup, where N agents minimize the sum of their individual cost functions using local communication over a network. The cost functions are twice continuously differentiable. We first study the algorithm with one momentum term and develop a distributed heavy-ball (D-HB) method by adding one momentum term on to the distributed gradient algorithm. By borrowing tools from the control theory, we provide a simple convergence proof and an explicit expression of the optimal convergence rate. Furthermore, we consider adding two momentum terms case and propose a distributed double-heavy-ball (D-DHB) method. We show that adding one momentum term allows faster convergence while adding two momentum terms does not perform any superiorities. Finally, simulation examples are given to illustrate our findings.

Parameter identification algorithm for ship manoeuvrability and wave peak model based multi-innovation stochastic gradient algorithm use data filtering technique

This paper addresses the issue of identifying ship motion parameters and wave peak frequency. Utilising the Euler discretisation principle, we establish a discrete-time auto-regressive moving average model with exogenous input (ARMAX) for the ship-wave system. Furthermore, we develop a filtering-based stochastic gradient algorithm for the system by applying filtering techniques and auxiliary model identification idea. A filtering-based multi-innovation stochastic gradient algorithm, utilizing the multi-innovation identification theory, was developed to enhance the convergence rate and accuracy of parameter identification. This approach was found to be more effective than the filtering-based stochastic gradient algorithm. Simulation results validate the efficacy of the proposed algorithm in parameter identification.

An uncertainty-based interpretable deep learning framework for predicting breast cancer outcome

BackgroundPredicting outcome of breast cancer is important for selecting appropriate treatments and prolonging the survival periods of patients. Recently, different deep learning-based methods have been carefully designed for cancer outcome prediction. However, the application of these methods is still challenged by interpretability. In this study, we proposed a novel multitask deep neural network called UISNet to predict the outcome of breast cancer. The UISNet is able to interpret the importance of features for the prediction model via an uncertainty-based integrated gradients algorithm. UISNet improved the prediction by introducing prior biological pathway knowledge and utilizing patient heterogeneity information.ResultsThe model was tested in seven public datasets of breast cancer, and showed better performance (average C-index = 0.691) than the state-of-the-art methods (average C-index = 0.650, ranged from 0.619 to 0.677). Importantly, the UISNet identified 20 genes as associated with breast cancer, among which 11 have been proven to be associated with breast cancer by previous studies, and others are novel findings of this study.ConclusionsOur proposed method is accurate and robust in predicting breast cancer outcomes, and it is an effective way to identify breast cancer-associated genes. The method codes are available at: https://github.com/chh171/UISNet.

Duty Cycle Scheduling in Wireless Sensor Networks Using an Exploratory Strategy-Directed MADDPG Algorithm

This paper presents an in-depth study of the application of Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithms with an exploratory strategy for duty cycle scheduling (DCS) in the wireless sensor networks (WSNs). The focus is on optimizing the performance of sensor nodes in terms of energy efficiency and event detection rates under varying environmental conditions. Through a series of simulations, we investigate the impact of key parameters such as the sensor specificity constant α and the Poisson rate of events on the learning and operational efficacy of sensor nodes. Our results demonstrate that the MADDPG algorithm with an exploratory strategy outperforms traditional reinforcement learning algorithms, particularly in environments characterized by high event rates and the need for precise energy management. The exploratory strategy enables a more effective balance between exploration and exploitation, leading to improved policy learning and adaptation in dynamic and uncertain environments. Furthermore, we explore the sensitivity of different algorithms to the tuning of the sensor specificity constant α, revealing that lower values generally yield better performance by reducing energy consumption without significantly compromising event detection. The study also examines the algorithms' robustness against the variability introduced by different event Poisson rates, emphasizing the importance of algorithm selection and parameter tuning in practical WSN applications. The insights gained from this research provide valuable guidelines for the deployment of sensor networks in real-world scenarios, where the trade-off between energy consumption and event detection is critical. Our findings suggest that the integration of exploratory strategies in MADDPG algorithms can significantly enhance the performance and reliability of sensor nodes in WSNs.

Single-lever control method design based on power management system and deep reinforcement learning for turboprop engines

This paper presents a single-lever control method based on Power Management System (PMS) and improved Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm for turboprop engines. In this approach, power level angle command, which is the single-lever command, is decoupled into controlled variable commands by PMS, and the controller based on improved TD3 algorithm can ensure that controlled variables track their commands rapidly and accurately. To achieve the optimal conversion relationship between different commands, an offline optimization process is used to design PMS. By optimization, specific fuel consumption and propeller efficiency are both improved after conversion. To deal with strong interactions between different control loops of a turboprop engine, TD3 algorithm which is a deep reinforcement learning algorithm is adopted. Two improvements which are the design method of observation state and prioritized experience replay are made to enhance the tracking accuracy. Simulation results show that improved TD3 algorithm can learn an optimal control policy to guarantee good control effect with fast response and small overshoot. The maximum settling time is less than 0.25s and the maximum overshoot is less than 0.1%. It also has a good robustness performance when the plant exists model uncertainties. The maximum fluctuations are less than 0.05%.

A Robust Distributed Algorithm for Solving the Economic Dispatch Problem with the Penetration of Renewables and Battery Systems

In the field of energy networks, for their effective functioning, it is necessary to distribute the required load between all online generating units in a proper way to cover the demand. The load schedule is obtained by solving the so-called Economic Dispatch Problem (EDP). The EDP can be solved in many ways, resulting in a power distribution plan between online generating units in the network so that the resulting price per unit of energy is minimal. This article focuses on designing a distributed gradient algorithm for solving EDP, supplemented by models of renewable sources, Battery Energy Storage System (BESS), variable fuel prices, and consideration of multiple uncertainties at once. Specifically, these are: time-variable transport delays, noisy gradient calculation, line losses, and drop-off packet representations. The algorithm can thus be denoted as robust, which can work even in unfavorable conditions commonly found in real applications. The capabilities of the presented algorithm will be demonstrated and evaluated on six examples.

Obtaining Lower Query Complexities Through Lightweight Zeroth-Order Proximal Gradient Algorithms.

Zeroth-order (ZO) optimization is one key technique for machine learning problems where gradient calculation is expensive or impossible. Several variance, reduced ZO proximal algorithms have been proposed to speed up ZO optimization for nonsmooth problems, and all of them opted for the coordinated ZO estimator against the random ZO estimator when approximating the true gradient, since the former is more accurate. While the random ZO estimator introduces a larger error and makes convergence analysis more challenging compared to coordinated ZO estimator, it requires only O(1) computation, which is significantly less than O(d) computation of the coordinated ZO estimator, with d being dimension of the problem space. To take advantage of the computationally efficient nature of the random ZO estimator, we first propose a ZO objective decrease (ZOOD) property that can incorporate two different types of errors in the upper bound of convergence rate. Next, we propose two generic reduction frameworks for ZO optimization, which can automatically derive the convergence results for convex and nonconvex problems, respectively, as long as the convergence rate for the inner solver satisfies the ZOOD property. With the application of two reduction frameworks on our proposed ZOR-ProxSVRG and ZOR-ProxSAGA, two variance-reduced ZO proximal algorithms with fully random ZO estimators, we improve the state-of-the-art function query complexities from Omindn1/2ε2,dε3 to O˜n+dε2 under d>n12 for nonconvex problems, and from Odε2 to O˜nlog1ε+dε for convex problems. Finally, we conduct experiments to verify the superiority of our proposed methods.

Fuzzy-based predictive deep reinforcement learning for robust and constrained optimal control of industrial solar thermal plants

Integrating distributed solar fields (DSFs) into conventional heat and power plants (CHPs) of industries is mostly constrained by the availability of a real-time capable control scheme. Safe and efficient operation of industrial DSF requires the supply of a fluctuating and periodically available energy at the required temperature while reducing losses and ensuring operational constraint. Reinforcement learning (RL), specifically Q-learning methods, are being recently applied for such control tasks. However, existing RL approaches cannot be directly applied to continuous domain control tasks or tasks with operation constraints. While using deep learning in model-free RL can remove these limitations, tractability and scalability boundaries on performing the deep learning are becoming significant, with scarce data and multi-objective concerns. In contrast, model-based deep RL schemes can be applied for sample efficiency and satisfying operational constraints, but their modeling framework is not general and accurate. To address these challenges, the work here develops a hybrid fuzzy convolution model (HFCM) that takes full advantages of data, models (dynamic and steady-state), and prior knowledge on industrial DSF. The HFCM is then extended for use in deep deterministic policy gradient (DDPG) algorithm to learn the DSF control task. This is done so by solving a multi-objective optimization problem, which is formulated as a constrained Markov decision process (CMDP) with continuous state and actions. Some practical and relevant findings were made with this HFCM. Firstly, it allowed the DDPG agent to learn independently a temperature state and a disturbance state, using only temperature measurement and single realizations of state vector. It also permitted the testing of an operational state violation strategy on a DDPG actions, and at the same time, the correction of the effects at a safety layer, if needed. Furthermore, the proposed control strategy simultaneously reduced mean temperature tracking error by 24%–51% and energy gain by 13.9%–17.35% when compared respectively to model-free DDPG and MPC baselines.

Motion planning framework based on dual-agent DDPG method for dual-arm robots guided by human joint angle constraints.

Reinforcement learning has been widely used in robot motion planning. However, for multi-step complex tasks of dual-arm robots, the trajectory planning method based on reinforcement learning still has some problems, such as ample exploration space, long training time, and uncontrollable training process. Based on the dual-agent depth deterministic strategy gradient (DADDPG) algorithm, this study proposes a motion planning framework constrained by the human joint angle, simultaneously realizing the humanization of learning content and learning style. It quickly plans the coordinated trajectory of dual-arm for complex multi-step tasks. The proposed framework mainly includes two parts: one is the modeling of human joint angle constraints. The joint angle is calculated from the human arm motion data measured by the inertial measurement unit (IMU) by establishing a human-robot dual-arm kinematic mapping model. Then, the joint angle range constraints are extracted from multiple groups of demonstration data and expressed as inequalities. Second, the segmented reward function is designed. The human joint angle constraint guides the exploratory learning process of the reinforcement learning method in the form of step reward. Therefore, the exploration space is reduced, the training speed is accelerated, and the learning process is controllable to a certain extent. The effectiveness of the framework was verified in the gym simulation environment of the Baxter robot's reach-grasp-align task. The results show that in this framework, human experience knowledge has a significant impact on the guidance of learning, and this method can more quickly plan the coordinated trajectory of dual-arm for multi-step tasks.

Improve exploration in deep reinforcement learning for UAV path planning using state and action entropy

Despite being a widely adopted development framework for unmanned aerial vehicle (UAV), deep reinforcement learning is often considered sample inefficient. Particularly, UAV struggles to fully explore the state and action space in environments with sparse rewards. While some exploration algorithms have been proposed to overcome the challenge of sparse rewards, they are not specifically tailored for UAV platform. Consequently, applying those algorithms to UAV path planning may lead to problems such as unstable training processes and neglect of action space comprehension, possibly causing negative impacts on the path planning results. To address the problem of sparse rewards in UAV path planning, we propose an information-theoretic exploration algorithm, Entropy Explorer (EE), specifically for UAV platform. The proposed EE generates intrinsic rewards based on state entropy and action entropy to compensate for the scarcity of extrinsic rewards. To further improve sampling efficiency, a framework integrating EE and Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithms is proposed. Finally, the TD3-EE algorithm is tested in AirSim and compared against benchmarking algorithms. The simulation outcomes manifest that TD3-EE effectively stimulates the UAV to comprehensively explore both state and action spaces, thereby attaining superior performance compared to the benchmark algorithms in the realm of path planning.

DDPG-based reinforcement learning for controlling a spatial three-section continuum robot

This paper proposes an approach to controlling a spatial three-section continuum robot using reinforcement learning (RL). Rather than relying on traditional methods that use bending and orientation angles, this study utilizes the curvature of each section to achieve the desired position. The training process involves identifying a 3D workspace for the robot and utilizing a deep neural network to optimize its control policies. The results demonstrate that the robot’s end effector can follow a given trajectory with a maximum error of 1 [1/mm] based on the generated curvature for each section. The Deep Deterministic Policy Gradients (DDPG) algorithm is employed to optimize the robot’s control policies. Additionally, the developed DDPG algorithm is compared to the Deep Q-Network (DQN) from a precision standpoint, particularly during the tracking of a circular trajectory. This comparison highlights the superior accuracy of DDPG in managing the intricate movements required for continuum robotics, making it a promising tool for applications where precision is crucial, such as in medical and surgical robotics.

Closest Farthest Widest

The current paper proposes and tests algorithms for finding the diameter of a compact convex set and the farthest point in the set to another point. For these two nonconvex problems, I construct Frank–Wolfe and projected gradient ascent algorithms. Although these algorithms are guaranteed to go uphill, they can become trapped by local maxima. To avoid this defect, I investigate a homotopy method that gradually deforms a ball into the target set. Motivated by the Frank–Wolfe algorithm, I also find the support function of the intersection of a convex cone and a ball centered at the origin and elaborate a known bisection algorithm for calculating the support function of a convex sublevel set. The Frank–Wolfe and projected gradient algorithms are tested on five compact convex sets: (a) the box whose coordinates range between −1 and 1, (b) the intersection of the unit ball and the non-negative orthant, (c) the probability simplex, (d) the Manhattan-norm unit ball, and (e) a sublevel set of the elastic net penalty. Frank–Wolfe and projected gradient ascent are about equally fast on these test problems. Ignoring homotopy, the Frank–Wolfe algorithm is more reliable. However, homotopy allows projected gradient ascent to recover from its failures.