Energy Arrival Process Research Articles

On Optimal Power Control for Energy Harvesting Communications With Lookahead

Consider the problem of power control for an energy harvesting communication system, where the transmitter is equipped with a finite-sized rechargeable battery and is able to look ahead to observe a fixed number of future energy arrivals. An implicit characterization of the maximum average throughput over an additive white Gaussian noise channel and the associated optimal power control policy is provided via the Bellman equation under the assumption that the energy arrival process is stationary and memoryless. A more explicit characterization is obtained for the case of Bernoulli energy arrivals by means of asymptotically tight upper and lower bounds on both the maximum average throughput and the optimal power control policy. Apart from their pivotal role in deriving the desired analytical results, such bounds are highly valuable from a numerical perspective as they can be efficiently computed using convex optimization solvers.

Deep Reinforcement Learning Optimal Transmission Algorithm for Cognitive Internet of Things With RF Energy Harvesting

Spectrum scarcity and energy limitation are becoming two critical issues in designing Internet of Things (IoT). As two promising technologies, cognitive radio (CR) and radio frequency (RF) energy harvesting can be used together to improve both energy and spectral efficiency. In this paper, an optimal transmission problem in a cognitive IoT (CIoT) with RF energy harvesting capability is investigated, where the optimization problem is formulated as a Markov decision process (MDP) without any priori-knowledge. Considering that the channel activity states of primary user network (PUN), RF energy arrival process and channel information are not available in advance, a deep reinforcement learning (DRL) based deep deterministic policy gradient (DDPG) algorithm is proposed to deal with the dynamic uplink access, working mode selection and continuous power allocation to maximize a long term uplink throughput. The simulation results show that the proposed algorithm is valid and efficient to achieve better performances when compared with deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula> -network (DQN) based, myopic and random algorithms.

Learning-Aided Dynamic Access Control in MEC-Enabled Green IoT Networks: A Convolutional Reinforcement Learning Approach

The ongoing growth of Internet of Things (IoT) traffic increasesthe emergence of Multi-Access Edge Computing (MEC), which can serve IoT devices to process edge missions properly. The power supply for IoT devices is addressed by energy harvesting (EH) technology. However, the energy arrival model obeys the unknown energy arrival process. In addition, due to the dynamical changes of network state, it is crucial to study the access control in the MEC-based green IoT system. In this paper, we propose an MEC-based access control strategy to maximize the system utility for the green IoT system, considering the spectral efficiency and the successful access ratio from the perspective of operators and users respectively. Specifically, we model the access control problem as a Constrained Markov Decision Process (CMDP), and develop a novel centralized reward-based experience replay deep convolutional Q network algorithm (RCQN) to attain the optimal access control for EH IoT devices. Additionally, to enrich the prior knowledge of access control strategy, we design a Long Short-Term Memory (LSTM) based energy prediction module to predict the energy state of IoT devices by learning the historical energy arrival information, whose output is taken as the state information of RCQN. Simulation results demonstrate that the proposed LSTM-based RCQN access control algorithm significantly improves the system utility with a proper reward design and training mechanism.

On the Optimality of the Greedy Policy for Battery Limited Energy Harvesting Communications

Consider a battery limited energy harvesting communication system with online power control. Assuming independent and identically distributed (i.i.d.) energy arrivals and the harvest-store-use architecture, it is shown that the greedy policy achieves the maximum throughput if and only if the battery capacity is below a certain positive threshold that admits a precise characterization. Simple lower and upper bounds on this threshold are established. The asymptotic relationship between the threshold and the mean of the energy arrival process is analyzed for several examples.

Secure Communication With Energy-Harvesting Buffer-Aided Jammer

Information security and self-sustainability are crucial to resource-limited wireless networks. In this paper, we assess the secrecy performance of the legitimate link without having eavesdropper's channel state information (CSI) while considering an energy harvesting (EH) jammer with finite and infinite energy buffer sizes. The EH jammer utilizes accumulated energy from its buffer and transmits artificial noise (AN) to ensure secure communication. To manage the uncertainty in harvested energy and consumption, we analyze on-off (OO) jamming and always-on (AO) jamming schemes. The analytical expressions are obtained for secrecy outage probability (SOP). This analysis models the energy arrival and departure process by a discrete-time continuous-state Markov chain to eliminate the computational complexity and inaccuracy of a discrete-state Markov chain. Both the schemes are compared using SOP and secure throughput. This analysis shows the adequacy of a finite buffer for an EH jammer and also the fact that the availability of jamming power has a more pronounced effect on SOP as compared to a larger number of antennas at the jammer.

A Reinforcement Learning Approach to Optimize Energy Usage in RF-Charging Sensor Networks

We consider a Radio Frequency (RF)-charging network where sensor devices harvest energy from a solar-powered Hybrid Access Point (HAP) and transmit their data to the HAP. We aim to optimize the power allocation of both the HAP and devices to maximize their Energy Efficiency (EE), which is defined as the total received data (in bits) for each Joule of consumed energy. Unlike prior works, we consider the case where both the HAP and devices have causal knowledge of channel state information and their energy arrival process. We model the power allocation problem as a Two-layer Markov Decision Process (TMDP), where the first layer corresponds to the HAP and the second layer consists of devices. We then outline a novel, decentralized Q-Learning (QL) solution that employs linear function approximation to represent the large state space. The simulation results show that when the HAP and devices employ our solution, their EE is orders of magnitude higher than competing policies.

Throughput Maximization by Deep Reinforcement Learning With Energy Cooperation for Renewable Ultradense IoT Networks

Ultradense network (UDN) is considered as one of the key technologies for the explosive growth of mobile traffic demand on the Internet of Things (IoT). It enhances network capacity by deploying small base stations in large quantities, but it simultaneously causes great energy consumption. In this article, we use energy harvesting (EH) and energy cooperation technologies to maximize system throughput and save energy. Considering that the energy arrival process and channel information are not available a priori , we propose an optimal deep reinforcement learning (DRL) algorithm to solve this average throughput maximization problem over a finite horizon. We also propose a multiagent DRL method to solve the dimensionality problem caused by the expansion of the state and action dimensions. Finally, we compare these algorithms with two traditional algorithms, greedy algorithm and conservative algorithm. The numerical results show that the proposed algorithms are valid and effective in increasing system average throughput on the long term.

Energy-Efficient Time-Domain Equilibrium Scheduling and Optimization Scheme for Energy Harvesting-Powered D2D Communication

Energy Harvesting- (EH-) powered Device-to-Device (D2D) Communication underlaying Cellular Network (EH-DCCN) has been deemed as one of the basic building blocks of Internet of Things due to its green energy efficiency and adjacent communication. But available energy will be one of the biggest obstacles when implementing EH-DCCN due to the immaturity of EH technology and the volatility of environmental energy resources. To improve energy utilization, this study investigates an efficient scheduling and power allocation scheme about transmission load equilibrium in the time domain. Accordingly, a short-term Sum Energy Efficiency (stSEE) maximization problem for EH-powered D2D communication is modelled, while ensuring a fundamental transmission rate requirement of cellular users. Consequently, the optimization problem is a nonconvex mixed integer nonlinear programming problem. Thus, we propose a two-layer convex approximation iteration algorithm which can obtain a feasible quasioptimal solution for the stSEE problem. Simultaneously, a two-step heuristic algorithm in a slot-by-slot fashion is also developed to acquire a suboptimal solution without requiring statistical knowledge of channel and energy arrival processes. Simulated analysis indicates that the short-term scheduling strategy can obtain better performances in terms of energy efficiency and transmission rate than conventional real-time scheduling scheme. Besides, the maximum scheduled number of EH-D2D pairs underlaying one cellular user under different EH efficiency is analysed, which can give us a theoretical reference about the deployment of future EH-DCCN.

A Maximin Optimal Online Power Control Policy for Energy Harvesting Communications

A general theory of online power control for discrete-time battery limited energy harvesting communications is developed, which leads to, among other things, an explicit characterization of a maximin optimal policy. This policy only requires the knowledge of the (effective) mean of the energy arrival process and maximizes the minimum asymptotic expected average reward (with the minimization taken over all energy arrival distributions of a given (effective) mean). Moreover, it is universally near optimal and has a strictly better worst-case performance as well as a strictly improved lower multiplicative factor in comparison with the fixed fraction policy proposed by Shaviv and \"{O}zg\"{u}r when the objective is to maximize the throughput over an additive white Gaussian noise channel. The competitiveness of this maximin optimal policy is also demonstrated via numerical examples.

Counterexamples on the monotonicity of delay optimal strategies for energy harvesting transmitters

We consider cross-layer design of delay optimal transmission strategies for energy harvesting transmitters where the data and energy arrival processes are stochastic. Using Markov decision theory, we show that the value function is weakly increasing in the queue state and weakly decreasing in the battery state. It is natural to expect that the delay optimal policy should be weakly increasing in the queue and battery states. We show via counterexamples that this is not the case. In fact, we show that for some sample scenarios the delay optimal policy may perform 5–13% better than the best monotone policy.

Energy Harvesting Wireless Sensor Networks With Channel Estimation: Delay and Packet Loss Performance Analysis

This paper considers a single-source status monitoring system with energy harvesting in the context of applications where keeping information up-to-date is of primary interest. Specifically, a sensor node, relying on harvested energy and operating according to a harvest-then-use protocol, estimates the channel state to decide whether to perform or defer data delivery to a sink. The sink keeps track of the system status through the successfully delivered updates. To comprehensively evaluate the performance of the proposed scheme, we analytically derive the exact-closed form expressions of the packet loss probability, the age of information and the update interval metric statistics considering both constant-rate and random energy arrival processes and taking into account both the time and energy costs of sensing, transmitting and estimating the channel state. We asymptotically obtain the necessary conditions under which estimating the channel state before transmitting, despite the associated time and energy costs, allow to efficiently manage the harvested energy by avoiding erroneous transmissions and performs strictly better than transmitting without estimating the channel state. Numerical results demonstrate that in most cases, estimating the channel state before transmitting significantly reduces the packet loss probability, the age of information and the update interval of node-sink transmission.

Non-Asymptotic Achievable Rates for Gaussian Energy-Harvesting Channels: Save-and-Transmit and Best-Effort

An additive white Gaussian noise energy-harvesting channel with an infinite-sized battery is considered. The energy arrival process is modeled as a sequence of independent and identically distributed random variables. The channel capacity $\frac {1}{2}\log (1+P)$ is achievable by the so-called best-effort and save-and-transmit schemes where $P$ denotes the battery recharge rate. This paper analyzes the save-and-transmit scheme whose transmit power is strictly less than $P$ and the best-effort scheme as a special case of save-and-transmit without a saving phase. In the finite blocklength regime, we obtain new non-asymptotic achievable rates for these schemes that approach the capacity with gaps vanishing at rates proportional to $1/\sqrt {n}$ and $({(\log n)/n})^{1/2}$ respectively where $n$ denotes the blocklength. The proof technique involves analyzing the escape probability of a Markov process. When $P$ is sufficiently large, we show that allowing the transmit power to back off from $P$ can improve the performance for save-and-transmit. The results are extended to a block energy arrival model where the length of each energy block $L$ grows sublinearly in $n$ . We show that the save-and-transmit and best-effort schemes achieve coding rates that approach the capacity with gaps vanishing at rates proportional to $\sqrt {L/n}$ and $({\max \{\log n, L\}/n})^{1/2}$ , respectively.

On the Energy and Data Storage Management in Energy Harvesting Wireless Communications

Energy harvesting (EH) in wireless communications has become the focus of recent transmission technology studies. Herein, energy storage modeling is one of the crucial design benchmarks that must be treated carefully. Understanding the energy storage dynamics and the throughput levels is essential especially for communication systems in which the performance depends solely on harvested energy. While energy outages should be avoided, energy overflows should also be prevented in order to utilize all harvested energy. Hence, a simple, yet comprehensive, analytical model that can represent the characteristics of a general class of EH wireless communication systems needs to be established. In this paper, invoking tools from large deviation theory along with Markov processes, a firm connection between the energy state of the battery and the data transmission process over a wireless channel is established for an EH transmitter. In particular, a simple exponential approximation for the energy overflow probability is formulated, with which the energy decay rate in the battery as a measure of energy usage is characterized. Then, projecting the energy outages and supplies on a Markov process, a discrete state model is established and an expression for the energy outage probability for given energy arrival and demand processes is provided. Finally, under energy overflow and outage constraints, the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">average data service (transmission) rate</i> over the wireless channel is obtained and the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">effective capacity</i> of the system, which characterizes the maximum data arrival rate at the transmitter buffer under quality-of-service (QoS) constraints imposed on the data buffer overflow probability, is derived.

Deep Reinforcement Learning Optimal Transmission Policy for Communication Systems With Energy Harvesting and Adaptive MQAM

In this paper, we study an optimal transmission problem in a point-to-point wireless communication system with energy harvesting and limited battery at its transmitter. Considering the non-availability of prior information about distribution on energy arrival process and channel coefficient, we propose a deep reinforcement learning (DRL) based optimal policy to allocate transmission power and adaptively adjust multi-ary modulation level according to the obtained causal information on harvested energy, battery state, and channel gain to achieve maximum throughput of the system. This optimization problem is formulated as a Markov decision process with unknown state transition probability. Applying the principle of the DRL, we use a deep Q-network to find the optimal solution in continuous state space, which provides rapid convergence since there is no additional memory required. Simulation results show that the proposed policy is effective and valid and it can improve the throughput of the system compared with Q-learning, greedy, random, and constant modulation level transmission policies.

Reinforcement Learning-Based Multiaccess Control and Battery Prediction With Energy Harvesting in IoT Systems

Energy harvesting (EH) is a promising technique to fulfill the long-term and self-sustainable operations for Internet of Things (IoT) systems. In this paper, we study the joint access control and battery prediction problems in a small-cell IoT system including multiple EH user equipments (UEs) and one base station (BS) with limited uplink access channels. Each UE has a rechargeable battery with finite capacity. The system control is modeled as a Markov decision process without complete prior knowledge assumed at the BS, which also deals with large sizes in both state and action spaces. First, to handle the access control problem assuming causal battery and channel state information, we propose a scheduling algorithm that maximizes the uplink transmission sum rate based on reinforcement learning (RL) with deep Q -network enhancement. Second, for the battery prediction problem, with a fixed round-robin access control policy adopted, we develop an RL-based algorithm to minimize the prediction loss (error) without any model knowledge about the energy source and energy arrival process. Finally, the joint access control and battery prediction problem is investigated, where we propose a two-layer RL network to simultaneously deal with maximizing the sum rate and minimizing the prediction loss: the first layer is for battery prediction, the second layer generates the access policy based on the output from the first layer. Experiment results show that the three proposed RL algorithms can achieve better performances compared with existing benchmarks.

Power Allocation for Energy Harvesting Wireless Communications With Energy State Information

Power allocation is investigated for a wireless transceiver pair, in which the transmitter is powered by energy harvesting and some knowledge of the energy arrival process is known. We propose to perform power allocation in two levels. In the upper level, average transmit power is allocated for stages (a stage contains a number of fading blocks). In the lower level, power allocation over fading blocks is performed within each stage, given average transmit power level allocated in the upper level for the stage. Optimal solution to the lower level that aims at ergodic capacity maximization is obtained in semi-closed-form. The upper level aims at maximization of the system throughput over the stages. As the upper-level problem is nonconvex, by using an approximation, we transform the problem into a monotonic optimization problem, which can be solved by existing numerical algorithms. Simulation demonstrates that, compared to an exhaustive search method, our proposed method has significantly less running time, at the cost of a not-significant throughput loss.

Optimal Aging–Aware Channel Access and Power Allocation for Battery–Powered Devices With Radio Frequency Energy Harvesting

Energy harvesting is a key technology for future energy neutral ubiquitous wireless communication systems. Among the many available sources of energy, RF harvesting is a natural choice for wireless communications since the devices can be efficiently designed for RF harvesting operations. To achieve energy autonomy a rechargeable energy storage is needed. However, the bursty nature of energy arrival associated with wireless power transfer and the energy usage pattern induced by devices’ activity may cause considerable stress to the battery and reduce its life span. In fact, deep charging and discharging cycles degrade the battery state of health, that is, the maximum amount of energy that can be stored. In this paper, a framework for the optimization of wireless nodes’ transmission and charging strategy is presented, where the battery aging rate is made explicit as a performance metric. The proposed framework is based on MDP theory, where the embedded stochastic model captures the energy arrival and data generation processes, and the control variable takes the form of a transmission power selection and energy acceptance/rejection. The tradeoff between the achievable throughput and the aging rate is explored for different parameters setting.

Online Power Control for Block i.i.d. Energy Harvesting Channels

We study the problem of online power control for energy harvesting communication nodes with random energy arrivals and a finite battery. We assume a block i.i.d. stochastic model for the energy arrivals, in which the energy arrivals are constant for a fixed duration ${T}$ , but are independent across different blocks, drawn from an arbitrary distribution. This model serves as a simple approximation to a random process with coherence time ${T}$ . We propose a simple online power control policy, and prove that its performance gap to the optimal throughput is bounded by a constant which is independent of the parameters of the problem. This also yields a simple formula for the approximately optimal long-term average throughput, which sheds some light on the qualitative behavior of the throughput and how it depends on the coherence time of the energy arrival process. Our results show that, perhaps counter-intuitively, for a fixed mean energy arrival rate the approximate throughput decreases with increasing coherence time ${T}$ of the energy arrival process. In particular, the battery size needed to approach the AWGN capacity of the channel increases linearly with the coherence time of the process. Finally, we show that our results can provide an approximation to the information-theoretic capacity of the same channel.

Remote Estimation of Correlated Sources Under Energy Harvesting Constraints

Remote estimation with an energy harvesting sensor with a limited data and energy buffer is considered. The sensor node observes an unknown temporally correlated field and communicates its observations to a remote fusion center using the energy it harvested. The fusion center employs linear minimum mean-square error estimation to reconstruct the unknown field. We provide performance guarantees for the estimation error under a block transmission scheme, where at each transmission block, data and energy buffers are completely emptied. Our bounds provide insights into how statistical properties of the energy harvesting process and buffer sizes may affect the estimation error. In particular, these bounds suggest insensitivity of the performance to buffer sizes for signals with low degree of freedom and suggest performance improvements with increasing buffer sizes for signals with relatively higher degree of freedom. Depending only on the mean, variance, and finite support of the energy arrival process, these results provide insights for the energy and data buffer sizes for deployment in future energy harvesting wireless sensing systems.

Energy Harvesting Communications Under Energy Underflow Constraints

We focus on energy management in a transmission system that harvests energy from the environment and stores the harvested energy in a battery. Because of the constraint in certain batteries that we cannot drain the stored energy completely, we study energy underflow, which is a result of the case the energy in the battery falls below a certain level. We formulate the available space to store more energy in the battery. Then, using the stochastic steady-state relation between the energy arrival and demand processes, we provide a closed-form expression for the energy underflow probability in the battery. Finally, we employ different energy demand processes from the battery, and then compare the average data service rates in the wireless channel under different energy underflow constraints.