Energy Arrival Research Articles

LSTM-Characterized Deep Reinforcement Learning for Continuous Flight Control and Resource Allocation in UAV-Assisted Sensor Network

Unmanned aerial vehicles (UAVs) can be employed to collect sensory data in remote wireless sensor networks (WSNs). Due to UAV’s maneuvering, scheduling a sensor device to transmit data can overflow data buffers of the unscheduled ground devices. Moreover, lossy airborne channels can result in packet reception errors at the scheduled sensor. This article proposes a new deep reinforcement learning-based flight resource allocation framework (DeFRA) to minimize the overall data packet loss in a continuous action space. DeFRA is based on deep deterministic policy gradient (DDPG), optimally controls instantaneous headings and speeds of the UAV, and selects the ground device for data collection. Furthermore, a state characterization layer, leveraging long short-term memory (LSTM), is developed to predict network dynamics, resulting from time-varying airborne channels and energy arrivals at the ground devices. To validate the effectiveness of DeFRA, experimental data collected from a real-world UAV testbed and energy harvesting WSN are utilized to train the actions of the UAV. Numerical results demonstrate that the proposed DeFRA achieves a fast convergence while reducing the packet loss by over 15%, as compared to the existing deep reinforcement learning solutions.

Learning-Aided Dynamic Access Control in MEC-Enabled Green IoT Networks: A Convolutional Reinforcement Learning Approach

The ongoing growth of Internet of Things (IoT) traffic increasesthe emergence of Multi-Access Edge Computing (MEC), which can serve IoT devices to process edge missions properly. The power supply for IoT devices is addressed by energy harvesting (EH) technology. However, the energy arrival model obeys the unknown energy arrival process. In addition, due to the dynamical changes of network state, it is crucial to study the access control in the MEC-based green IoT system. In this paper, we propose an MEC-based access control strategy to maximize the system utility for the green IoT system, considering the spectral efficiency and the successful access ratio from the perspective of operators and users respectively. Specifically, we model the access control problem as a Constrained Markov Decision Process (CMDP), and develop a novel centralized reward-based experience replay deep convolutional Q network algorithm (RCQN) to attain the optimal access control for EH IoT devices. Additionally, to enrich the prior knowledge of access control strategy, we design a Long Short-Term Memory (LSTM) based energy prediction module to predict the energy state of IoT devices by learning the historical energy arrival information, whose output is taken as the state information of RCQN. Simulation results demonstrate that the proposed LSTM-based RCQN access control algorithm significantly improves the system utility with a proper reward design and training mechanism.

Deep Reinforcement Learning Resource Allocation in Wireless Sensor Networks With Energy Harvesting and Relay

Green wireless communications have been extensively studied in wireless sensor networks (WSNs), including the use of new energy, renewable energy, and low-power consumption and energy-saving technologies for years. In these networks, due to channel fading, insufficient and random energy arrival, some possible bad deployment of sensors, etc., the communication among sensor nodes in a WSNs will inevitably be affected or even interrupted sometimes, which may result in unacceptable performance in the entire network. In order to solve this problem, we propose a WSN composing of several local subnetworks with amplified forwarding relay and specially designed working time cycle. In this network, we study our resource allocation policies to manage both power and time for throughput maximization. We use deep reinforcement learning (DRL) to develop our resource allocation policies under the model constructed as a Markov decision process for this optimization problem in the subnetwork. We apply an actor–critic strategy to find our optimal solution in continuous state and action space and adaptively achieve maximum throughput of this network based on energy harvesting, causal information of battery state and channel gains. The simulation results demonstrate that the proposed transmission policies can produce higher throughput in the local network and finally improve overall system performance in comparison with greedy policy, random policy, and conservative policy.

A Stochastic Geometry Analysis for Energy-Harvesting-Based Device-to-Device Communication

The rapidly developing energy harvesting (EH) technology is a promising solution to the durability issue in the battery-powered Internet of Things (IoT) systems. In this article, underlaid device-to-device (D2D) transmission powered by radio signals harvested from cellular systems is studied. By considering the dilemmas among EH, D2D transmission opportunity, and interference management, we propose two transmission policies: 1) Policy 1 requires that the available power in the battery should be no less than the D2D transmission power and 2) Policy 2 not only sets the constraint on available power but also introduces the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">guard zone</i> rule to protect D2D transmissions from severe interference. The employment of a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">guard zone</i> in this situation is technically challenging since the original distribution of energy arrival will thereby be changed. We derive expressions in closed or semiclosed forms for the considered D2D transmission performance metrics with the stochastic geometry framework and Poisson hole process. With numerical simulation results, the influences of varying network parameters on D2D performances are illustrated. The results show that by introducing a guard zone, the D2D successful transmission rate can be increased by 41.2%. All the developed D2D frameworks and the summarized useful remarks are used to provide meaningful design insights and guidelines for the deployment strategies of EH-based D2D wireless networks.

CyEnSe: Cyclic energy-aware scheduling for energy-harvested embedded systems

New advances in energy harvesting from human's body enable battery-less wearable and implantable devices to have more advanced functionalities and hence higher number of tasks to be scheduled on the device. The nature of variable and cyclic energy harvesting from periodic body movements (e.g: walking, heartbeat) challenges the tasks scheduling in these real-time systems, especially for the safety-critical applications when the tasks deadlines are hard. For the first time, this paper addresses cyclic energy harvesting in an offline cyclic-executive scheduling. The paper defines CyEnSe, a scheduling problem in presence of cyclic energy. It proposes two methods, a heuristic, and a linear-programming to solve the hard-deadline periodic task scheduling problem while considers variable cyclic energy arrivals from the energy harvester. We present a comparison among the proposed methods and a First-Fit approach from different optimization point of views, for instance, task utilization, number of tasks, and amount of harvested energy.

Organizational Resource Allocation by Mobile Edge Computing in the Context of the Internet of Things

In recent years, the number of electronic devices and users has increased sharply with the rapid development and progress of electronic information technology. The motivation of this paper is to optimize the organization’s resource allocation strategy in the Internet of Things (IoT) environment. The optimal path planning and information processing efficiency are improved through Unmanned Aerial Vehicle (UAV) technology and migration optimization algorithm. The research method is migration optimization algorithm and UAV dynamic network. The information processing capability of traditional wireless communication systems has gradually been unable to meet the actual information processing needs. Therefore, a static network migration algorithm is constructed based on multiple-user multilateral edge computing servers. It migrates each user’s information processing task to the neighboring edge confidence processing server and uses the information processing server to perform auxiliary calculations. The simulation model adds a utility function that simulates energy consumption, delay weighting, and maximum extreme value, combined with the allocation strategy of optimizing each user’s information processing task to achieve the optimization goal. The static network migration algorithm established in this simulation has better results than other benchmark algorithms. Both scenario 1 and scenario 2 in the simulation show very close performance to the optimal solution. Meanwhile, a migration algorithm that can provide wireless charging for UAVs is built by a dynamic edge computing model based on the time associated with the UAV base station and multiple end users. Combined with completing the information processing tasks in each time slot, the energy arrival is also non-directional. The dynamic network migration algorithm can optimize the number of tasks absorbed by the end-user based on the current online status of the system without knowing the global information. The optimized target equation is related to the queue stability, and the parameter V has a linear relationship with the queue backlog length. Here, the problem of computing migration is studied in Mobile Edge Computing (MEC). The results show that the utility function of the weighted sum has an approximately linear relationship with the weights. As the value of the utility function increases, so does the weight function. The optimal data throughput of the proposed model is 70,000 bits, while the optimal data throughput of the state-of-the-art model is 68,000 bits. Therefore, the data transmission performance of the model presented here is better than that of other models. MEC can be significantly improved the efficiency of organizational resource allocation. Combining UAV and wireless charging technology, the computing and communication resource allocation issues of the UAV’s edge computing system are comprehensively discussed to improve the performance and efficiency of the network.

Optimizing Information Freshness in MEC-Assisted Status Update Systems With Heterogeneous Energy Harvesting Devices

The ever-growing number of Internet-of-Things (IoT) devices makes multiaccess edge computing (MEC)-assisted status update system more and more attractive, which can be deployed to enable remote data acquisition and analysis from urban space. The ambient computing resource at edge automatically extracts valuable status update information from the data collected by IoT devices, which supports the real-time remote monitoring applications. In this article, we employ the concept of Age of Information (AoI) to quantify the freshness of status updates. To combat the limited battery capacity at IoT devices, energy harvesting (EH) is leveraged to capture the green energy from ambient environment. Specifically, we investigate an age minimization problem by considering the randomness in energy arrivals, heterogeneity in harvesting mode, and the stochasticity in transmission and computing process. The formulated problem is a long-term stochastic optimization problem. Then, we transform the original problem into a series of per-time slot deterministic optimization problem. An online scheduling policy is proposed to obtain the energy management decisions at devices, and the transmission and computing scheduling decisions among multiple devices without any prior knowledge on the network dynamics, which is facilitated to be implemented. Simulation results show that the performance of our proposed algorithm is competitive when compared with other existing schemes.

On the Optimality of the Greedy Policy for Battery Limited Energy Harvesting Communications

Consider a battery limited energy harvesting communication system with online power control. Assuming independent and identically distributed (i.i.d.) energy arrivals and the harvest-store-use architecture, it is shown that the greedy policy achieves the maximum throughput if and only if the battery capacity is below a certain positive threshold that admits a precise characterization. Simple lower and upper bounds on this threshold are established. The asymptotic relationship between the threshold and the mean of the energy arrival process is analyzed for several examples.

Online Power Control and Optimization for Energy Harvesting Communication System Based on State of Charge

In this paper, the online power control problem for energy harvesting wireless communication system with a finite storage capacity battery is addressed, where the channel state and energy harvesting rate are both unknown. A low complexity algorithm based online convex optimization is proposed to guarantee energy availability of energy harvesting node and maximize average long-term throughput. The proposed algorithm restricts maximum transmission power with the information of state of charge, and allocates transmission power based on historical information. In addition, energy availability constraint is given by rigorous theoretical analysis to guarantee the optimization of average long-term throughput. Simulations have been conducted to demonstrate the effectiveness of the algorithm without considering probability distribution of energy arrival or channel coefficients. The proposed algorithm outperforms counterparts in different energy harvested rates.

Efficient Blockchain-Enabled Large Scale Parked Vehicular Computing With Green Energy Supply

While the vehicular network enables geographically distributed cooperative computation, its mature implementation has long been constrained due to the lack of an effective management platform. In this paper, employing the security and privacy attributes of blockchain, we propose a novel Blockchain-enabled Large-scale Parked Vehicular Computing (BLPVC) architecture to utilize the potential solar energy and vehicular computational resources in the outdoor parking lot. However, the uneven green power supply and random arrival time of electric vehicles compose the highly complex environment. Accordingly, in this paper, to handle the efficient utilization of the distributed resources by blockchain technology, we propose an integrated optimization framework which leverages the green energy utilization and service latency limit among the processes of block generation, task computing, and communication, whereas such a design leads to the mixed-timescale stochastic optimization problem. To this end, corresponding to the dynamic solar energy arrival, we propose a shaped deep deterministic policy gradient (DDPG) algorithm to accelerate the learning rate of computational frequency control in the short-term stage; while in the long-term stage, for the mixed-integer programming (MIP) of task offloading and blockchain parameters adjustment, a series of transformation is employed to preserve convexity. Finally, experiments are carried out on Python demonstrating that the proposed scheme achieves a balanced performance between service latency and distributed resources, while the battery depreciation cost is heavily reduced.

Online Power Control Algorithm for Secure Relay Transmission Network with Energy Harvesting

For amplify-and-forward relay networks where both the source node and the relay node are powered by the harvested energy and the information for the two destination nodes are required to keep secrecy each other, an algorithm is proposed to maximize the long-term average secrecy rate by jointly optimizing the transmission power of the source node and the relay node. Since the energy arrivals and channel states are stochastic processes, the problem is a stochastic optimization problem. The Lyapunov optimization framework is used to transform the long-term optimization problem into a “virtual queue drift plus penalty” minimization problem per time slot under the constraints of battery operation and energy using. The transformed optimization problem is solved. The simulation results show that the proposed algorithm has significant advantages over the comparison algorithms in the long-term average secrecy rate. Furthermore, the proposed algorithm only depends on the current battery state and channel state to make the decision, which is a practical and low-complexity algorithm.

Learning to Charge RF-Energy Harvesting Devices in WiFi Networks

In this paper, we consider a solar-powered Access Point (AP) that is tasked with supporting both non-energy harvesting or legacy data users such as laptops, and devices with Radio Frequency (RF)-energy harvesting and sensing capabilities. We propose two solutions that enable the AP to manage its harvested energy via transmit power control and also ensure devices perform sensing tasks frequently. Advantageously, our solutions are suitable for current wireless networks and do not require perfect channel gain information or non-causal energy arrival at devices. The first solution uses a deep Q-network (DQN) whilst the second solution uses Model Predictive Control (MPC) to control the AP's transmit power. Our results show that our DQN and MPC solutions improve energy efficiency and user satisfaction by respectively 16% to 35%, and 10% to 42% as compared to competing algorithms.

Computation Offloading in Heterogeneous Mobile Edge Computing With Energy Harvesting

Energy harvesting aided mobile edge computing (MEC) has gained much attention for its widespread application in the computation-intensive, latency-sensitive and energy-hungry scenario. Computation offloading, which leverages powerful MEC servers (MEC-ss) to augment the computing capability of less powerful mobile devices (MDs), is intrinsically a distributed computing over heterogeneous MEC networks. In this article, computation offloading from multi-MD to multi-MEC-s in heterogeneous MEC systems with energy harvesting is investigated from a game theoretic perspective. The objective is to minimize the average response time of an MD that consists of communication time, waiting time and processing time. M/G/1 queueing models are established for MDs' computation generation and MEC-ss' computation task receiving. The interference among MDs, the randomness in computation task generation, harvested energy arrival, wireless channel state, queueing at the MEC-s, and the power budget constraint of an MD are taken into consideration. A noncooperative computation offloading game is formulated. The action is a vector that denotes the amount of computation tasks offloaded to all MEC-ss (the element value can be zero) and local process. We give the definition and existence analysis of the Nash equilibrium (NE). Furthermore, we reconstruct the optimization problem of an MD. A 2-step decomposition is presented and performed. Thereby, we arrive at a one-dimensional search problem and a greatly shrunken sub-problem. The sub-problem is nonconvex, but its Karush-Kuhn-Tucker (KKT) conditions have finite solutions. We can obtain the optimal solution of the sub-problem by seeking the finite solutions. Thereafter, a distributive NE-orienting iterated best-response algorithm is designed. Simulations are carried out to illustrate the convergence performance and effectiveness of the proposed algorithm.

Outage Aware Power Control for Vehicular Wireless Energy Transfer Over Dynamic Channels

Wireless powered devices suffer from dynamic energy arrival in case of fast fading channel conditions caused by high mobility of the energy base station (BS), which may lead to energy outage for practical applications executed at the wireless device (WD). We use effective bandwidth theory to investigate the tail distribution of the WD's energy queue, from which the energy outage probability (EOP) is derived. Then, a power control problem is proposed to minimize the mobile vehicular BS's transmit power while guaranteeing the EOP for the WD, which can be equivalently addressed by solving a single equation. The match between simulation results and theoretical analysis demonstrates the validity of the proposed EOP analysis. Furthermore, the proposed scheme can reduce the power consumption of the vehicular BS while guaranteeing EOP requirement of the WD.

Secure Communication With Energy-Harvesting Buffer-Aided Jammer

Information security and self-sustainability are crucial to resource-limited wireless networks. In this paper, we assess the secrecy performance of the legitimate link without having eavesdropper's channel state information (CSI) while considering an energy harvesting (EH) jammer with finite and infinite energy buffer sizes. The EH jammer utilizes accumulated energy from its buffer and transmits artificial noise (AN) to ensure secure communication. To manage the uncertainty in harvested energy and consumption, we analyze on-off (OO) jamming and always-on (AO) jamming schemes. The analytical expressions are obtained for secrecy outage probability (SOP). This analysis models the energy arrival and departure process by a discrete-time continuous-state Markov chain to eliminate the computational complexity and inaccuracy of a discrete-state Markov chain. Both the schemes are compared using SOP and secure throughput. This analysis shows the adequacy of a finite buffer for an EH jammer and also the fact that the availability of jamming power has a more pronounced effect on SOP as compared to a larger number of antennas at the jammer.

Improvement on Age of Information for Information Update Systems With HARQ Chase Combining and Sensor Harvesting-Transmitting Diversities

The age of information (AoI) has attracted increasing attentions as a new metric and tool to capture the timeliness of reception and freshness of data, which is different from the conventional metric such as outage, delay, and throughput. While most existing works focused on AoI analysis and optimizing method, the information update system design by utilizing diversity to reduce AoI is not fully exploited so far. Inspired by this fact, a novel information status update system is designed by jointing wireless energy harvesting diversity, status update transmitting diversity, and hybrid automatic repeat request (HARQ). Specially, the update system consists of two sensors that can be seen as an unified entity from the destination's perspective, and one destination. The two sensors, which have finite-size batteries and synchronously operate in half-duplex mode, harvest energy from a Bernoulli process source, and cooperatively deliver the status update to destination by sharing a common information source. To exploit the information of the unsuccessfully decoding update packet, the HARQ and chase combining (HARQ-CC) are jointly used at destination so that the HARQ-CC receiving diversity is also exploited. By using Bernoulli process energy arrival probability and sensor transmission probability, we first propose one novel sensor energy harvesting and update delivering model that can be represented by Markov Chain (MC) and develop one novel construction method of the transition matrix of sensor battery energy states. Considering the developed energy state transmission matrix column stochastic, irreducible, and aperiodic, we obtain the stationary distribution of sensor battery energy states so that the steady transmitting power is derived. Secondly, the average AoI of the dual-sensor status update system is derived in terms of a closed-form expression by accurately calculating the statistical descriptions of the maximum retransmission number (time) in HARQ-CC and the service time of one packet. The analytical results are validated by using the comparison from simulations. Moreover, compared with traditional single sensor system, the dual-sensor system not only harvests more energy, but also has smaller average AoI because of the joint exploitation of multi-diversity. Besides, the impact of the system parameters on harvested energy and average AoI is provided, which provides insights in terms of the optimal design of Internet of things system.

Dynamic Resource Optimization for Decentralized Estimation in Energy Harvesting IoT Networks

We study decentralized estimation of time-varying signals at a fusion center, when energy harvesting sensors transmit sampled data over rate-constrained links. We propose dynamic strategies to select radio parameters, sampling set, and harvested energy at each node, with the aim of estimating a time-varying signal while ensuring: 1) the accuracy of the recovery procedure and 2) the stability of the batteries around a prescribed operating level. The approach is based on stochastic optimization tools, which enable adaptive optimization without the need of apriori knowledge of the statistics of radio channels and energy arrivals processes. Numerical results validate the proposed approach for decentralized signal estimation under communication and energy constraints typical of Internet-of-Things scenarios.

A Reinforcement Learning Approach to Optimize Energy Usage in RF-Charging Sensor Networks

We consider a Radio Frequency (RF)-charging network where sensor devices harvest energy from a solar-powered Hybrid Access Point (HAP) and transmit their data to the HAP. We aim to optimize the power allocation of both the HAP and devices to maximize their Energy Efficiency (EE), which is defined as the total received data (in bits) for each Joule of consumed energy. Unlike prior works, we consider the case where both the HAP and devices have causal knowledge of channel state information and their energy arrival process. We model the power allocation problem as a Two-layer Markov Decision Process (TMDP), where the first layer corresponds to the HAP and the second layer consists of devices. We then outline a novel, decentralized Q-Learning (QL) solution that employs linear function approximation to represent the large state space. The simulation results show that when the HAP and devices employ our solution, their EE is orders of magnitude higher than competing policies.

Power Control Optimization for Large-Scale Multi-Antenna Systems

Large-scale multi-antenna systems can effectively improve data transmission reliability and throughput for smart grid. However, the massive number of antennas and radio frequency (RF) chains also result in high complexity and energy cost. In this paper, we develop a new performance benchmark named energy economic efficiency for measuring the time-average throughput per energy cost. Then, we investigate how to maximize long-term energy economic efficiency via the joint optimization of communication and energy resource allocation. The formulated joint optimization problem is NP-hard because it not only involves long-term nonlinear optimization objective and constraints, but also involves both integer and continuous optimization variables. Next, we propose an online joint antenna selection and power control algorithm by combining nonlinear fractional programming, Lyapunov optimization, and bisection method. The proposed algorithm can achieve bounded performance deviation from the optimum performance without requiring the prior knowledge of future channel state information (CSI), energy arrival, and electricity price. Finally, a comprehensive theoretical analysis is provided, and the proposed algorithm is verified through simulations under various system configurations.

How Intramolecular Vibrational Energy Transport Changes with Rigidity and Polarity of the Environment?

Vibrational energy transport through oligomeric polyethylene glycol (PEG) chains can occur ballistically via optical vibrational chain bands, showing fast and constant transport speed and high efficiency of the transport, thus offering means to transfer significant quanta of energy, exceeding 1000 cm–1, to large distances exceeding 60 A. We report how the intramolecular energy transport time, through-chain transport speed, and end-group cooling rate depend on the rigidity and polarity of the environment. The experiments were performed with end-group labeled PEG oligomers using two-dimensional infrared (2DIR) spectroscopy. The ballistic energy transport was initiated at one end of the chain by exciting an azido moiety at ca. 2100 cm–1 and recorded at another end of the chain by probing the carbonyl stretching mode of succinimide ester. We found that the rigidity of the environment, polystyrene (PS) matrix vs. a solution of similar polarity, did not change the energy transport times much, nor the through-chain transport speed. These results suggest that in mildly polar media, dynamic fluctuations, occurring in solution but largely frozen in a solid matrix, are not the dominant cause of the dephasing of the chain states, despite the presence of fast relaxation components in the solution. The similarity of the transport times in different media suggests that the secondary chain structure does not affect much the transport in PEG chains. The solvent polarity affected the intramolecular transport significantly: the transport efficiency in polar DMSO is ca. 1.6 fold smaller than that in nonpolar CCl4 or PS. The cooling time of the succinimide ester end group is reduced in more polar solvents affecting the waiting time dependence shape and thus the energy arrival time to the reporter. The analysis of different ways of extracting the energy arrival time from the data is presented. The observed dependences of the through-chain transport time on the solvent polarity suggests the presence of multiple wavepackets propagating in the PEG chain with different group velocities.