The UAV Trajectory Optimization for Data Collection from Time-Constrained IoT Devices: A Hierarchical Deep Q-Network Approach

Zhenquan Qin,Xinwei Zhang,Bingxian Lu,Zhonghao Liu,Linlin Guo,Xuan Zhang

doi:10.3390/app12052546

Zhenquan Qin, Xinwei Zhang + Show 4 more

Open Access

https://doi.org/10.3390/app12052546

Copy DOI

Abstract

Recently, using unmanned aerial vehicles (UAVs) to collect information from distributed sensors has become one of the hotspots in the Internet of Things (IoT) research. However, previous studies on the UAV-assisted data acquisition systems focused mainly on shortening the acquisition time, reducing the energy consumption, and increasing the amount of collected data, but it lacked the optimization of data freshness. Moreover, we hope that UAVs can perform long-term data collection tasks in dynamic scenarios within a constantly changing age of information (AoI) and within their own power levels. Therefore, we aim to maximize the quality of service (QoS) based on the freshness of data, while considering the endurance of the UAVs. Since our scenario is not an inertial order decision process with uniform time slots, we first transform the optimization problem into a semi-Markov decision process (SMDP) through modeling, and then we propose a hierarchical deep Q-network (DQN)-based path-planning algorithm to learn the optimal strategy. The simulation results show that the algorithm is better than the benchmark algorithm, and the tradeoff between the system QoS and the safe power state can be achieved by adjusting the parameter βe.

Highlights

Unmanned aerial vehicles (UAVs) have received extensive attention in wireless communication networks due to their small size, flexibility, mobility, and controllability [1–4]
This is due to the large action state space of the traditional deep Q-network (DQN) algorithm, while the hierarchical DQN (H-DQN) method is modeled through the semi-Markov decision process (SMDP) and the option, which greatly reduces the action state space, making it easier to learn the optimal strategy
In order to enable UAVs to perform long-term data collection tasks in dynamic scenarios with a constantly changing age of information (AoI) and their own power levels, we propose a method based on the quality of service (QoS) maximization problem of the AoI, and we use a semi-Markov decision process (SMDP) to describe this problem

Summary

Introduction

Unmanned aerial vehicles (UAVs) have received extensive attention in wireless communication networks due to their small size, flexibility, mobility, and controllability [1–4]. UAVs can fly close to each ground node and can establish air-to-ground channels dominated by their line of sight (LoS), which can greatly reduce the transmission energy consumption, and can significantly improve the throughput of the ground nodes. These advantages have made UAV-assisted IoT networks attract extensive attention in recent years. Zhan et al [11] considered jointly optimizing the wake-up time of the sensor nodes and the trajectory of UAVs. While ensuring the required amount of data is collected, the energy consumption is minimized, and the suboptimal solution is obtained by using the iterative convex optimization technique. Moataz et al [13] considered the problem of UAV-assisted data acquisition in delay-sensitive application scenarios, and they maximized the available quantity of devices serving the IoT by optimizing the trajectory and resource allocation of UAVs

Objectives

Results

Conclusion