Abstract

In Part I of this paper (“Delay-Optimal Dynamic Mode Selection and Resource Allocation in Device-to-Device Communications—Part I: Optimal Policy”), we investigated dynamic mode selection and subchannel allocation for an orthogonal frequency-division multiple access (OFDMA) cellular network with device-to-device (D2D) communications to minimize the average end-to-end delay performance under the dropping probability constraint. We formulated the optimal resource control problem into an infinite-horizon average-reward constrained Markov decision process (CMDP), and the optimal control policy derived in Part I using the brute-force offline value iteration algorithm based on the reduced-state equivalent Bellman equation still faces the well-known curse-of-dimensionality problem, which limits its practical application in realistic scenarios with multiple D2D users and cellular users. In Part II of this paper, we use linear value approximation techniques to further reduce the state space. Moreover, an online stochastic learning algorithm with two timescales is applied to update the value functions and Lagrangian multipliers (LMs) based on the real-time observations of channel state information (CSI) and queue state information (QSI). The combined online stochastic learning solution converges almost surely to a global optimal solution under some realistic conditions. Simulation results show that the proposed approach achieves nearly the same performance as the offline value iteration algorithm and outperforms the conventional CSI-only scheme and throughput-optimal scheme in a stability sense.

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call

Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.