Transceiver Design for Nonregenerative MIMO Cognitive Relay Networks With Tomlinson–Harashima Precoding

Abstract

Transceiver with Tomlinson-Harashima (TH) precoding outperforms the linear minimum mean-square-error (MSE) architecture in terms of minimum achievable MSE. In this paper, we investigate transceiver design optimization problem for nonregenerative multiple-input multiple-output cognitive relay networks (CRNs) with TH precoding. In the CRN, a secondary user (SU) source, an SU relay and an SU destination employ a TH precoder, a relay precoder, and a linear equalizer, respectively. For scenario in which SUs know perfect channel state information (CSI) from SUs to primary users, we propose an alternating optimization (AO)-based suboptimal algorithm. Given TH precoder and relay precoder, we derive a closed-form optimal solution of linear equalizer. Given relay precoder, TH precoder can be found by convex optimization. Given TH precoder, we transform nonconvex relay precoder design problem into a difference of convex programming and propose a constrained concave convex procedure-based iterative algorithm to find its local optimum. For scenario in which SUs know imperfect CSI, the channel uncertainties are modeled by worst case model. We derive equivalent worst case interference power constraints and extend the proposed AO-based suboptimal algorithm to cope with the worst case interference power constraints. Simulation results demonstrate that the proposed transceiver design with TH precoding outperforms linear transceiver designs.