In this paper, we consider the secure communications in a cognitive untrusted relay network, where the secondary source intends to communicate with a secondary destination through an untrusted relay in the presence of the direct source–destination link. Specifically, we first examine the connection outage probability (COP) and secrecy outage probability (SOP) to investigate the reliability and security performance in two cases, where the secondary destination exploits the maximum ratio combining (MRC) scheme or the selection combining (SC) scheme. To characterize the tradeoff between reliability and security, we then investigate the effective secrecy throughput (EST) performance by including the COP and SOP in a unified manner. In order to gain additional insights from the performance evaluation, we also provide the asymptotic expressions for the COP, SOP, and EST in high signal-to-noise ratio region. It is demonstrated that the interference temperature constraint incurred from the primary network enables a tradeoff between reliability and security of the secondary network. Moreover, the resulting analysis shows that using the untrusted relay to forward the transmitted message is unnecessary when the secondary destination employs the SC scheme and the untrusted relay operates in half-duplex mode.