AbstractIn the fifth generation (5G) wireless communication system, the non‐orthogonal multiple access (NOMA) scheme has recently been proposed as a possible candidate for improving spectral efficiency. We look at power‐domain NOMA in this study, in which, multiple simultaneous transmissions are allowed at the same channel with different transmission powers. To achieve reliable transmissions in NOMA, the hybrid automatic repeat request (HARQ) scheme can be integrated into NOMA. The performance of a downlink NOMA system using the Type I HARQ method is investigated in this research. This type of HARQ does not entail expensive device which is considered one of the requirements for a massive 5G‐based Internet of Things network. Unfortunately, in the literature, it has been shown that deriving closed‐form expressions for the outage probabilities of HARQ‐aided NOMA systems is intractable due to the incorporation of multiple fractional random variables. Consequently, in this article, we follow a simple yet efficient methodology to study our considered HARQ‐aided NOMA system. The HARQ‐assisted NOMA system on the downlink is modeled as an absorbing discrete‐time Markov chain (DTMC). Thereafter, under perfect and imperfect channel estimation, we use this model to examine system performance in terms of outage probability and estimated number of retransmissions. Interestingly, we demonstrate that our DTMC‐based analysis is straightforward, repeatable, and accurate. Moreover, in this article, we propose a cooperative HARQ (CHARQ) scheme for the downlink NOMA system. According to CHARQ, the users cooperate with the base station in the retransmission process that will result in improving the performance of NOMA. Monte Carlo simulations have been conducted to not only verify the accuracy of our DTMC‐based analysis, but also manifest the superiority of CHARQ over HARQ.