SummaryOne of the promising technologies for 5G cellular networks is machine‐to‐machine (M2M) communications. We propose a cognitive radio network (CR) that includes a primary cellular system and a secondary cognitive system. The primary cellular system has a primary client (PC) and the secondary cognitive system has M2M clients which are called secondary clients (SCs). In a conventional system of CR network based on orthogonal frequency division multiple access (CR‐OFDMA), when the primary client (PC) is absent, only one SC can obtain the idle spectrum. But, the SC must leave the spectrum when detecting the existence of the PC. So, the spectrum usage of this system is very low. This paper proposes a cooperative CR network based on nonorthogonal multiple access (CCR‐NOMA) for spectrum sensing using energy detection (ED) to allow multiple SCs to share the same frequency at the same time, but are differentiated according to the power domain or code domain, to improve spectrum efficiency of 5G communications and the transmission performance of CR network at the absence and presence of the PC. To evaluate the channel sensing performance of the ED technique in CCR‐NOMA, we derived a closed‐form expression between the achievable throughput and sensing time for the CCR‐NOMA system. The same analysis for the case of CR‐OFDMA is reproduced for the sake of comparison. The analysis showed that the CR‐NOMA system for M2M communication outperforms the CR‐OFDMA system for M2M communication for the same noncooperative and cooperative spectrum sensing and physical layer parameters.