Abstract Cognitive radio (CR) is an intelligent and adaptive radio technology that automatically detects the available channels in the wireless spectrum and sometimes changes the transmission parameters to enable effective communication. Spectrum sensing in CR prevents harmful interference with the licensed users and maximizes the spectrum utilization. Thus, this paper proposes a technique for optimal channel estimation and spectrum sensing for MAC layer protocol in CR networks such that the scheduling issues are addressed. Initially, in the CR networks, spectrum sensing is done using the proposed optimal naive Bayes classifier (ONBC) based on the signal statistics, such as energy and likelihood ratio. The ONBC is developed by integrating the bat–bird swarm algorithm (BBSA) with the naive Bayes classifier, which works based on the Bayesian concept. The BBSA is newly developed by integrating the bird swarm algorithm (BSA) and bat algorithm. Finally, the channel estimation is done using the pilot-based sequential procedure and least square estimation (LSE). The analysis of the proposed method is done in the Rayleigh and Rician environments using 256 and 512 sub-carriers. From the results, it is exposed that the proposed BBSA + LSE pilot-based sequential method obtains the bit error rate, normalized energy and Probability detection (PD) of is 0.0126, 0.8446 and 0.9355, respectively.