SummaryRelay selection plays a crucial role in enhancing the performance of wireless networks particularly in the context of cognitive radio (CR) systems with energy harvesters. In this paper, we propose a novel approach, namely, CGAPSO Shapley, for the best relay selection while simultaneously optimizing the parameters of signal‐to‐interference‐plus‐noise ratio (SINR), throughput, and outage probability. The CGAPSO Shapley algorithm combines the Shapley value, a cooperative game theory concept, with cellular genetic algorithm particle swarm optimization (CGAPSO) to achieve effective and efficient optimization of relay selection. The CGAPSO framework provides a hybrid structure that integrates cellular genetic algorithm (CGA) and particle swarm optimization (PSO), enabling simultaneous evolution of the population and particles within cells. The incorporation of the Shapley value and the hybrid CGAPSO framework enables effective exploration of the solution space and provides decision‐makers with comprehensive insights for relay selection. By utilizing the Shapley value, we assign weights to the relay nodes based on their contributions to the overall optimization objectives, considering their CR capabilities and energy harvesting capabilities. Some benchmark test functions are used to compare the hybrid algorithm with both the standard CGAPSO, Particle swarm optimization gravitational search algorithm (PSOGSA) and PSO algorithms in evolving best solution. The results show the hybrid algorithm possesses a better capability to escape from local optimums with faster convergence than the standard algorithms. The novel CGAPSO Shapley approach achieves an outage probability of 0.323324, marking a significant improvement of 60% over the outage probability achieved with conventional approach.