The exponential growth in wireless communication has resulted in a high demand for novel wireless services within the frequency bands that can deal with the challenge pertaining to the current spectrum shortage. Thus, a 5G-MIMO based CR communication system has been developed to enhance the spectrum by employing a spectrum sensing (SS) algorithm. This recommended SS algorithm, based on the hybrid filter detection (HFD) method, employs the Cosine law for filtering the traffic signal, and then enables segmentation by applying the Welch algorithm. Then, the Hann algorithm is employed to window the traffic signal, and facilitates damping the MIMO impact for different waveforms. This also includes universal filtered multi-carrier (UFMC), filtered-orthogonal frequency division multiplexing (F-OFDM), filter bank multi-carrier (FBMC), non-orthogonal multiple access (NOMA), and Generalized Frequency Division Multiplexing (GFDM) waveforms. Also, evaluation of the operating parameters such as SNR, the antenna count, the signal span, and power as well as computational complexity was done. Based on the simulation results, a notable achievement was seen for the parameters that were below 0 dB of SNR, <0.09 % global system error probability, >96 % global detection probability, <0.06 % global false alarm probability, and possessing simple complexity. The recommended system’s parameters were assessed, which demonstrated its superiority versus other previously developed systems. This algorithm was found to identify all types of 5G signals and not just one type of 5G.