In contemporary wireless communication systems, multicarrier modulation schemes have become widely adopted over single-carrier techniques due to their improved capacity to address challenges posed by multipath fading channels, leading to enhanced spectral efficiency. Orthogonal frequency division multiplexing (OFDM), a prevalent multicarrier scheme in 4G, is favored for its ease of implementation, interference resilience, and high data rate provision. But it falls short of meeting the requirements for 5G and beyond due to limitations such as out-of-band (OOB) emissions and cyclic prefixes. This paper introduces the filter bank multicarrier (FBMC) and universal filtered multicarrier (UFMC) with quadrature amplitude modulation (QAM) and phase shift keying (PSK) waveforms through Additive White Gaussian Noise channel (AWGN), Rayleigh fading channel and Rician channel. The objective of this paper is to enhance the performance of UFMC with reduced complexity through the new filtering approach for achieving optimal outcomes. The proposed scheme, incorporating Tukey filtering technique, demonstrates superior performance in reducing peak-to-average power ratio (PAPR) and improving bit error ratio (BER) compared to the original UFMC signal without necessitating additional power increments. Specifically, the UFMC system with Tukey filtering achieves a notable net gain of 5 dB. Simulation results demonstrate that utilizing various filter types in FBMC and UFMC systems, combined with QAM modulation, significantly reduces OOB emissions compared to conventional systems. In aspect to BER, Tukey window showed almost 10−6 at 15 dB SNR in UFMC which is better than FBMC.