A proposed waveform for fifth generation (5G) communications is the filtered orthogonal frequency division multiplexing (F-OFDM) system. Filtering-based waveform frameworks distinguish themselves by suppressing out-of-band emission and asynchronous transmission. Meanwhile, the high Peak-To-Average Power Ratio (PAPR) remains a barrier to new waveform possibilities. Partials transmit sequence (PTS) is a successful strategy for reducing high PAPR in multicarrier networks. This work presents a multicarrier in OFDM and UFMC based on the Filter Bank Multicarrier (FBMC) waveform for communication systems. Furthermore, FBMC waveforms have limited orthogonality between subcarriers, which can mitigate Doppler and multipath effects. This FBMC approach uses the cosine law to filter the traffic signal, and the Welch algorithm is offered for segmentation, which lowers noise in estimated power spectra. As a result, a hybrid Selective Mapping (SLM) with Cuckoo Search and Ant Lion Optimization (CS-ALO) algorithms is suggested using the FBMC framework to reduce PAPR in the OFDM system. The proposed work is implemented, and detailed experimental research is carried out to show the increased performance in 5G networks. Based on a fair comparison, the F-OFDM scheme outperforms previous approaches in terms of BER and multicarrier system capacity. Simulation results show that the proposed waveform design and processing method outperform standard OFDM and F-OFDM methods in PAPR reduction and communication integrated systems.