This research aims to optimize the interference mitigation and improve system performance metrics, such as bit error rates, inter-carrier interference (ICI), and inter-symbol interference (ISI), by integrating the Redundant Discrete Wavelet Transform (RDWT) with the Arithmetic Optimization Algorithm (AOA). This will increase the spectral efficiency of MIMO-OFDM systems for ultra-high data rate (UHDR) transmission in 5G networks. The most important contribution of this study is the innovative combination of RDWT and AOA, which effectively addresses the down sampling issues in DWT-OFDM systems and significantly improves both error rates and data rates in high-speed wireless communication. Fifth-generation wireless networks require transmission at ultra-high data rates, which necessitates reducing ISI and ICI. Multiple-input multiple-output orthogonal frequency division multiplexing (MIMO-OFDM) is employed to achieve the UHDR. The bandwidth and orthogonality of DWT-OFDM (discrete wavelet transform-based OFDM) are increased; however system performance is degraded due to down sampling. The redundant discrete wavelet transform (RDWT) is proposed for eliminating down sampling complexities. Simulation results demonstrate that RDWT effectively lowers bit error rates, ICI, and ISI by increasing the carrier-to-interference power ratio (CIR). The Arithmetic Optimization Algorithm is used to optimize ICI cancellation weights, further enhancing spectrum efficiency. The proposed method is executed in MATLAB and achieves notable performance gains: up to 82.95% lower error rates and 39.88% higher data rates compared to the existing methods. Conclusion: The integration of RDWT with AOA represents a significant advancement in enhancing the spectral efficiency of MIMO-OFDM systems for UHDR transmission in 5G networks. The proposed method not only enhances system performance but also lays a foundation for future developments in high-speed wireless communication by addressing down sampling issues and optimizing interference mitigation.