The advent of 5G technology aims to revolutionize wireless communication by providing significantly higher data rates, ultra- reliable low latency, and enhanced connectivity. However, optimizing 5G network performance remains a challenging task, particularly in terms of Bit Error Rate (BER) and spectral efficiency. Multiple-Input Multiple-Output (MIMO) technology, coupled with beamforming algorithms, offers a promising solution to these challenges. This study investigates the application of the Fractional Least Mean Square (FLMS) algorithm for beamforming in MIMO systems to improve BER and spectral efficiency in 5G networks. The primary problem addressed is the need for effective interference mitigation and signal enhancement in densely populated environments, which traditional methods struggle to handle. The proposed method utilizes the FLMS algorithm to dynamically adjust the beamforming weights, thereby optimizing the signal reception quality. The algorithm's fractional nature allows for finer adjustments and better adaptation to varying channel conditions compared to conventional LMS algorithms. Simulation results show the efficacy of the proposed method. Specifically, the implementation of the FLMS algorithm in a 4x4 MIMO system shows a reduction in BER by 30% and an improvement in spectral efficiency by 25% compared to traditional beamforming techniques. These numerical values highlight the potential of FLMS in enhancing 5G network performance, making it a viable approach for future wireless communication systems.