Smart antenna system is one of the most suitable techniques making the sharing between incumbent Fixed Satellite Services (FSS) and the Fifth Telecommunication Generation (5G) possible, because adaptive beamforming can make null the beam towards the FSS earth station. This paper proposes the use of Recursive Least Square (RLS) algorithm for adaptive beamforming in 5G Base Station (BS) having a null beam toward FSS earth station. We perform a computer simulation to evaluate the proposed RLS algorithm and compare the result in terms of separation distance between FSS earth station and 5G BS with and without RLS algorithm. The results confirmed that the RLS algorithm for adaptive beamforming makes the coexistence between 5G and FSS possible indicated by the significant reduction of separation distance between FSS earth station and 5G BS leading to the minimum interference of 5G BS to the FSS earth station.