Resonator Fiber optic gyro (RFOG) is a high accuracy inertial rotation sensor based on the Sagnac effect. The optical fiber ring resonator (OFRR) is the key rotation sensing element. The backscattering characteristics of the OFRR are fully investigated. The Rayleigh backscattering dominates the backscattering spectrum with the input power below the threshold. High carrier suppression is crucial to reduce this Rayleigh backscattering error. Being different from the intrinsic Rayleigh scattering of the fiber, the stimulated Brillouin scattering (SBS) in the OFRR should be avoided. This is because the finesse and the resonance notch depth of the OFRR decrease for the pump depletion through the SBS process. The shot noise limited sensitivity (SLS) of the RFOG is improved by increasing the input power. Through analysis, it is found that the threshold input power is improved after phase modulation. The SLS of the RFOG is analyzed at different modulation parameters and its relevant SBS threshold. Accordingly, the optimized modulation frequency and the corresponding maximum input power are all obtained. A simple method of testing the frequency shift in the SBS is also proposed. In this method, the central frequency of the laser source is locked to the resonance frequency of the OFRR in one direction. A steady SBS light is observed and its frequency shift is measured together.