In this paper, we systematically studied the relationship between magnetometer sensitivity and microwave frequency modulation. It is proved by experiments that the sensitivity of the noise limit will reach 1 nT Hz−1/2 (the theoretically calculated sensitivity is 0.5 nT Hz−1/2), and a debugging method is proposed. At the same time, the optimal modulation depth is obtained by the existing theoretical model simulation. In theory, the sensitivity of the magnetometer will reach 0.1 nT Hz−1/2.