The ultrasensitive magnetometer has a vital importance in fundamental research and applications. Currently, the spin-exchange relaxation-free (SERF) atomic magnetometer has been reported with a sensitivity around the level of fT/Hz1/2. To enhance the sensitivity, a gradiometer configuration has usually been introduced to cancel the common-mode noise between two separate channels. However, the signal and response from different channels are not the same due to the attenuation of the pump beam. Here, we proposed a counter-propagating optical sideband pumping method to polarize the atoms, using the electro-optic modulator to modulate the single-pump beam, generating two symmetrically red- and blue-detuned sidebands of frequency. This scheme leads to a significant reduction of undesirable effects coming along with the optical pumping, such as light shifts and spatial inhomogeneity in atomic spin polarization. With the help of this pumping scheme, the two channels have the same magnetic response, and we have built a gradiometer atomic magnetometer with a sensitivity of 0.5 fT/Hz1/2 ranging from 5 to 40 Hz. Our results propose the possibility of creating larger arrays of atomic magnetometers (AMs) with high sensitivity and spatial resolution based on single-vapor cells for magnetocardiography and magnetoencephalography imaging or searching for exotic spin-dependent interactions.