The measurement accuracy of the magnetometer system is closely related to the calibration effect. Available magnetometer calibration methods face the limitation on the assumption that the background magnetic field is uniform. The fact is that the experiment magnetic environment is inhomogeneous for the presence of human activities mainly. This problem could be partially addressed by an auxiliary large-scale triaxial Helmholtz coil to compensate the nonuniformity of magnetic field. This paper proposes a novel calibration method based on the tensor invariants in the nonuniform magnetic field without extra device. The inhomogeneity of background field implies a nonzero gradient field; a new correction model in the gradient field has been established. Calibration parameters are estimated to compensate the observed data using the concept of magnetic gradient tensor and the corresponding rotational tensor invariants, combined with the Levenberg Marquardt optimization. And after the calibration operation, the tensor invariants should be close to a constant. Numerical simulations have been designed and carried out, showing that the proposed algorithm had a good compensation performance with fast convergence, robustness in any gradient field. Field experiments with fluxgate magnetometer array were also conducted to verify this calibration procedure. The measured results are consistent with the simulation results. The root-mean-square (RMS) error is reduced from 616.239 to 1.608 nT/m, after calibration. In addition, the machining error of the frame, the time stability of the background field, the scale of gradient field, and robustness of this proposed method have been discussed.