The probe-based measurement presents an economical and efficient on-machine measurement (OMM) method. However, the low position accuracy of robot presents significant challenges for probe-based robot OMM. To improve the measurement accuracy of robot OMM system, this paper calibrates the geometric parameters of robot OMM system through a closed-loop calibration method with sphere constraint. First, the geometric error model for robot OMM system is established. Given the difficulty in model parameter identification, this paper defines a new index, the relative distance sensitivity index (RDSI), to reveal the contribution of model parameter to relative distance accuracy. Subsequently, sensitivity analysis is conducted on model parameters based on RDSI and model parameters are grouped according to the analysis results. In parameter identification, parameters are identified step by step according to the grouping results. Experiment results show that after calibration, the max distance measurement error of robot OMM system is reduced from 0.54 mm to 0.12 mm.