The measurement of the stiffness of a precision reducer is essential to estimating the reducer. Since the angular sensor’s measurement results include the angle measurement error caused by the instrument’s torsional deformation, it cannot be used as the actual torsional deformation of the reducer. This paper analyzes the instrument’s torsional deformation characteristics to reduce the angle measurement error. Based on the analysis, a new method of calibrating the angle measurement error based on the improved B-spline curve fitting-gradient descent and particle swarm optimization -radial basis function neural network (IBSCF-GDPSO-RBF) method is proposed. The method can eliminate the angle measurement error caused by the instrument’s torsional deformation. The steps of the IBSCF-GDPSO-RBF method are presented, and the angular measurement error compensation is executed under load conditions. The experiment shows that the instrument deformation caused angle measurement error after compensation is within ± two angular seconds. This paper’s innovation proposes the error calibration method based on the IBSCF-GDPSO-RBF method. It provides a reference for measuring and evaluating the actual torsional rigidity of the Rotary Vector (RV) reducer under any load.