Robot calibration of an industrial robot is of importance for those applications requiring high positioning accuracy. The use of numerical optimization techniques to identify the accurate kinematic parameters and tackle the residual errors for improving positioning accuracy is still challenging. The difficulties in using the conventional optimization techniques is associated with the expensive computation of the second derivative terms in the Hessian matrix and even more linked to the additional improvement of the residual errors induced by the configuration and payload effects for the practical applications. A hybrid calibration procedure, which is robust and efficient, is proposed to handle the said problems in this study. The first step is to utilize the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm together with the quasi-Newton method to optimize the cost function with non-smooth nature established using massive measurement data. In the second step, two kinds of artificial neural network (ANN) algorithms are employed to further improve the nonlinear residual errors induced by payload and configuration effects. After the completion of the hybrid calibration, improvements of 88.1% and 80.1% have been attained, respectively, for the mean and maximum positioning errors of the robot end-effector, i.e., the mean/maximum positioning errors are reduced from 2.613mm/6.294mm to 0.310mm/1.255mm for 840 untrained measurement data. The experimental results also validated the effectiveness of the proposed method.