Abstract Geometric accuracy is one of bottleneck problems restricting the extension of application of the hybrid kinematic machine (HKM). On the premise of having a certain basic manufacturing accuracy, kinematic calibration is an effective means of improving the geometric accuracy of HKMs. In the study, the kinematic calibration technology of a new-type 5-degree of freedom (DOF) HKM was investigated. Moreover, by applying regularisation method, the problem whereby the identification algorithm shows poor robustness due to the presence of an ill-posed identification matrix during error identification was solved. At first, a geometric error model for the HKM satisfying completeness and minimality was established based on screw theory. Then, when identifying error parameters, two indices for evaluating the effectiveness of identification algorithms were proposed: on this basis, the stabilities of solutions obtained through four regularisation identification algorithms and the predictive capability of solutions for the model was analysed and compared by simulated experiment. Additionally, the influence of measurement noises at different levels on five identification algorithms was explored. Finally, an error compensation strategy was proposed and a kinematic calibration experiment was conducted. The test results showed that, after kinematic calibration, the position and orientation errors of the HKM within the whole workspace separately reduced to less than 0.054 mm and less than 0.041o, thus validating the effectiveness of the proposed method.