This paper focuses on kinematic calibration of a novel 5-DOF hybrid robot and attention is paid to the suppression of ‘observation noise’ impact caused by the measurement system and unconsidered source errors. The proposed method that is pertinently developed to improve the accuracy of the targeted robot, is implemented as follows: with the aid of screw theory, a linear map between the pose error twist of the end effector and all possible geometric source errors of the hybrid machining robot is formulated. Following the measurement and solution of pose error twist, an extended Kalman filter (EKF) is then employed to obtain more reliable and stable identification results of geometric errors. A linearized error compensator is then proposed and used in calibration experiments. Compared with the least square method, the proposed EKF method is proved to be robust and has good compensation effect in the verification configurations. After kinematic calibration, the positional and angular errors of the robot are reduced to be less than 60.6 μm and 67.6 μrad respectively. The results verify the effectiveness and general applicability of the proposed identification and compensation strategy.