PurposeThe sensorless external force estimation of robot manipulator can be helpful for reducing the cost and complexity of the robot system. However, the complex friction phenomenon of the robot joint and uncertainty of robot model and signal noise significantly decrease the estimation accuracy. This study aims to investigate the friction modeling and the noise rejection of the external force estimation.Design/methodology/approachA LuGre-linear-hybrid (LuGre-L) friction model that combines the dynamic friction characteristics of the robot joint and static friction of the drive motor is proposed to improve the modeling accuracy of robot friction. The square root cubature Kalman filter (SCKF) is improved by integrating a Sage Window outer layer and a nonlinear disturbance observer (NDOB) inner layer. In the outer layer, Sage Window is integrated in the square root Kalman filter (W-SCKF) to dynamically adjust noise statistics. NDOB is applied as the inner layer of W-SCKF (NDOB-WSCKF) to obtain the uncertain state variables of the state model.FindingsA peg-in-hole contact experiment conducted on a real robot demonstrates that the average accuracy of the estimated joint torque based on LuGre-L is improved by 4.9% in contrast to the LuGre model. Based on the proposed NDOB-WSCKF, the average estimation accuracy of the external joint torque can reach up to 92.1%, which is improved by 4%–15.3% in contrast to other estimation methods (SCKF and NDOB).Originality/valueA LuGre-L friction model is proposed to handle the coupling of static and dynamic friction characteristics for the robot manipulator. An improved SCKF is applied to estimate the external force of the robot manipulator. To improve the noise rejection ability of the estimation method and make it more resistant to unmodeled state variable, SCKF is improved by integrating a Sage Window and NDOB, and a NDOB-WSCKF external force estimator is developed. Validation results demonstrate that the accuracy of the robot dynamics model and the estimated external force is improved by the proposed method.