The existing robot-world and hand-eye calibration algorithms always treated the robot pose as a deterministic value. Its uncertainty is not taken into account, which affects the calibration accuracy when the robot’s position accuracy is low. In this paper, we proposed a factor graph approach to simultaneous robot-world and hand-eye calibration, which considers the measurement error of the robot joint encoder. And then, the nonlinear optimization of reprojection error minimization based on a product of exponentials (POE)-based model is implemented to solve the calibration problem based on lie groups. Through simulation and experiment, we validated that the proposed algorithm can get more accurate calibration results compared with the state algorithm.