In monocular vision robot systems, the hand-eye calibration approach is crucial for ensuring operational accuracy. A double-layer (Levenberg–Marquardt ([LM] method) optimization (DLMO) method with outlier points screening is proposed to reduce the influence of random errors in robot vision systems and improve the calibration accuracy of the robot hand-eye matrix. First, the equation of the hand-eye matrix is established, and the initial value of the hand-eye matrix is solved by the linear least square method. Second, the Euler angle transformation is applied to the rotation matrix part to ensure its orthogonality. Next, an optimization model of the hand-eye matrix is constructed, and the traditional LM optimization method is used to optimize the initial hand-eye matrix for the first time. Finally, the optimization model of the hand-eye matrix is modified, and the LM optimization method with the outlier sample points screening is applied to optimize the hand-eye matrix for the second time. Hand-eye calibration tests are conducted on an industrial robot equipped with a monocular vision system using the proposed method. Experimental results demonstrate that the average position error of the calibration results obtained by the proposed DLMO method is 0.22 mm, which is superior to the traditional hand-eye calibration method and meets the working requirements of the vision robot in the industrial field.