In this paper, the pixel-level corner position extracted by the Harris algorithm is used as the initial value of sub-pixel corner detection, and then it is used to detect the sub-pixel corner detection. The sub-pixel corner detection algorithm based on the inner product of the gray gradient has weak anti-noise and poor anti-distortion ability, and the sub-pixel corner detection algorithm based on Gauss quadric surface fitting has a high dependence on the initial value. In this case, a quadric fitting based on an iterative search thought algorithm of subpixel corner detection is proposed in this paper. The experiment shows that the improved algorithm has a stronger ability in anti-noise and anti-distortion and low dependence on the initial value. Using this algorithm to calibrate the camera, the re-projection error is smaller and the calibration accuracy is higher.