Digital projectors are used as standard parts at present in fringe projection profilometry systems to project structured-light patterns onto the object surface to be measured, and the distortion of the projector lens must be calibrated and compensated accurately to satisfy the accuracy requirement of industrial applications. A novel method is proposed to determine the projector pixel coordinates of the marker points of a calibration target accurately in terms of projective transform. With the method, the projector can be calibrated with accuracy of sub-pixel level. The method is applicable for the calibration target with a chessboard pattern or a circle pattern, and the calibration result is independent on the results of camera calibration. Experimental results are shown to demonstrate the effectiveness and validity of the proposed method.