The crosstalk coefficient calibration of de-crosstalk in color fringe projection profilometry is an essential step for the high-accuracy measurement. In this paper, a novel approach for calibrating crosstalk matrix of color camera is proposed. The wrapped phase error model introduced by color crosstalk in orthogonal pattern is established. Compared with the existing calibration methods depending on calculating the modulation of the crosstalk channel, the crosstalk coefficients are obtained from phase error in our method. By projecting the designed color orthogonal phase-shifting fringe patterns onto a white plate, the phase-shifting fringe patterns in both horizontal and vertical directions can be separated from captured images. The coefficients between different channels are calibrated by fitting the error relationship between the wrapped phase containing crosstalk and the standard ones. Coefficient fitting simulations and experimental validations including shape measurement of a white plate and distance measurement of a step block are carried out to verify the effectiveness of the proposed method.