The friction force during the scanning of a three-dimensional (3D) probe inevitably leads to undesired tangential displacement, the processing of which is challenging, especially for probes with unequal stiffness. In this paper, a new friction-coupled measurement model of the probe is deduced based on the principle of mechanics so as to realize the positioning and tracking of any friction force between the stylus ball and the sample surface. Considering the two features of the time-varying normal displacement in the scanning and the 3D unequal stiffness of the probe, this paper reveals the nonlinear bidirectional conversion between the 3D resultant force and the resultant displacement at the measuring end. The measurement deviations and the geometric relationship among the frictional force, equivalent force, normal contact force and friction-coupled force are theoretically analyzed. In addition, the finite element analysis (FEA) and calibration experiment analyze the force characteristics and sensitivity of the probe, and obtain the friction-coupled measurement model for actual measurement. The designed measurement experiment on the scanning trajectory compares the calculation difference of the displacement and azimuth angles with and without friction, and obtains a conclusion consistent with the theory and simulation.