Since their ability to machine complex curved surface, five-axis CNC machine tools have been widely used in the machining field. To guarantee the accuracy of the complex surface machining, multi-axis linkage performance detection of five-axis machine tools is necessary. RTCP (rotation tool center point) is one of the basic essential functions for five-axis machine tools, which could effectively reduce the nonlinear errors caused by rotation axes when five axes move synchronously. On the basis of RTCP function, a way to detect multi-axes linkage performance of five-axis machine tools is introduced in this paper. Some simulations that exhibit the relationship between the tool center error trajectories and the factors influencing the linkage performance of the machine tool are established based on the linkage error model. Analyzing the rule of tool center error trajectories, a linkage error tracing method is proposed based on feature extraction and trajectory similarity. The experiments based on RTCP are carried out, and the tool center point error variation is shown to be consistent with the simulation result. Furthermore, the error of the error tracing method is less than 5%, which means the error tracing method could find the mismatch parameters effectively. This attempt provides a theoretical support for linkage performance detection and error tracing of five-axis CNC machine tools.