In the field of ultra-precision machining, the dynamic performance of ultra-precision equipment contributes a lot to the processing accuracy. In this paper, in order to study the dynamic performance of the flycutting machine tool, the virtual material method was adopted creatively to build a machine tool model. This method overcame the complexity of actual structure and obtained more accurate results than traditional methods. Subsequently, the finite element method was applied to analyze the dynamic performance of the virtual material model. Furthermore, the modal test for the flycutting machine tool was performed to verified the simulations by means of the one-point hitting and multi-point measurement. The simulation results indicate that the virtual material model has high comprehensive accuracy, of which the average error of natural frequency is 5.9% and errors are all less than 6% excepting the first order. Moreover, combined with the flycutting experiments, it can be found that the fifth order mode of machine tool contributes a lot to processing quality, which leads to the waviness of 53 mm on the machined surface directly because of the weak stiffness of beam-column joints. By increasing the stiffness of the key joints, the waviness can be eliminated, which greatly improves the surface quality of the workpiece.