The machine tool structure vibration during machining operation is directly linked to machining quality, production efficiency as well as machine reliability. Hence, this research focuses on the dynamic modeling of the whole machine tool considering the milling process to predict vibration response and dynamic characteristics during machining. Applying the lumped parameter and finite element hybrid method, in order to improve virtual simulation efficiency, the analytical models of the workpiece clamping worktable system and the tool clamping column spindle system are theoretically developed considering the nonlinearity of moving joints and the position-dependent dynamics. The integrated model of the whole machine tool dynamics is established by coupling the models of two subsystems with the generated nonlinear dynamic load in the ball end milling process including the effects of vibrations and tool system errors. Subsequently, the frequency response functions and the instantaneous cutting forces and vibration responses under different cutting conditions are measured and compared with simulation results to evaluate the prediction ability of the proposed model. Finally, the experimentally confirmed model is used to carry out the parameter analysis of machine tool. The investigations reveal that the worktable system dynamic characteristics vary with the joint assembly preload, but the change becomes no longer significant when the preload increases to a certain value. Besides, cutting conditions have a significant influence on the vibration characteristics of machine tool during machining.