Previous studies about the dynamics of the planetary gear system are mainly focused on transmission systems with a fixed transmission ratio, whereas the dynamic studies of vehicle automatic transmission with multiple transmission ratios achieved by engaging different clutches were seldom investigated. This is largely due to the lack of modeling strategies for the stochastic tooth impacts of detached clutches on the dynamics of transmission systems. Ignoring the stochastic tooth impacts on the system’s performance will result in inaccurate analysis that may overestimate the machine’s service performance. In this paper, an original dynamic model of the multistage planetary gear set-bearing-rotor-clutch coupling system with multi-shift and multi-speed transmission is established considering the tooth impacts of detached clutches. An embedded measurement method is developed to measure the in-situ torsional vibration responses of rotating parts under various shift and speed conditions, which are directly compared to simulated responses of the proposed model for validation purposes. Finally, the effects of the rotating speed and clutch’s tooth backlash on the random tooth impact characteristics of clutches are revealed. The modeling strategy of this work provides a more accurate and realistic simulation of automatic transmission’s dynamic performance and supplies theoretical guidance for the strength evaluation and early fatigue failure analysis between mating teeth of clutches.