The layered rock mass widely exists in mining, construction, transportation, and water conservancy projects, and the damage phenomena of plate crack and spalling often occurs in the process of coal and rock dynamic disaster in deep mining. Therefore, the rock mass nearby excavation surface is usually considered to be composed of layers of thin plate rock mass to reveal the damage and failure mechanism of rock mass. In the whole dynamic process of mining and coal and rock dynamic disaster, rock mass would bear the dynamic disturbance from mine earthquake, and at present, the mechanical characteristics of rock mass are mainly studied under static load, while dynamic mechanical response characteristics and the mechanisms of dynamic damage, failure, and disaster‐causing are still unclear. This study mainly focused on the dynamic response characteristic and failure mechanism of rock mass based on a rectangular thin plate model. The frequency equations and deflection equations of the thin plate rock mass with different boundary conditions (S‐F‐S‐F, S‐C‐S‐C, and C‐C‐C‐C) were established under free vibration by the thin plate model and the dual equation of the Hamilton system, and the deflection equations under impact load were derived based on the Duhamel integral. And then, the effective vibration modes of the thin plate rock mass with different boundary conditions and their natural frequencies were obtained by Newton’s iterative method. Based on the third‐strength theory and the numerical simulation results by LS‐DYNA, the maximum shear of the effective vibration modes and the processes of damage and failure under impact load were analyzed. The research results showed that the initial position of damage and failure may be determined by effective vibration mode with the lowest frequency; the develop tendency of which by the combined actions of other effective vibration modes and the effective vibration modes with lower frequency could have greater influence on the process of damage and failure of the thin plate rock mass, which are beneficial to revealing the mechanism of coal and rock dynamic disaster.