Coal seam mining causes movement and deformation of the overlying rock mass and ground surface, directly resulting in serious damage to the surface construction facilities. To investigate the deformation mechanism and characteristics of the overlying rock mass, three-dimensional numerical simulation and model test are carried out on an open-pit mine slope. The results show that the sliding instability characteristics of the slope can be identified based on the numerical analysis. Coal mining affects on the deformation characteristics of the overlying rock mass near the inner dump. The closer to the slope boundary, the greater the subsidence of overlying rock mass. The maximum displacement occurs near the inner dump. The slope mining has great influence on the deformation characteristics of inner dump. Moreover, the damage process of overlying rock mass can be further studied via the model test. After coal seams mining out, the overlying rock mass occurs overall settlement failure. There is a large difference between the lower end side of working face and the upper end side at the caving angle of roof. This work can provide a basis for the safe operation of coal mine disaster prevention and mitigation.