China has over 1500 open-pit mines and 5000 dumps consisting of waste rock from the mining process. Due to dump instability in an open-pit mine and its diverse foundations, landslides and mudslides frequently occur. Heaped loose waste rock and concentrated heavy rainfalls are the two important factors affecting slope stability of a dump. Using the high Dump II within the Nanfen Open-Pit Iron Mine with a slope height of 300 m as a case study, this paper first proposes a physical model similarity ratio according to the on-site engineering geological survey data. The governing principles of deformation in the shallow dump layers in terms of different heaped loads and rainfall were then determined using fiber optic sensing to conduct an experimental study on the monitoring of the dump stability with an indoor physical model. Experimental results confirm that the amount of rainfall and heaped load has a great impact on the landslide in a dump. With an increase in the amount of material heaped onto the pile, the place between two heaped load points is squeezed and slowly deformed. As rainfall begins, the deformation of the rock-soil mass is significant, and constantly increasing rainfall intensity is accompanied by a dangerous sliding surface. Finally, the FLAC3D method was used to simulate the deformation features in the shallow part of Dump II under different heaped load conditions and verify the experimental results of the indoor physical model. By comparing the physical model experiments and numerical simulation results, we propose monitoring the stability of Dump II using this fiber optic sensing technology and provide the scientific basis for stability monitoring of similar dumps to detect the early warning signs of mudslides or landslides.